--- I0313 01:20:52.345083 1 main.go:213] Received signal terminated. Forwarding to sub-process "hyperkube". I0313 01:20:52.345341 12 genericapiserver.go:554] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0313 01:20:52.345372 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ShutdownInitiated' Received signal to terminate, becoming unready, but keeping serving I0313 01:20:52.345380 12 controller.go:128] Shutting down kubernetes service endpoint reconciler I0313 01:20:52.360194 12 genericapiserver.go:544] "[graceful-termination] shutdown event" name="AfterShutdownDelayDuration" I0313 01:20:52.360241 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'AfterShutdownDelayDuration' The minimal shutdown duration of 0s finished W0313 01:20:52.367495 12 lease.go:265] Resetting endpoints for master service "kubernetes" to [] I0313 01:20:52.381586 12 genericapiserver.go:721] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0313 01:20:52.381626 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TerminationPreShutdownHooksFinished' All pre-shutdown hooks have been finished I0313 01:20:52.381802 12 genericapiserver.go:633] "[graceful-termination] shutdown event" name="NotAcceptingNewRequest" I0313 01:20:52.381825 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'InFlightRequestsDrained' All non long-running request(s) in-flight have drained I0313 01:20:52.381931 12 genericapiserver.go:670] [graceful-termination] in-flight non long-running request(s) have drained I0313 01:20:52.381954 12 genericapiserver.go:679] "[graceful-termination] not going to wait for active watch request(s) to drain" I0313 01:20:52.381967 12 genericapiserver.go:711] "[graceful-termination] shutdown event" name="InFlightRequestsDrained" I0313 01:20:52.382068 12 dynamic_serving_content.go:149] "Shutting down controller" name="aggregator-proxy-cert::/etc/kubernetes/static-pod-certs/secrets/aggregator-client/tls.crt::/etc/kubernetes/static-pod-certs/secrets/aggregator-client/tls.key" I0313 01:20:52.382101 12 controller.go:157] Shutting down quota evaluator I0313 01:20:52.382110 12 controller.go:176] quota evaluator worker shutdown I0313 01:20:52.382751 12 apf_controller.go:389] Shutting down API Priority and Fairness config worker I0313 01:20:52.383148 12 gc_controller.go:91] Shutting down apiserver lease garbage collector I0313 01:20:52.383190 12 controller.go:132] Ending legacy_token_tracking_controller I0313 01:20:52.383201 12 controller.go:133] Shutting down legacy_token_tracking_controller I0313 01:20:52.383284 12 clusterquotamapping.go:142] Shutting down ClusterQuotaMappingController controller I0313 01:20:52.383335 12 controller.go:120] Shutting down OpenAPI V3 controller I0313 01:20:52.383356 12 customresource_discovery_controller.go:328] Shutting down DiscoveryController I0313 01:20:52.383378 12 autoregister_controller.go:168] Shutting down autoregister controller I0313 01:20:52.383402 12 crdregistration_controller.go:146] Shutting down crd-autoregister controller I0313 01:20:52.383452 12 controller.go:170] Shutting down OpenAPI controller I0313 01:20:52.383469 12 apiapproval_controller.go:201] Shutting down KubernetesAPIApprovalPolicyConformantConditionController I0313 01:20:52.383474 12 apiaccess_count_controller.go:95] Shutting down APIRequestCount controller. I0313 01:20:52.383488 12 nonstructuralschema_controller.go:207] Shutting down NonStructuralSchemaConditionController I0313 01:20:52.383505 12 crd_finalizer.go:281] Shutting down CRDFinalizer I0313 01:20:52.383499 12 dynamic_cafile_content.go:175] "Shutting down controller" name="request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt" I0313 01:20:52.383519 12 establishing_controller.go:92] Shutting down EstablishingController I0313 01:20:52.383533 12 naming_controller.go:305] Shutting down NamingConditionController I0313 01:20:52.383554 12 apiservice_controller.go:134] Shutting down APIServiceRegistrationController I0313 01:20:52.383564 12 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt" I0313 01:20:52.383571 12 remote_available_controller.go:449] Shutting down RemoteAvailability controller I0313 01:20:52.383589 12 system_namespaces_controller.go:76] Shutting down system namespaces controller I0313 01:20:52.383601 12 local_available_controller.go:172] Shutting down LocalAvailability controller I0313 01:20:52.383385 12 controller.go:84] Shutting down OpenAPI AggregationController I0313 01:20:52.383831 12 controller.go:176] quota evaluator worker shutdown I0313 01:20:52.383844 12 controller.go:176] quota evaluator worker shutdown I0313 01:20:52.383909 12 controller.go:86] Shutting down OpenAPI V3 AggregationController I0313 01:20:52.383956 12 object_count_tracker.go:151] "StorageObjectCountTracker pruner is exiting" I0313 01:20:52.383998 12 dynamic_serving_content.go:149] "Shutting down controller" name="serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key" I0313 01:20:52.384045 12 controller.go:176] quota evaluator worker shutdown I0313 01:20:52.384063 12 controller.go:176] quota evaluator worker shutdown I0313 01:20:52.384099 12 tlsconfig.go:258] "Shutting down DynamicServingCertificateController" I0313 01:20:52.384194 12 secure_serving.go:258] Stopped listening on [::]:6443 I0313 01:20:52.384225 12 genericapiserver.go:624] "[graceful-termination] shutdown event" name="HTTPServerStoppedListening" I0313 01:20:52.384252 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'HTTPServerStoppedListening' HTTP Server has stopped listening I0313 01:20:52.384844 12 cluster_authentication_trust_controller.go:466] Shutting down cluster_authentication_trust_controller controller I0313 01:20:52.384893 12 storage_flowcontrol.go:186] APF bootstrap ensurer is exiting E0313 01:20:52.385078 12 wrap.go:53] "Timeout or abort while handling" logger="UnhandledError" method="GET" URI="/apis/user.openshift.io/v1/groups?allowWatchBookmarks=true&resourceVersion=11508&timeout=8m55s&timeoutSeconds=535&watch=true" auditID="9ea9f070-8d3a-4b9a-9817-ac17ae570416" I0313 01:20:52.385166 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" I0313 01:20:52.385413 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.crt::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.key" I0313 01:20:52.385421 12 dynamic_cafile_content.go:175] "Shutting down controller" name="request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt" I0313 01:20:52.385445 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.key" I0313 01:20:52.385474 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key" I0313 01:20:52.385557 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.key" I0313 01:20:52.385581 12 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt" I0313 01:20:52.394984 12 genericapiserver.go:735] "[graceful-termination] audit backend shutdown completed" I0313 01:20:54.384216 12 genericapiserver.go:742] [graceful-termination] apiserver is exiting I0313 01:20:54.384270 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TerminationGracefulTerminationFinished' All pending requests processed W0313 01:20:54.387866 12 cacher.go:171] Terminating all watchers from cacher apiservices.apiregistration.k8s.io W0313 01:20:54.388780 12 cacher.go:171] Terminating all watchers from cacher configmaps W0313 01:20:54.401226 12 cacher.go:171] Terminating all watchers from cacher resourcequotas W0313 01:20:54.402334 12 cacher.go:171] Terminating all watchers from cacher podtemplates W0313 01:20:54.405107 12 cacher.go:171] Terminating all watchers from cacher persistentvolumes W0313 01:20:54.406158 12 cacher.go:171] Terminating all watchers from cacher pods W0313 01:20:54.410335 12 cacher.go:171] Terminating all watchers from cacher secrets W0313 01:20:54.416070 12 cacher.go:171] Terminating all watchers from cacher endpoints W0313 01:20:54.418592 12 cacher.go:171] Terminating all watchers from cacher nodes W0313 01:20:54.420147 12 cacher.go:171] Terminating all watchers from cacher replicationcontrollers W0313 01:20:54.420662 12 cacher.go:171] Terminating all watchers from cacher services W0313 01:20:54.424105 12 cacher.go:171] Terminating all watchers from cacher limitranges W0313 01:20:54.425073 12 cacher.go:171] Terminating all watchers from cacher persistentvolumeclaims W0313 01:20:54.425863 12 cacher.go:171] Terminating all watchers from cacher serviceaccounts W0313 01:20:54.428158 12 cacher.go:171] Terminating all watchers from cacher namespaces W0313 01:20:54.433313 12 cacher.go:171] Terminating all watchers from cacher horizontalpodautoscalers.autoscaling W0313 01:20:54.434926 12 cacher.go:171] Terminating all watchers from cacher jobs.batch W0313 01:20:54.436375 12 cacher.go:171] Terminating all watchers from cacher cronjobs.batch W0313 01:20:54.437249 12 cacher.go:171] Terminating all watchers from cacher certificatesigningrequests.certificates.k8s.io W0313 01:20:54.437934 12 cacher.go:171] Terminating all watchers from cacher leases.coordination.k8s.io W0313 01:20:54.438703 12 cacher.go:171] Terminating all watchers from cacher endpointslices.discovery.k8s.io W0313 01:20:54.441102 12 cacher.go:171] Terminating all watchers from cacher networkpolicies.networking.k8s.io W0313 01:20:54.441713 12 cacher.go:171] Terminating all watchers from cacher ingresses.networking.k8s.io W0313 01:20:54.442795 12 cacher.go:171] Terminating all watchers from cacher ingressclasses.networking.k8s.io W0313 01:20:54.443621 12 cacher.go:171] Terminating all watchers from cacher runtimeclasses.node.k8s.io W0313 01:20:54.444598 12 cacher.go:171] Terminating all watchers from cacher poddisruptionbudgets.policy W0313 01:20:54.445430 12 cacher.go:171] Terminating all watchers from cacher clusterroles.rbac.authorization.k8s.io W0313 01:20:54.446087 12 cacher.go:171] Terminating all watchers from cacher clusterrolebindings.rbac.authorization.k8s.io W0313 01:20:54.447624 12 cacher.go:171] Terminating all watchers from cacher roles.rbac.authorization.k8s.io W0313 01:20:54.448752 12 cacher.go:171] Terminating all watchers from cacher rolebindings.rbac.authorization.k8s.io W0313 01:20:54.451146 12 cacher.go:171] Terminating all watchers from cacher csidrivers.storage.k8s.io W0313 01:20:54.453336 12 cacher.go:171] Terminating all watchers from cacher csistoragecapacities.storage.k8s.io W0313 01:20:54.454434 12 cacher.go:171] Terminating all watchers from cacher storageclasses.storage.k8s.io W0313 01:20:54.455243 12 cacher.go:171] Terminating all watchers from cacher volumeattachments.storage.k8s.io W0313 01:20:54.455896 12 cacher.go:171] Terminating all watchers from cacher csinodes.storage.k8s.io W0313 01:20:54.458337 12 cacher.go:171] Terminating all watchers from cacher daemonsets.apps W0313 01:20:54.459339 12 cacher.go:171] Terminating all watchers from cacher replicasets.apps W0313 01:20:54.460539 12 cacher.go:171] Terminating all watchers from cacher statefulsets.apps W0313 01:20:54.461111 12 cacher.go:171] Terminating all watchers from cacher controllerrevisions.apps W0313 01:20:54.462056 12 cacher.go:171] Terminating all watchers from cacher deployments.apps W0313 01:20:54.464530 12 cacher.go:171] Terminating all watchers from cacher validatingadmissionpolicybindings.admissionregistration.k8s.io W0313 01:20:54.464951 12 cacher.go:171] Terminating all watchers from cacher validatingwebhookconfigurations.admissionregistration.k8s.io W0313 01:20:54.465507 12 cacher.go:171] Terminating all watchers from cacher mutatingwebhookconfigurations.admissionregistration.k8s.io W0313 01:20:54.467192 12 cacher.go:171] Terminating all watchers from cacher validatingadmissionpolicies.admissionregistration.k8s.io W0313 01:20:54.468994 12 cacher.go:171] Terminating all watchers from cacher customresourcedefinitions.apiextensions.k8s.io W0313 01:20:54.470286 12 cacher.go:171] Terminating all watchers from cacher clusterextensions.olm.operatorframework.io W0313 01:20:54.471237 12 cacher.go:171] Terminating all watchers from cacher consoles.config.openshift.io W0313 01:20:54.471975 12 cacher.go:171] Terminating all watchers from cacher alertmanagerconfigs.monitoring.coreos.com W0313 01:20:54.472472 12 cacher.go:171] Terminating all watchers from cacher alertmanagerconfigs.monitoring.coreos.com W0313 01:20:54.473057 12 cacher.go:171] Terminating all watchers from cacher images.config.openshift.io W0313 01:20:54.474059 12 cacher.go:171] Terminating all watchers from cacher machineconfigpools.machineconfiguration.openshift.io W0313 01:20:54.475785 12 cacher.go:171] Terminating all watchers from cacher kubecontrollermanagers.operator.openshift.io W0313 01:20:54.476512 12 cacher.go:171] Terminating all watchers from cacher catalogsources.operators.coreos.com W0313 01:20:54.476998 12 cacher.go:171] Terminating all watchers from cacher thanosrulers.monitoring.coreos.com W0313 01:20:54.477598 12 cacher.go:171] Terminating all watchers from cacher machineautoscalers.autoscaling.openshift.io W0313 01:20:54.478716 12 cacher.go:171] Terminating all watchers from cacher clustercatalogs.olm.operatorframework.io W0313 01:20:54.479243 12 cacher.go:171] Terminating all watchers from cacher servicemonitors.monitoring.coreos.com W0313 01:20:54.479988 12 cacher.go:171] Terminating all watchers from cacher clusterserviceversions.operators.coreos.com W0313 01:20:54.481273 12 cacher.go:171] Terminating all watchers from cacher clusterautoscalers.autoscaling.openshift.io W0313 01:20:54.481896 12 cacher.go:171] Terminating all watchers from cacher olmconfigs.operators.coreos.com W0313 01:20:54.482843 12 cacher.go:171] Terminating all watchers from cacher alertrelabelconfigs.monitoring.openshift.io W0313 01:20:54.483527 12 cacher.go:171] Terminating all watchers from cacher bmceventsubscriptions.metal3.io W0313 01:20:54.484856 12 cacher.go:171] Terminating all watchers from cacher storageversionmigrations.migration.k8s.io W0313 01:20:54.485653 12 cacher.go:171] Terminating all watchers from cacher cloudcredentials.operator.openshift.io W0313 01:20:54.486129 12 cacher.go:171] Terminating all watchers from cacher dataimages.metal3.io W0313 01:20:54.486680 12 cacher.go:171] Terminating all watchers from cacher podmonitors.monitoring.coreos.com W0313 01:20:54.487729 12 cacher.go:171] Terminating all watchers from cacher volumesnapshots.snapshot.storage.k8s.io W0313 01:20:54.489134 12 cacher.go:171] Terminating all watchers from cacher ipaddressclaims.ipam.cluster.x-k8s.io W0313 01:20:54.489527 12 cacher.go:171] Terminating all watchers from cacher configs.samples.operator.openshift.io W0313 01:20:54.490044 12 cacher.go:171] Terminating all watchers from cacher firmwareschemas.metal3.io W0313 01:20:54.490457 12 cacher.go:171] Terminating all watchers from cacher openshiftapiservers.operator.openshift.io W0313 01:20:54.491132 12 cacher.go:171] Terminating all watchers from cacher metal3remediations.infrastructure.cluster.x-k8s.io W0313 01:20:54.491599 12 cacher.go:171] Terminating all watchers from cacher authentications.config.openshift.io W0313 01:20:54.492153 12 cacher.go:171] Terminating all watchers from cacher configs.imageregistry.operator.openshift.io W0313 01:20:54.493311 12 cacher.go:171] Terminating all watchers from cacher imagecontentsourcepolicies.operator.openshift.io W0313 01:20:54.493992 12 cacher.go:171] Terminating all watchers from cacher egressqoses.k8s.ovn.org W0313 01:20:54.494871 12 cacher.go:171] Terminating all watchers from cacher alertingrules.monitoring.openshift.io W0313 01:20:54.495192 12 cacher.go:171] Terminating all watchers from cacher hardwaredata.metal3.io W0313 01:20:54.495553 12 cacher.go:171] Terminating all watchers from cacher egressservices.k8s.ovn.org W0313 01:20:54.496823 12 cacher.go:171] Terminating all watchers from cacher openshiftcontrollermanagers.operator.openshift.io W0313 01:20:54.497696 12 cacher.go:171] Terminating all watchers from cacher egressrouters.network.operator.openshift.io W0313 01:20:54.498422 12 cacher.go:171] Terminating all watchers from cacher overlappingrangeipreservations.whereabouts.cni.cncf.io W0313 01:20:54.499017 12 cacher.go:171] Terminating all watchers from cacher schedulers.config.openshift.io W0313 01:20:54.499471 12 cacher.go:171] Terminating all watchers from cacher infrastructures.config.openshift.io W0313 01:20:54.500343 12 cacher.go:171] Terminating all watchers from cacher operatorhubs.config.openshift.io W0313 01:20:54.500781 12 cacher.go:171] Terminating all watchers from cacher alertmanagers.monitoring.coreos.com W0313 01:20:54.501168 12 cacher.go:171] Terminating all watchers from cacher hostfirmwaresettings.metal3.io W0313 01:20:54.501730 12 cacher.go:171] Terminating all watchers from cacher userdefinednetworks.k8s.ovn.org W0313 01:20:54.502499 12 cacher.go:171] Terminating all watchers from cacher nodes.config.openshift.io W0313 01:20:54.504778 12 cacher.go:171] Terminating all watchers from cacher preprovisioningimages.metal3.io W0313 01:20:54.505622 12 cacher.go:171] Terminating all watchers from cacher oauths.config.openshift.io W0313 01:20:54.506136 12 cacher.go:171] Terminating all watchers from cacher prometheusrules.monitoring.coreos.com W0313 01:20:54.506776 12 cacher.go:171] Terminating all watchers from cacher operators.operators.coreos.com W0313 01:20:54.507171 12 cacher.go:171] Terminating all watchers from cacher machineosconfigs.machineconfiguration.openshift.io W0313 01:20:54.507782 12 cacher.go:171] Terminating all watchers from cacher machinesets.machine.openshift.io W0313 01:20:54.508275 12 cacher.go:171] Terminating all watchers from cacher kubeapiservers.operator.openshift.io W0313 01:20:54.509285 12 cacher.go:171] Terminating all watchers from cacher machines.machine.openshift.io W0313 01:20:54.510001 12 cacher.go:171] Terminating all watchers from cacher operatorpkis.network.operator.openshift.io W0313 01:20:54.512015 12 cacher.go:171] Terminating all watchers from cacher ingresscontrollers.operator.openshift.io W0313 01:20:54.512620 12 cacher.go:171] Terminating all watchers from cacher dnsrecords.ingress.operator.openshift.io W0313 01:20:54.513908 12 cacher.go:171] Terminating all watchers from cacher installplans.operators.coreos.com W0313 01:20:54.514563 12 cacher.go:171] Terminating all watchers from cacher dnses.operator.openshift.io W0313 01:20:54.515207 12 cacher.go:171] Terminating all watchers from cacher clusterversions.config.openshift.io W0313 01:20:54.519022 12 cacher.go:171] Terminating all watchers from cacher machineconfigs.machineconfiguration.openshift.io W0313 01:20:54.520510 12 cacher.go:171] Terminating all watchers from cacher kubeletconfigs.machineconfiguration.openshift.io W0313 01:20:54.521261 12 cacher.go:171] Terminating all watchers from cacher metal3remediationtemplates.infrastructure.cluster.x-k8s.io W0313 01:20:54.521712 12 cacher.go:171] Terminating all watchers from cacher consoleplugins.console.openshift.io W0313 01:20:54.522468 12 cacher.go:171] Terminating all watchers from cacher adminpolicybasedexternalroutes.k8s.ovn.org W0313 01:20:54.522923 12 cacher.go:171] Terminating all watchers from cacher network-attachment-definitions.k8s.cni.cncf.io W0313 01:20:54.523546 12 cacher.go:171] Terminating all watchers from cacher rolebindingrestrictions.authorization.openshift.io W0313 01:20:54.524042 12 cacher.go:171] Terminating all watchers from cacher clusteruserdefinednetworks.k8s.ovn.org W0313 01:20:54.524538 12 cacher.go:171] Terminating all watchers from cacher egressfirewalls.k8s.ovn.org W0313 01:20:54.525036 12 cacher.go:171] Terminating all watchers from cacher clusteroperators.config.openshift.io W0313 01:20:54.526165 12 cacher.go:171] Terminating all watchers from cacher consoleclidownloads.console.openshift.io W0313 01:20:54.526645 12 cacher.go:171] Terminating all watchers from cacher nodeslicepools.whereabouts.cni.cncf.io W0313 01:20:54.527064 12 cacher.go:171] Terminating all watchers from cacher networks.config.openshift.io W0313 01:20:54.527612 12 cacher.go:171] Terminating all watchers from cacher dnses.config.openshift.io W0313 01:20:54.528239 12 cacher.go:171] Terminating all watchers from cacher ingresses.config.openshift.io W0313 01:20:54.529275 12 cacher.go:171] Terminating all watchers from cacher builds.config.openshift.io W0313 01:20:54.530846 12 cacher.go:171] Terminating all watchers from cacher baremetalhosts.metal3.io W0313 01:20:54.531937 12 cacher.go:171] Terminating all watchers from cacher machinehealthchecks.machine.openshift.io W0313 01:20:54.532326 12 cacher.go:171] Terminating all watchers from cacher csisnapshotcontrollers.operator.openshift.io W0313 01:20:54.533209 12 cacher.go:171] Terminating all watchers from cacher credentialsrequests.cloudcredential.openshift.io W0313 01:20:54.533528 12 cacher.go:171] Terminating all watchers from cacher controlplanemachinesets.machine.openshift.io W0313 01:20:54.533925 12 cacher.go:171] Terminating all watchers from cacher profiles.tuned.openshift.io W0313 01:20:54.535043 12 cacher.go:171] Terminating all watchers from cacher featuregates.config.openshift.io W0313 01:20:54.536820 12 cacher.go:171] Terminating all watchers from cacher containerruntimeconfigs.machineconfiguration.openshift.io W0313 01:20:54.537356 12 cacher.go:171] Terminating all watchers from cacher tuneds.tuned.openshift.io W0313 01:20:54.540675 12 cacher.go:171] Terminating all watchers from cacher clusterresourcequotas.quota.openshift.io W0313 01:20:54.541525 12 cacher.go:171] Terminating all watchers from cacher volumesnapshotcontents.snapshot.storage.k8s.io W0313 01:20:54.542295 12 cacher.go:171] Terminating all watchers from cacher proxies.config.openshift.io W0313 01:20:54.542722 12 cacher.go:171] Terminating all watchers from cacher securitycontextconstraints.security.openshift.io W0313 01:20:54.543791 12 cacher.go:171] Terminating all watchers from cacher insightsoperators.operator.openshift.io W0313 01:20:54.544316 12 cacher.go:171] Terminating all watchers from cacher projecthelmchartrepositories.helm.openshift.io W0313 01:20:54.545643 12 cacher.go:171] Terminating all watchers from cacher ipaddresses.ipam.cluster.x-k8s.io W0313 01:20:54.545957 12 cacher.go:171] Terminating all watchers from cacher networks.operator.openshift.io W0313 01:20:54.546506 12 cacher.go:171] Terminating all watchers from cacher consoles.operator.openshift.io W0313 01:20:54.547129 12 cacher.go:171] Terminating all watchers from cacher imagedigestmirrorsets.config.openshift.io W0313 01:20:54.547725 12 cacher.go:171] Terminating all watchers from cacher operatorgroups.operators.coreos.com W0313 01:20:54.549962 12 cacher.go:171] Terminating all watchers from cacher performanceprofiles.performance.openshift.io W0313 01:20:54.550452 12 cacher.go:171] Terminating all watchers from cacher ipamclaims.k8s.cni.cncf.io W0313 01:20:54.551027 12 cacher.go:171] Terminating all watchers from cacher adminnetworkpolicies.policy.networking.k8s.io W0313 01:20:54.551444 12 cacher.go:171] Terminating all watchers from cacher provisionings.metal3.io W0313 01:20:54.552176 12 cacher.go:171] Terminating all watchers from cacher probes.monitoring.coreos.com W0313 01:20:54.553452 12 cacher.go:171] Terminating all watchers from cacher apiservers.config.openshift.io W0313 01:20:54.554712 12 cacher.go:171] Terminating all watchers from cacher operatorconditions.operators.coreos.com W0313 01:20:54.555167 12 cacher.go:171] Terminating all watchers from cacher ippools.whereabouts.cni.cncf.io W0313 01:20:54.555474 12 cacher.go:171] Terminating all watchers from cacher hostupdatepolicies.metal3.io W0313 01:20:54.555955 12 cacher.go:171] Terminating all watchers from cacher egressips.k8s.ovn.org W0313 01:20:54.556680 12 cacher.go:171] Terminating all watchers from cacher baselineadminnetworkpolicies.policy.networking.k8s.io W0313 01:20:54.556988 12 cacher.go:171] Terminating all watchers from cacher volumesnapshotclasses.snapshot.storage.k8s.io W0313 01:20:54.558372 12 cacher.go:171] Terminating all watchers from cacher controllerconfigs.machineconfiguration.openshift.io W0313 01:20:54.558987 12 cacher.go:171] Terminating all watchers from cacher prometheuses.monitoring.coreos.com W0313 01:20:54.559251 12 cacher.go:171] Terminating all watchers from cacher subscriptions.operators.coreos.com W0313 01:20:54.560817 12 cacher.go:171] Terminating all watchers from cacher imagetagmirrorsets.config.openshift.io W0313 01:20:54.561320 12 cacher.go:171] Terminating all watchers from cacher machineconfigurations.operator.openshift.io W0313 01:20:54.562112 12 cacher.go:171] Terminating all watchers from cacher hostfirmwarecomponents.metal3.io W0313 01:20:54.562808 12 cacher.go:171] Terminating all watchers from cacher imagepruners.imageregistry.operator.openshift.io I0313 01:20:54.595586 1 main.go:235] Termination finished with exit code 0 I0313 01:20:54.595850 1 main.go:188] Deleting termination lock file "/var/log/kube-apiserver/.terminating" --- I0313 01:30:07.547569 1 main.go:213] Received signal terminated. Forwarding to sub-process "hyperkube". I0313 01:30:07.548437 12 genericapiserver.go:554] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0313 01:30:07.548453 12 controller.go:128] Shutting down kubernetes service endpoint reconciler I0313 01:30:07.548507 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ShutdownInitiated' Received signal to terminate, becoming unready, but keeping serving I0313 01:30:07.555735 12 genericapiserver.go:544] "[graceful-termination] shutdown event" name="AfterShutdownDelayDuration" I0313 01:30:07.555799 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'AfterShutdownDelayDuration' The minimal shutdown duration of 0s finished W0313 01:30:07.560913 12 lease.go:265] Resetting endpoints for master service "kubernetes" to [] I0313 01:30:07.584870 12 genericapiserver.go:721] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0313 01:30:07.584922 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TerminationPreShutdownHooksFinished' All pre-shutdown hooks have been finished I0313 01:30:07.585143 12 genericapiserver.go:633] "[graceful-termination] shutdown event" name="NotAcceptingNewRequest" I0313 01:30:07.585173 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'InFlightRequestsDrained' All non long-running request(s) in-flight have drained I0313 01:30:07.585331 12 genericapiserver.go:679] "[graceful-termination] not going to wait for active watch request(s) to drain" I0313 01:30:07.631295 12 genericapiserver.go:670] [graceful-termination] in-flight non long-running request(s) have drained I0313 01:30:07.631338 12 genericapiserver.go:711] "[graceful-termination] shutdown event" name="InFlightRequestsDrained" I0313 01:30:07.631393 12 genericapiserver.go:735] "[graceful-termination] audit backend shutdown completed" I0313 01:30:07.631431 12 controller.go:86] Shutting down OpenAPI V3 AggregationController I0313 01:30:07.631453 12 controller.go:157] Shutting down quota evaluator I0313 01:30:07.631461 12 controller.go:176] quota evaluator worker shutdown I0313 01:30:07.632027 12 storage_flowcontrol.go:186] APF bootstrap ensurer is exiting I0313 01:30:07.632143 12 apiservice_controller.go:134] Shutting down APIServiceRegistrationController I0313 01:30:07.632170 12 apf_controller.go:389] Shutting down API Priority and Fairness config worker I0313 01:30:07.632195 12 controller.go:132] Ending legacy_token_tracking_controller I0313 01:30:07.632196 12 dynamic_cafile_content.go:175] "Shutting down controller" name="request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt" I0313 01:30:07.632205 12 controller.go:133] Shutting down legacy_token_tracking_controller I0313 01:30:07.632237 12 clusterquotamapping.go:142] Shutting down ClusterQuotaMappingController controller I0313 01:30:07.632278 12 controller.go:120] Shutting down OpenAPI V3 controller I0313 01:30:07.632296 12 customresource_discovery_controller.go:328] Shutting down DiscoveryController I0313 01:30:07.632315 12 autoregister_controller.go:168] Shutting down autoregister controller I0313 01:30:07.632332 12 crdregistration_controller.go:146] Shutting down crd-autoregister controller I0313 01:30:07.632350 12 nonstructuralschema_controller.go:207] Shutting down NonStructuralSchemaConditionController I0313 01:30:07.632361 12 dynamic_serving_content.go:149] "Shutting down controller" name="aggregator-proxy-cert::/etc/kubernetes/static-pod-certs/secrets/aggregator-client/tls.crt::/etc/kubernetes/static-pod-certs/secrets/aggregator-client/tls.key" I0313 01:30:07.632367 12 controller.go:170] Shutting down OpenAPI controller I0313 01:30:07.632384 12 object_count_tracker.go:151] "StorageObjectCountTracker pruner is exiting" I0313 01:30:07.632384 12 apiapproval_controller.go:201] Shutting down KubernetesAPIApprovalPolicyConformantConditionController I0313 01:30:07.632392 12 crd_finalizer.go:281] Shutting down CRDFinalizer I0313 01:30:07.632410 12 dynamic_cafile_content.go:175] "Shutting down controller" name="request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt" I0313 01:30:07.632438 12 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt" I0313 01:30:07.633249 12 tlsconfig.go:258] "Shutting down DynamicServingCertificateController" I0313 01:30:07.632413 12 establishing_controller.go:92] Shutting down EstablishingController E0313 01:30:07.633670 12 wrap.go:53] "Timeout or abort while handling" logger="UnhandledError" method="GET" URI="/apis/user.openshift.io/v1/groups?allowWatchBookmarks=true&resourceVersion=16524&timeout=8m47s&timeoutSeconds=527&watch=true" auditID="ea34ad1c-d612-4511-94ae-926fa2832489" I0313 01:30:07.633781 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.key" I0313 01:30:07.632420 12 naming_controller.go:305] Shutting down NamingConditionController I0313 01:30:07.633937 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key" I0313 01:30:07.633976 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.key" I0313 01:30:07.633995 12 dynamic_serving_content.go:149] "Shutting down controller" name="serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key" I0313 01:30:07.634016 12 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt" I0313 01:30:07.634343 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.crt::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.key" I0313 01:30:07.634382 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" I0313 01:30:07.632432 12 apiaccess_count_controller.go:95] Shutting down APIRequestCount controller. I0313 01:30:07.632455 12 controller.go:84] Shutting down OpenAPI AggregationController I0313 01:30:07.632464 12 controller.go:176] quota evaluator worker shutdown I0313 01:30:07.632469 12 controller.go:176] quota evaluator worker shutdown I0313 01:30:07.632472 12 controller.go:176] quota evaluator worker shutdown I0313 01:30:07.632475 12 controller.go:176] quota evaluator worker shutdown I0313 01:30:07.632658 12 gc_controller.go:91] Shutting down apiserver lease garbage collector I0313 01:30:07.632672 12 remote_available_controller.go:449] Shutting down RemoteAvailability controller I0313 01:30:07.632679 12 local_available_controller.go:172] Shutting down LocalAvailability controller I0313 01:30:07.632684 12 system_namespaces_controller.go:76] Shutting down system namespaces controller I0313 01:30:07.633304 12 secure_serving.go:258] Stopped listening on [::]:6443 I0313 01:30:07.636425 12 genericapiserver.go:624] "[graceful-termination] shutdown event" name="HTTPServerStoppedListening" I0313 01:30:07.636443 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'HTTPServerStoppedListening' HTTP Server has stopped listening I0313 01:30:07.633319 12 cluster_authentication_trust_controller.go:466] Shutting down cluster_authentication_trust_controller controller I0313 01:30:09.633413 12 genericapiserver.go:742] [graceful-termination] apiserver is exiting I0313 01:30:09.633478 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TerminationGracefulTerminationFinished' All pending requests processed W0313 01:30:09.638769 12 cacher.go:171] Terminating all watchers from cacher apiservices.apiregistration.k8s.io W0313 01:30:09.640594 12 cacher.go:171] Terminating all watchers from cacher configmaps W0313 01:30:09.648123 12 cacher.go:171] Terminating all watchers from cacher replicationcontrollers W0313 01:30:09.654508 12 cacher.go:171] Terminating all watchers from cacher nodes W0313 01:30:09.662259 12 cacher.go:171] Terminating all watchers from cacher namespaces W0313 01:30:09.663334 12 cacher.go:171] Terminating all watchers from cacher persistentvolumeclaims W0313 01:30:09.676815 12 cacher.go:171] Terminating all watchers from cacher resourcequotas W0313 01:30:09.679091 12 cacher.go:171] Terminating all watchers from cacher limitranges W0313 01:30:09.681721 12 cacher.go:171] Terminating all watchers from cacher services W0313 01:30:09.685869 12 cacher.go:171] Terminating all watchers from cacher endpoints W0313 01:30:09.695227 12 cacher.go:171] Terminating all watchers from cacher persistentvolumes W0313 01:30:09.695965 12 cacher.go:171] Terminating all watchers from cacher secrets W0313 01:30:09.707051 12 cacher.go:171] Terminating all watchers from cacher serviceaccounts W0313 01:30:09.708347 12 cacher.go:171] Terminating all watchers from cacher pods W0313 01:30:09.712172 12 cacher.go:171] Terminating all watchers from cacher podtemplates W0313 01:30:09.712624 12 cacher.go:171] Terminating all watchers from cacher horizontalpodautoscalers.autoscaling W0313 01:30:09.714155 12 cacher.go:171] Terminating all watchers from cacher cronjobs.batch W0313 01:30:09.714708 12 cacher.go:171] Terminating all watchers from cacher jobs.batch W0313 01:30:09.715334 12 cacher.go:171] Terminating all watchers from cacher certificatesigningrequests.certificates.k8s.io W0313 01:30:09.715922 12 cacher.go:171] Terminating all watchers from cacher leases.coordination.k8s.io W0313 01:30:09.716345 12 cacher.go:171] Terminating all watchers from cacher endpointslices.discovery.k8s.io W0313 01:30:09.716839 12 cacher.go:171] Terminating all watchers from cacher ingresses.networking.k8s.io W0313 01:30:09.717252 12 cacher.go:171] Terminating all watchers from cacher ingressclasses.networking.k8s.io W0313 01:30:09.720024 12 cacher.go:171] Terminating all watchers from cacher networkpolicies.networking.k8s.io W0313 01:30:09.720567 12 cacher.go:171] Terminating all watchers from cacher runtimeclasses.node.k8s.io W0313 01:30:09.720962 12 cacher.go:171] Terminating all watchers from cacher poddisruptionbudgets.policy W0313 01:30:09.721848 12 cacher.go:171] Terminating all watchers from cacher rolebindings.rbac.authorization.k8s.io W0313 01:30:09.723562 12 cacher.go:171] Terminating all watchers from cacher clusterroles.rbac.authorization.k8s.io W0313 01:30:09.725419 12 cacher.go:171] Terminating all watchers from cacher clusterrolebindings.rbac.authorization.k8s.io W0313 01:30:09.726618 12 cacher.go:171] Terminating all watchers from cacher roles.rbac.authorization.k8s.io W0313 01:30:09.728388 12 cacher.go:171] Terminating all watchers from cacher priorityclasses.scheduling.k8s.io W0313 01:30:09.730192 12 cacher.go:171] Terminating all watchers from cacher storageclasses.storage.k8s.io W0313 01:30:09.730978 12 cacher.go:171] Terminating all watchers from cacher volumeattachments.storage.k8s.io W0313 01:30:09.731613 12 cacher.go:171] Terminating all watchers from cacher csinodes.storage.k8s.io W0313 01:30:09.732113 12 cacher.go:171] Terminating all watchers from cacher csidrivers.storage.k8s.io W0313 01:30:09.732771 12 cacher.go:171] Terminating all watchers from cacher csistoragecapacities.storage.k8s.io W0313 01:30:09.734508 12 cacher.go:171] Terminating all watchers from cacher flowschemas.flowcontrol.apiserver.k8s.io W0313 01:30:09.734898 12 cacher.go:171] Terminating all watchers from cacher prioritylevelconfigurations.flowcontrol.apiserver.k8s.io W0313 01:30:09.735247 12 cacher.go:171] Terminating all watchers from cacher controllerrevisions.apps W0313 01:30:09.735596 12 cacher.go:171] Terminating all watchers from cacher daemonsets.apps W0313 01:30:09.737958 12 cacher.go:171] Terminating all watchers from cacher replicasets.apps W0313 01:30:09.738377 12 cacher.go:171] Terminating all watchers from cacher deployments.apps W0313 01:30:09.739889 12 cacher.go:171] Terminating all watchers from cacher statefulsets.apps W0313 01:30:09.740254 12 cacher.go:171] Terminating all watchers from cacher validatingwebhookconfigurations.admissionregistration.k8s.io W0313 01:30:09.740867 12 cacher.go:171] Terminating all watchers from cacher mutatingwebhookconfigurations.admissionregistration.k8s.io W0313 01:30:09.741346 12 cacher.go:171] Terminating all watchers from cacher validatingadmissionpolicies.admissionregistration.k8s.io W0313 01:30:09.742169 12 cacher.go:171] Terminating all watchers from cacher validatingadmissionpolicybindings.admissionregistration.k8s.io W0313 01:30:09.744832 12 cacher.go:171] Terminating all watchers from cacher customresourcedefinitions.apiextensions.k8s.io W0313 01:30:09.745789 12 cacher.go:171] Terminating all watchers from cacher consoles.operator.openshift.io W0313 01:30:09.746345 12 cacher.go:171] Terminating all watchers from cacher clusterresourcequotas.quota.openshift.io W0313 01:30:09.747071 12 cacher.go:171] Terminating all watchers from cacher consoleyamlsamples.console.openshift.io W0313 01:30:09.747720 12 cacher.go:171] Terminating all watchers from cacher projecthelmchartrepositories.helm.openshift.io W0313 01:30:09.748173 12 cacher.go:171] Terminating all watchers from cacher consolelinks.console.openshift.io W0313 01:30:09.749240 12 cacher.go:171] Terminating all watchers from cacher performanceprofiles.performance.openshift.io W0313 01:30:09.749656 12 cacher.go:171] Terminating all watchers from cacher credentialsrequests.cloudcredential.openshift.io W0313 01:30:09.750367 12 cacher.go:171] Terminating all watchers from cacher ipaddressclaims.ipam.cluster.x-k8s.io W0313 01:30:09.750804 12 cacher.go:171] Terminating all watchers from cacher operatorgroups.operators.coreos.com W0313 01:30:09.751704 12 cacher.go:171] Terminating all watchers from cacher prometheusrules.monitoring.coreos.com W0313 01:30:09.752668 12 cacher.go:171] Terminating all watchers from cacher subscriptions.operators.coreos.com W0313 01:30:09.753437 12 cacher.go:171] Terminating all watchers from cacher ingresscontrollers.operator.openshift.io W0313 01:30:09.754604 12 cacher.go:171] Terminating all watchers from cacher clusterautoscalers.autoscaling.openshift.io W0313 01:30:09.755693 12 cacher.go:171] Terminating all watchers from cacher etcds.operator.openshift.io W0313 01:30:09.755979 12 cacher.go:171] Terminating all watchers from cacher egressrouters.network.operator.openshift.io W0313 01:30:09.756475 12 cacher.go:171] Terminating all watchers from cacher controlplanemachinesets.machine.openshift.io W0313 01:30:09.756924 12 cacher.go:171] Terminating all watchers from cacher baselineadminnetworkpolicies.policy.networking.k8s.io W0313 01:30:09.757335 12 cacher.go:171] Terminating all watchers from cacher consoleplugins.console.openshift.io W0313 01:30:09.758251 12 cacher.go:171] Terminating all watchers from cacher consolesamples.console.openshift.io W0313 01:30:09.758541 12 cacher.go:171] Terminating all watchers from cacher openshiftcontrollermanagers.operator.openshift.io W0313 01:30:09.758988 12 cacher.go:171] Terminating all watchers from cacher hostupdatepolicies.metal3.io W0313 01:30:09.759405 12 cacher.go:171] Terminating all watchers from cacher openshiftapiservers.operator.openshift.io W0313 01:30:09.760100 12 cacher.go:171] Terminating all watchers from cacher alertrelabelconfigs.monitoring.openshift.io W0313 01:30:09.760649 12 cacher.go:171] Terminating all watchers from cacher configs.imageregistry.operator.openshift.io W0313 01:30:09.761136 12 cacher.go:171] Terminating all watchers from cacher proxies.config.openshift.io W0313 01:30:09.762204 12 cacher.go:171] Terminating all watchers from cacher userdefinednetworks.k8s.ovn.org W0313 01:30:09.762495 12 cacher.go:171] Terminating all watchers from cacher ipamclaims.k8s.cni.cncf.io W0313 01:30:09.762933 12 cacher.go:171] Terminating all watchers from cacher nodes.config.openshift.io W0313 01:30:09.763742 12 cacher.go:171] Terminating all watchers from cacher metal3remediationtemplates.infrastructure.cluster.x-k8s.io W0313 01:30:09.764206 12 cacher.go:171] Terminating all watchers from cacher machineosbuilds.machineconfiguration.openshift.io W0313 01:30:09.764490 12 cacher.go:171] Terminating all watchers from cacher imagetagmirrorsets.config.openshift.io W0313 01:30:09.765212 12 cacher.go:171] Terminating all watchers from cacher csisnapshotcontrollers.operator.openshift.io W0313 01:30:09.765817 12 cacher.go:171] Terminating all watchers from cacher kubeschedulers.operator.openshift.io W0313 01:30:09.766402 12 cacher.go:171] Terminating all watchers from cacher consoleclidownloads.console.openshift.io W0313 01:30:09.766869 12 cacher.go:171] Terminating all watchers from cacher adminnetworkpolicies.policy.networking.k8s.io W0313 01:30:09.767152 12 cacher.go:171] Terminating all watchers from cacher storages.operator.openshift.io W0313 01:30:09.767556 12 cacher.go:171] Terminating all watchers from cacher egressfirewalls.k8s.ovn.org W0313 01:30:09.768276 12 cacher.go:171] Terminating all watchers from cacher consolenotifications.console.openshift.io W0313 01:30:09.768578 12 cacher.go:171] Terminating all watchers from cacher volumesnapshotclasses.snapshot.storage.k8s.io W0313 01:30:09.769321 12 cacher.go:171] Terminating all watchers from cacher adminpolicybasedexternalroutes.k8s.ovn.org W0313 01:30:09.769771 12 cacher.go:171] Terminating all watchers from cacher tuneds.tuned.openshift.io W0313 01:30:09.770475 12 cacher.go:171] Terminating all watchers from cacher preprovisioningimages.metal3.io W0313 01:30:09.770869 12 cacher.go:171] Terminating all watchers from cacher alertmanagerconfigs.monitoring.coreos.com W0313 01:30:09.771196 12 cacher.go:171] Terminating all watchers from cacher alertmanagerconfigs.monitoring.coreos.com W0313 01:30:09.771541 12 cacher.go:171] Terminating all watchers from cacher configs.samples.operator.openshift.io W0313 01:30:09.771943 12 cacher.go:171] Terminating all watchers from cacher consolequickstarts.console.openshift.io W0313 01:30:09.772384 12 cacher.go:171] Terminating all watchers from cacher dataimages.metal3.io W0313 01:30:09.772941 12 cacher.go:171] Terminating all watchers from cacher nodeslicepools.whereabouts.cni.cncf.io W0313 01:30:09.773280 12 cacher.go:171] Terminating all watchers from cacher hostfirmwaresettings.metal3.io W0313 01:30:09.773582 12 cacher.go:171] Terminating all watchers from cacher podmonitors.monitoring.coreos.com W0313 01:30:09.773999 12 cacher.go:171] Terminating all watchers from cacher servicecas.operator.openshift.io W0313 01:30:09.774321 12 cacher.go:171] Terminating all watchers from cacher configs.operator.openshift.io W0313 01:30:09.775063 12 cacher.go:171] Terminating all watchers from cacher kubeapiservers.operator.openshift.io W0313 01:30:09.775381 12 cacher.go:171] Terminating all watchers from cacher infrastructures.config.openshift.io W0313 01:30:09.776474 12 cacher.go:171] Terminating all watchers from cacher operators.operators.coreos.com W0313 01:30:09.776748 12 cacher.go:171] Terminating all watchers from cacher baremetalhosts.metal3.io W0313 01:30:09.777108 12 cacher.go:171] Terminating all watchers from cacher clusterversions.config.openshift.io W0313 01:30:09.778727 12 cacher.go:171] Terminating all watchers from cacher operatorpkis.network.operator.openshift.io W0313 01:30:09.779159 12 cacher.go:171] Terminating all watchers from cacher oauths.config.openshift.io W0313 01:30:09.779697 12 cacher.go:171] Terminating all watchers from cacher olmconfigs.operators.coreos.com W0313 01:30:09.780206 12 cacher.go:171] Terminating all watchers from cacher imagedigestmirrorsets.config.openshift.io W0313 01:30:09.780833 12 cacher.go:171] Terminating all watchers from cacher volumesnapshots.snapshot.storage.k8s.io W0313 01:30:09.781920 12 cacher.go:171] Terminating all watchers from cacher dnsrecords.ingress.operator.openshift.io W0313 01:30:09.782773 12 cacher.go:171] Terminating all watchers from cacher clusteroperators.config.openshift.io W0313 01:30:09.783836 12 cacher.go:171] Terminating all watchers from cacher clusteruserdefinednetworks.k8s.ovn.org W0313 01:30:09.784290 12 cacher.go:171] Terminating all watchers from cacher apiservers.config.openshift.io W0313 01:30:09.784907 12 cacher.go:171] Terminating all watchers from cacher machineconfigs.machineconfiguration.openshift.io W0313 01:30:09.785372 12 cacher.go:171] Terminating all watchers from cacher bmceventsubscriptions.metal3.io W0313 01:30:09.786039 12 cacher.go:171] Terminating all watchers from cacher ippools.whereabouts.cni.cncf.io W0313 01:30:09.786595 12 cacher.go:171] Terminating all watchers from cacher clusterextensions.olm.operatorframework.io W0313 01:30:09.787573 12 cacher.go:171] Terminating all watchers from cacher egressqoses.k8s.ovn.org W0313 01:30:09.788315 12 cacher.go:171] Terminating all watchers from cacher machinehealthchecks.machine.openshift.io W0313 01:30:09.789563 12 cacher.go:171] Terminating all watchers from cacher schedulers.config.openshift.io W0313 01:30:09.790311 12 cacher.go:171] Terminating all watchers from cacher builds.config.openshift.io W0313 01:30:09.791174 12 cacher.go:171] Terminating all watchers from cacher operatorconditions.operators.coreos.com W0313 01:30:09.791930 12 cacher.go:171] Terminating all watchers from cacher hardwaredata.metal3.io W0313 01:30:09.792272 12 cacher.go:171] Terminating all watchers from cacher probes.monitoring.coreos.com W0313 01:30:09.792797 12 cacher.go:171] Terminating all watchers from cacher images.config.openshift.io W0313 01:30:09.793382 12 cacher.go:171] Terminating all watchers from cacher profiles.tuned.openshift.io W0313 01:30:09.794229 12 cacher.go:171] Terminating all watchers from cacher consoles.config.openshift.io W0313 01:30:09.794678 12 cacher.go:171] Terminating all watchers from cacher ingresses.config.openshift.io W0313 01:30:09.795467 12 cacher.go:171] Terminating all watchers from cacher featuregates.config.openshift.io W0313 01:30:09.797600 12 cacher.go:171] Terminating all watchers from cacher rangeallocations.security.internal.openshift.io W0313 01:30:09.798235 12 cacher.go:171] Terminating all watchers from cacher operatorhubs.config.openshift.io W0313 01:30:09.798664 12 cacher.go:171] Terminating all watchers from cacher kubeletconfigs.machineconfiguration.openshift.io W0313 01:30:09.798993 12 cacher.go:171] Terminating all watchers from cacher prometheuses.monitoring.coreos.com W0313 01:30:09.799341 12 cacher.go:171] Terminating all watchers from cacher alertingrules.monitoring.openshift.io W0313 01:30:09.800488 12 cacher.go:171] Terminating all watchers from cacher kubestorageversionmigrators.operator.openshift.io W0313 01:30:09.801074 12 cacher.go:171] Terminating all watchers from cacher machineconfigpools.machineconfiguration.openshift.io W0313 01:30:09.802317 12 cacher.go:171] Terminating all watchers from cacher ipaddresses.ipam.cluster.x-k8s.io W0313 01:30:09.802845 12 cacher.go:171] Terminating all watchers from cacher installplans.operators.coreos.com W0313 01:30:09.803236 12 cacher.go:171] Terminating all watchers from cacher authentications.operator.openshift.io W0313 01:30:09.803749 12 cacher.go:171] Terminating all watchers from cacher insightsoperators.operator.openshift.io W0313 01:30:09.804199 12 cacher.go:171] Terminating all watchers from cacher storageversionmigrations.migration.k8s.io W0313 01:30:09.804627 12 cacher.go:171] Terminating all watchers from cacher networks.config.openshift.io W0313 01:30:09.805473 12 cacher.go:171] Terminating all watchers from cacher networks.operator.openshift.io W0313 01:30:09.805982 12 cacher.go:171] Terminating all watchers from cacher machineosconfigs.machineconfiguration.openshift.io W0313 01:30:09.806517 12 cacher.go:171] Terminating all watchers from cacher egressservices.k8s.ovn.org W0313 01:30:09.807127 12 cacher.go:171] Terminating all watchers from cacher alertmanagers.monitoring.coreos.com W0313 01:30:09.807583 12 cacher.go:171] Terminating all watchers from cacher overlappingrangeipreservations.whereabouts.cni.cncf.io W0313 01:30:09.807909 12 cacher.go:171] Terminating all watchers from cacher machineautoscalers.autoscaling.openshift.io W0313 01:30:09.808470 12 cacher.go:171] Terminating all watchers from cacher egressips.k8s.ovn.org W0313 01:30:09.808828 12 cacher.go:171] Terminating all watchers from cacher network-attachment-definitions.k8s.cni.cncf.io W0313 01:30:09.809289 12 cacher.go:171] Terminating all watchers from cacher authentications.config.openshift.io W0313 01:30:09.809592 12 cacher.go:171] Terminating all watchers from cacher clusterserviceversions.operators.coreos.com W0313 01:30:09.810576 12 cacher.go:171] Terminating all watchers from cacher clustercsidrivers.operator.openshift.io W0313 01:30:09.811016 12 cacher.go:171] Terminating all watchers from cacher containerruntimeconfigs.machineconfiguration.openshift.io W0313 01:30:09.811437 12 cacher.go:171] Terminating all watchers from cacher olms.operator.openshift.io W0313 01:30:09.811769 12 cacher.go:171] Terminating all watchers from cacher machines.machine.openshift.io W0313 01:30:09.812222 12 cacher.go:171] Terminating all watchers from cacher hostfirmwarecomponents.metal3.io W0313 01:30:09.812589 12 cacher.go:171] Terminating all watchers from cacher apirequestcounts.apiserver.openshift.io W0313 01:30:09.813000 12 cacher.go:171] Terminating all watchers from cacher catalogsources.operators.coreos.com W0313 01:30:09.813723 12 cacher.go:171] Terminating all watchers from cacher imagepruners.imageregistry.operator.openshift.io W0313 01:30:09.814187 12 cacher.go:171] Terminating all watchers from cacher kubecontrollermanagers.operator.openshift.io W0313 01:30:09.814705 12 cacher.go:171] Terminating all watchers from cacher securitycontextconstraints.security.openshift.io W0313 01:30:09.814978 12 cacher.go:171] Terminating all watchers from cacher imagecontentsourcepolicies.operator.openshift.io W0313 01:30:09.815308 12 cacher.go:171] Terminating all watchers from cacher volumesnapshotcontents.snapshot.storage.k8s.io W0313 01:30:09.816114 12 cacher.go:171] Terminating all watchers from cacher storagestates.migration.k8s.io W0313 01:30:09.816516 12 cacher.go:171] Terminating all watchers from cacher projects.config.openshift.io W0313 01:30:09.817042 12 cacher.go:171] Terminating all watchers from cacher dnses.config.openshift.io W0313 01:30:09.817430 12 cacher.go:171] Terminating all watchers from cacher thanosrulers.monitoring.coreos.com W0313 01:30:09.817943 12 cacher.go:171] Terminating all watchers from cacher imagecontentpolicies.config.openshift.io W0313 01:30:09.818473 12 cacher.go:171] Terminating all watchers from cacher servicemonitors.monitoring.coreos.com W0313 01:30:09.818924 12 cacher.go:171] Terminating all watchers from cacher provisionings.metal3.io W0313 01:30:09.819300 12 cacher.go:171] Terminating all watchers from cacher rolebindingrestrictions.authorization.openshift.io W0313 01:30:09.819718 12 cacher.go:171] Terminating all watchers from cacher helmchartrepositories.helm.openshift.io W0313 01:30:09.820068 12 cacher.go:171] Terminating all watchers from cacher machinesets.machine.openshift.io W0313 01:30:09.820603 12 cacher.go:171] Terminating all watchers from cacher clustercatalogs.olm.operatorframework.io W0313 01:30:09.821046 12 cacher.go:171] Terminating all watchers from cacher firmwareschemas.metal3.io W0313 01:30:09.821454 12 cacher.go:171] Terminating all watchers from cacher controllerconfigs.machineconfiguration.openshift.io W0313 01:30:09.822291 12 cacher.go:171] Terminating all watchers from cacher metal3remediations.infrastructure.cluster.x-k8s.io W0313 01:30:09.822789 12 cacher.go:171] Terminating all watchers from cacher cloudcredentials.operator.openshift.io W0313 01:30:09.823079 12 cacher.go:171] Terminating all watchers from cacher consoleexternalloglinks.console.openshift.io W0313 01:30:09.823378 12 cacher.go:171] Terminating all watchers from cacher dnses.operator.openshift.io W0313 01:30:09.823748 12 cacher.go:171] Terminating all watchers from cacher machineconfigurations.operator.openshift.io I0313 01:30:09.871407 1 main.go:235] Termination finished with exit code 0 I0313 01:30:09.871703 1 main.go:188] Deleting termination lock file "/var/log/kube-apiserver/.terminating" ---