--- I0313 10:43:11.423781 1 main.go:213] Received signal terminated. Forwarding to sub-process "hyperkube". I0313 10:43:11.424304 12 genericapiserver.go:554] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0313 10:43:11.424331 12 controller.go:128] Shutting down kubernetes service endpoint reconciler I0313 10:43:11.424368 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ShutdownInitiated' Received signal to terminate, becoming unready, but keeping serving I0313 10:43:11.430600 12 genericapiserver.go:544] "[graceful-termination] shutdown event" name="AfterShutdownDelayDuration" I0313 10:43:11.430642 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'AfterShutdownDelayDuration' The minimal shutdown duration of 0s finished W0313 10:43:11.435204 12 lease.go:265] Resetting endpoints for master service "kubernetes" to [] I0313 10:43:11.451216 12 genericapiserver.go:721] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0313 10:43:11.451340 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TerminationPreShutdownHooksFinished' All pre-shutdown hooks have been finished I0313 10:43:11.451579 12 genericapiserver.go:633] "[graceful-termination] shutdown event" name="NotAcceptingNewRequest" I0313 10:43:11.451640 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'InFlightRequestsDrained' All non long-running request(s) in-flight have drained I0313 10:43:11.451744 12 genericapiserver.go:679] "[graceful-termination] not going to wait for active watch request(s) to drain" I0313 10:43:11.482185 12 genericapiserver.go:670] [graceful-termination] in-flight non long-running request(s) have drained I0313 10:43:11.482239 12 genericapiserver.go:711] "[graceful-termination] shutdown event" name="InFlightRequestsDrained" I0313 10:43:11.482294 12 controller.go:157] Shutting down quota evaluator I0313 10:43:11.482303 12 controller.go:176] quota evaluator worker shutdown I0313 10:43:11.482322 12 genericapiserver.go:735] "[graceful-termination] audit backend shutdown completed" I0313 10:43:11.482387 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.key" I0313 10:43:11.482409 12 controller.go:176] quota evaluator worker shutdown I0313 10:43:11.482440 12 controller.go:132] Ending legacy_token_tracking_controller I0313 10:43:11.482452 12 controller.go:133] Shutting down legacy_token_tracking_controller I0313 10:43:11.482453 12 controller.go:176] quota evaluator worker shutdown I0313 10:43:11.482466 12 controller.go:176] quota evaluator worker shutdown I0313 10:43:11.482473 12 controller.go:176] quota evaluator worker shutdown I0313 10:43:11.482499 12 gc_controller.go:91] Shutting down apiserver lease garbage collector I0313 10:43:11.482527 12 apiaccess_count_controller.go:95] Shutting down APIRequestCount controller. I0313 10:43:11.482554 12 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt" I0313 10:43:11.482570 12 dynamic_cafile_content.go:175] "Shutting down controller" name="request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt" I0313 10:43:11.482599 12 object_count_tracker.go:151] "StorageObjectCountTracker pruner is exiting" I0313 10:43:11.482631 12 dynamic_serving_content.go:149] "Shutting down controller" name="aggregator-proxy-cert::/etc/kubernetes/static-pod-certs/secrets/aggregator-client/tls.crt::/etc/kubernetes/static-pod-certs/secrets/aggregator-client/tls.key" I0313 10:43:11.482664 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.crt::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.key" I0313 10:43:11.482678 12 dynamic_cafile_content.go:175] "Shutting down controller" name="request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt" I0313 10:43:11.482542 12 storage_flowcontrol.go:186] APF bootstrap ensurer is exiting I0313 10:43:11.482586 12 cluster_authentication_trust_controller.go:466] Shutting down cluster_authentication_trust_controller controller I0313 10:43:11.482661 12 dynamic_serving_content.go:149] "Shutting down controller" name="serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key" I0313 10:43:11.482627 12 controller.go:84] Shutting down OpenAPI AggregationController I0313 10:43:11.482659 12 controller.go:86] Shutting down OpenAPI V3 AggregationController I0313 10:43:11.482797 12 secure_serving.go:258] Stopped listening on [::]:6443 I0313 10:43:11.482814 12 genericapiserver.go:624] "[graceful-termination] shutdown event" name="HTTPServerStoppedListening" I0313 10:43:11.482829 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'HTTPServerStoppedListening' HTTP Server has stopped listening I0313 10:43:11.483010 12 tlsconfig.go:258] "Shutting down DynamicServingCertificateController" I0313 10:43:11.483116 12 clusterquotamapping.go:142] Shutting down ClusterQuotaMappingController controller I0313 10:43:11.483141 12 autoregister_controller.go:168] Shutting down autoregister controller I0313 10:43:11.483182 12 crdregistration_controller.go:146] Shutting down crd-autoregister controller I0313 10:43:11.483105 12 controller.go:120] Shutting down OpenAPI V3 controller I0313 10:43:11.483213 12 customresource_discovery_controller.go:328] Shutting down DiscoveryController I0313 10:43:11.483230 12 system_namespaces_controller.go:76] Shutting down system namespaces controller I0313 10:43:11.483234 12 controller.go:170] Shutting down OpenAPI controller I0313 10:43:11.483263 12 crd_finalizer.go:281] Shutting down CRDFinalizer I0313 10:43:11.483277 12 local_available_controller.go:172] Shutting down LocalAvailability controller I0313 10:43:11.483283 12 naming_controller.go:305] Shutting down NamingConditionController I0313 10:43:11.483290 12 remote_available_controller.go:449] Shutting down RemoteAvailability controller I0313 10:43:11.483295 12 establishing_controller.go:92] Shutting down EstablishingController I0313 10:43:11.483313 12 nonstructuralschema_controller.go:207] Shutting down NonStructuralSchemaConditionController I0313 10:43:11.483346 12 apiservice_controller.go:134] Shutting down APIServiceRegistrationController I0313 10:43:11.483315 12 apf_controller.go:389] Shutting down API Priority and Fairness config worker I0313 10:43:11.483247 12 apiapproval_controller.go:201] Shutting down KubernetesAPIApprovalPolicyConformantConditionController I0313 10:43:11.484186 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.key" I0313 10:43:11.484486 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" I0313 10:43:11.484512 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key" I0313 10:43:11.485158 12 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt" E0313 10:43:11.485868 12 wrap.go:53] "Timeout or abort while handling" logger="UnhandledError" method="GET" URI="/apis/user.openshift.io/v1/groups?allowWatchBookmarks=true&resourceVersion=11258&timeout=9m26s&timeoutSeconds=566&watch=true" auditID="d3b410bd-74b6-4cb5-884d-2291cb48f79b" I0313 10:43:13.483564 12 genericapiserver.go:742] [graceful-termination] apiserver is exiting I0313 10:43:13.483640 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TerminationGracefulTerminationFinished' All pending requests processed W0313 10:43:13.486517 12 cacher.go:171] Terminating all watchers from cacher apiservices.apiregistration.k8s.io W0313 10:43:13.488149 12 cacher.go:171] Terminating all watchers from cacher namespaces W0313 10:43:13.490476 12 cacher.go:171] Terminating all watchers from cacher replicationcontrollers W0313 10:43:13.491389 12 cacher.go:171] Terminating all watchers from cacher services W0313 10:43:13.496978 12 cacher.go:171] Terminating all watchers from cacher secrets W0313 10:43:13.502407 12 cacher.go:171] Terminating all watchers from cacher configmaps W0313 10:43:13.507757 12 cacher.go:171] Terminating all watchers from cacher persistentvolumes W0313 10:43:13.513735 12 cacher.go:171] Terminating all watchers from cacher resourcequotas W0313 10:43:13.514446 12 cacher.go:171] Terminating all watchers from cacher serviceaccounts W0313 10:43:13.515991 12 cacher.go:171] Terminating all watchers from cacher endpoints W0313 10:43:13.529414 12 cacher.go:171] Terminating all watchers from cacher podtemplates W0313 10:43:13.530014 12 cacher.go:171] Terminating all watchers from cacher persistentvolumeclaims W0313 10:43:13.530726 12 cacher.go:171] Terminating all watchers from cacher nodes W0313 10:43:13.540512 12 cacher.go:171] Terminating all watchers from cacher limitranges W0313 10:43:13.604241 12 cacher.go:171] Terminating all watchers from cacher pods W0313 10:43:13.605600 12 cacher.go:171] Terminating all watchers from cacher horizontalpodautoscalers.autoscaling W0313 10:43:13.606198 12 cacher.go:171] Terminating all watchers from cacher jobs.batch W0313 10:43:13.606805 12 cacher.go:171] Terminating all watchers from cacher cronjobs.batch W0313 10:43:13.607438 12 cacher.go:171] Terminating all watchers from cacher certificatesigningrequests.certificates.k8s.io W0313 10:43:13.609551 12 cacher.go:171] Terminating all watchers from cacher leases.coordination.k8s.io W0313 10:43:13.610597 12 cacher.go:171] Terminating all watchers from cacher endpointslices.discovery.k8s.io W0313 10:43:13.611228 12 cacher.go:171] Terminating all watchers from cacher networkpolicies.networking.k8s.io W0313 10:43:13.612686 12 cacher.go:171] Terminating all watchers from cacher ingresses.networking.k8s.io W0313 10:43:13.613633 12 cacher.go:171] Terminating all watchers from cacher ingressclasses.networking.k8s.io W0313 10:43:13.614413 12 cacher.go:171] Terminating all watchers from cacher runtimeclasses.node.k8s.io W0313 10:43:13.615074 12 cacher.go:171] Terminating all watchers from cacher poddisruptionbudgets.policy W0313 10:43:13.616284 12 cacher.go:171] Terminating all watchers from cacher roles.rbac.authorization.k8s.io W0313 10:43:13.617428 12 cacher.go:171] Terminating all watchers from cacher rolebindings.rbac.authorization.k8s.io W0313 10:43:13.619031 12 cacher.go:171] Terminating all watchers from cacher clusterroles.rbac.authorization.k8s.io W0313 10:43:13.619919 12 cacher.go:171] Terminating all watchers from cacher clusterrolebindings.rbac.authorization.k8s.io W0313 10:43:13.620429 12 cacher.go:171] Terminating all watchers from cacher priorityclasses.scheduling.k8s.io W0313 10:43:13.620964 12 cacher.go:171] Terminating all watchers from cacher csidrivers.storage.k8s.io W0313 10:43:13.621551 12 cacher.go:171] Terminating all watchers from cacher csistoragecapacities.storage.k8s.io W0313 10:43:13.622007 12 cacher.go:171] Terminating all watchers from cacher storageclasses.storage.k8s.io W0313 10:43:13.623820 12 cacher.go:171] Terminating all watchers from cacher volumeattachments.storage.k8s.io W0313 10:43:13.624515 12 cacher.go:171] Terminating all watchers from cacher csinodes.storage.k8s.io W0313 10:43:13.625408 12 cacher.go:171] Terminating all watchers from cacher prioritylevelconfigurations.flowcontrol.apiserver.k8s.io W0313 10:43:13.625904 12 cacher.go:171] Terminating all watchers from cacher flowschemas.flowcontrol.apiserver.k8s.io W0313 10:43:13.626328 12 cacher.go:171] Terminating all watchers from cacher replicasets.apps W0313 10:43:13.627255 12 cacher.go:171] Terminating all watchers from cacher deployments.apps W0313 10:43:13.628983 12 cacher.go:171] Terminating all watchers from cacher statefulsets.apps W0313 10:43:13.629985 12 cacher.go:171] Terminating all watchers from cacher controllerrevisions.apps W0313 10:43:13.630554 12 cacher.go:171] Terminating all watchers from cacher daemonsets.apps W0313 10:43:13.632135 12 cacher.go:171] Terminating all watchers from cacher validatingwebhookconfigurations.admissionregistration.k8s.io W0313 10:43:13.632952 12 cacher.go:171] Terminating all watchers from cacher mutatingwebhookconfigurations.admissionregistration.k8s.io W0313 10:43:13.633685 12 cacher.go:171] Terminating all watchers from cacher validatingadmissionpolicies.admissionregistration.k8s.io W0313 10:43:13.634153 12 cacher.go:171] Terminating all watchers from cacher validatingadmissionpolicybindings.admissionregistration.k8s.io W0313 10:43:13.635577 12 cacher.go:171] Terminating all watchers from cacher customresourcedefinitions.apiextensions.k8s.io W0313 10:43:13.637197 12 cacher.go:171] Terminating all watchers from cacher hostupdatepolicies.metal3.io W0313 10:43:13.637688 12 cacher.go:171] Terminating all watchers from cacher clusteroperators.config.openshift.io W0313 10:43:13.638864 12 cacher.go:171] Terminating all watchers from cacher metal3remediations.infrastructure.cluster.x-k8s.io W0313 10:43:13.639923 12 cacher.go:171] Terminating all watchers from cacher dnses.operator.openshift.io W0313 10:43:13.640578 12 cacher.go:171] Terminating all watchers from cacher projects.config.openshift.io W0313 10:43:13.641076 12 cacher.go:171] Terminating all watchers from cacher cloudcredentials.operator.openshift.io W0313 10:43:13.641569 12 cacher.go:171] Terminating all watchers from cacher openshiftcontrollermanagers.operator.openshift.io W0313 10:43:13.642430 12 cacher.go:171] Terminating all watchers from cacher thanosrulers.monitoring.coreos.com W0313 10:43:13.643189 12 cacher.go:171] Terminating all watchers from cacher helmchartrepositories.helm.openshift.io W0313 10:43:13.643719 12 cacher.go:171] Terminating all watchers from cacher hostfirmwarecomponents.metal3.io W0313 10:43:13.644227 12 cacher.go:171] Terminating all watchers from cacher networks.config.openshift.io W0313 10:43:13.644606 12 cacher.go:171] Terminating all watchers from cacher machines.machine.openshift.io W0313 10:43:13.645017 12 cacher.go:171] Terminating all watchers from cacher kubestorageversionmigrators.operator.openshift.io W0313 10:43:13.645439 12 cacher.go:171] Terminating all watchers from cacher kubeschedulers.operator.openshift.io W0313 10:43:13.646025 12 cacher.go:171] Terminating all watchers from cacher consoleclidownloads.console.openshift.io W0313 10:43:13.646355 12 cacher.go:171] Terminating all watchers from cacher bmceventsubscriptions.metal3.io W0313 10:43:13.646897 12 cacher.go:171] Terminating all watchers from cacher provisionings.metal3.io W0313 10:43:13.647672 12 cacher.go:171] Terminating all watchers from cacher clusterautoscalers.autoscaling.openshift.io W0313 10:43:13.648323 12 cacher.go:171] Terminating all watchers from cacher consoles.operator.openshift.io W0313 10:43:13.648857 12 cacher.go:171] Terminating all watchers from cacher hostfirmwaresettings.metal3.io W0313 10:43:13.649308 12 cacher.go:171] Terminating all watchers from cacher ipamclaims.k8s.cni.cncf.io W0313 10:43:13.649718 12 cacher.go:171] Terminating all watchers from cacher machineosbuilds.machineconfiguration.openshift.io W0313 10:43:13.650198 12 cacher.go:171] Terminating all watchers from cacher userdefinednetworks.k8s.ovn.org W0313 10:43:13.650861 12 cacher.go:171] Terminating all watchers from cacher etcds.operator.openshift.io W0313 10:43:13.651262 12 cacher.go:171] Terminating all watchers from cacher subscriptions.operators.coreos.com W0313 10:43:13.651778 12 cacher.go:171] Terminating all watchers from cacher storagestates.migration.k8s.io W0313 10:43:13.652203 12 cacher.go:171] Terminating all watchers from cacher machineautoscalers.autoscaling.openshift.io W0313 10:43:13.652769 12 cacher.go:171] Terminating all watchers from cacher egressservices.k8s.ovn.org W0313 10:43:13.653421 12 cacher.go:171] Terminating all watchers from cacher nodes.config.openshift.io W0313 10:43:13.653952 12 cacher.go:171] Terminating all watchers from cacher baselineadminnetworkpolicies.policy.networking.k8s.io W0313 10:43:13.654752 12 cacher.go:171] Terminating all watchers from cacher imagedigestmirrorsets.config.openshift.io W0313 10:43:13.655168 12 cacher.go:171] Terminating all watchers from cacher volumesnapshotcontents.snapshot.storage.k8s.io W0313 10:43:13.655922 12 cacher.go:171] Terminating all watchers from cacher authentications.operator.openshift.io W0313 10:43:13.656286 12 cacher.go:171] Terminating all watchers from cacher alertrelabelconfigs.monitoring.openshift.io W0313 10:43:13.656820 12 cacher.go:171] Terminating all watchers from cacher configs.samples.operator.openshift.io W0313 10:43:13.657984 12 cacher.go:171] Terminating all watchers from cacher ipaddresses.ipam.cluster.x-k8s.io W0313 10:43:13.658478 12 cacher.go:171] Terminating all watchers from cacher machineconfigurations.operator.openshift.io W0313 10:43:13.658814 12 cacher.go:171] Terminating all watchers from cacher olmconfigs.operators.coreos.com W0313 10:43:13.659382 12 cacher.go:171] Terminating all watchers from cacher ingresses.config.openshift.io W0313 10:43:13.660349 12 cacher.go:171] Terminating all watchers from cacher machineconfigpools.machineconfiguration.openshift.io W0313 10:43:13.661970 12 cacher.go:171] Terminating all watchers from cacher network-attachment-definitions.k8s.cni.cncf.io W0313 10:43:13.662846 12 cacher.go:171] Terminating all watchers from cacher imagetagmirrorsets.config.openshift.io W0313 10:43:13.663516 12 cacher.go:171] Terminating all watchers from cacher overlappingrangeipreservations.whereabouts.cni.cncf.io W0313 10:43:13.663918 12 cacher.go:171] Terminating all watchers from cacher storageversionmigrations.migration.k8s.io W0313 10:43:13.664713 12 cacher.go:171] Terminating all watchers from cacher operatorhubs.config.openshift.io W0313 10:43:13.665389 12 cacher.go:171] Terminating all watchers from cacher volumesnapshots.snapshot.storage.k8s.io W0313 10:43:13.666279 12 cacher.go:171] Terminating all watchers from cacher performanceprofiles.performance.openshift.io W0313 10:43:13.666819 12 cacher.go:171] Terminating all watchers from cacher imagecontentpolicies.config.openshift.io W0313 10:43:13.667969 12 cacher.go:171] Terminating all watchers from cacher tuneds.tuned.openshift.io W0313 10:43:13.668545 12 cacher.go:171] Terminating all watchers from cacher proxies.config.openshift.io W0313 10:43:13.669314 12 cacher.go:171] Terminating all watchers from cacher configs.imageregistry.operator.openshift.io W0313 10:43:13.669772 12 cacher.go:171] Terminating all watchers from cacher alertingrules.monitoring.openshift.io W0313 10:43:13.670456 12 cacher.go:171] Terminating all watchers from cacher egressips.k8s.ovn.org W0313 10:43:13.671519 12 cacher.go:171] Terminating all watchers from cacher configs.operator.openshift.io W0313 10:43:13.672188 12 cacher.go:171] Terminating all watchers from cacher rangeallocations.security.internal.openshift.io W0313 10:43:13.672657 12 cacher.go:171] Terminating all watchers from cacher controllerconfigs.machineconfiguration.openshift.io W0313 10:43:13.673437 12 cacher.go:171] Terminating all watchers from cacher insightsoperators.operator.openshift.io W0313 10:43:13.673987 12 cacher.go:171] Terminating all watchers from cacher clustercatalogs.olm.operatorframework.io W0313 10:43:13.674587 12 cacher.go:171] Terminating all watchers from cacher clusterresourcequotas.quota.openshift.io W0313 10:43:13.674944 12 cacher.go:171] Terminating all watchers from cacher operatorpkis.network.operator.openshift.io W0313 10:43:13.675606 12 cacher.go:171] Terminating all watchers from cacher adminnetworkpolicies.policy.networking.k8s.io W0313 10:43:13.675976 12 cacher.go:171] Terminating all watchers from cacher servicemonitors.monitoring.coreos.com W0313 10:43:13.677179 12 cacher.go:171] Terminating all watchers from cacher consoles.config.openshift.io W0313 10:43:13.678041 12 cacher.go:171] Terminating all watchers from cacher machineconfigs.machineconfiguration.openshift.io W0313 10:43:13.678722 12 cacher.go:171] Terminating all watchers from cacher operators.operators.coreos.com W0313 10:43:13.679515 12 cacher.go:171] Terminating all watchers from cacher baremetalhosts.metal3.io W0313 10:43:13.680052 12 cacher.go:171] Terminating all watchers from cacher firmwareschemas.metal3.io W0313 10:43:13.680564 12 cacher.go:171] Terminating all watchers from cacher kubeapiservers.operator.openshift.io W0313 10:43:13.681182 12 cacher.go:171] Terminating all watchers from cacher machineosconfigs.machineconfiguration.openshift.io W0313 10:43:13.681806 12 cacher.go:171] Terminating all watchers from cacher storages.operator.openshift.io W0313 10:43:13.682423 12 cacher.go:171] Terminating all watchers from cacher consoleyamlsamples.console.openshift.io W0313 10:43:13.682875 12 cacher.go:171] Terminating all watchers from cacher podmonitors.monitoring.coreos.com W0313 10:43:13.683573 12 cacher.go:171] Terminating all watchers from cacher consolequickstarts.console.openshift.io W0313 10:43:13.684237 12 cacher.go:171] Terminating all watchers from cacher metal3remediationtemplates.infrastructure.cluster.x-k8s.io W0313 10:43:13.684722 12 cacher.go:171] Terminating all watchers from cacher dataimages.metal3.io W0313 10:43:13.685295 12 cacher.go:171] Terminating all watchers from cacher probes.monitoring.coreos.com W0313 10:43:13.686195 12 cacher.go:171] Terminating all watchers from cacher dnses.config.openshift.io W0313 10:43:13.687015 12 cacher.go:171] Terminating all watchers from cacher hardwaredata.metal3.io W0313 10:43:13.687619 12 cacher.go:171] Terminating all watchers from cacher networks.operator.openshift.io W0313 10:43:13.688088 12 cacher.go:171] Terminating all watchers from cacher openshiftapiservers.operator.openshift.io W0313 10:43:13.688606 12 cacher.go:171] Terminating all watchers from cacher egressrouters.network.operator.openshift.io W0313 10:43:13.689374 12 cacher.go:171] Terminating all watchers from cacher apiservers.config.openshift.io W0313 10:43:13.690618 12 cacher.go:171] Terminating all watchers from cacher builds.config.openshift.io W0313 10:43:13.691656 12 cacher.go:171] Terminating all watchers from cacher containerruntimeconfigs.machineconfiguration.openshift.io W0313 10:43:13.692505 12 cacher.go:171] Terminating all watchers from cacher adminpolicybasedexternalroutes.k8s.ovn.org W0313 10:43:13.692984 12 cacher.go:171] Terminating all watchers from cacher alertmanagers.monitoring.coreos.com W0313 10:43:13.693440 12 cacher.go:171] Terminating all watchers from cacher imagepruners.imageregistry.operator.openshift.io W0313 10:43:13.693801 12 cacher.go:171] Terminating all watchers from cacher clusterversions.config.openshift.io W0313 10:43:13.695738 12 cacher.go:171] Terminating all watchers from cacher clustercsidrivers.operator.openshift.io W0313 10:43:13.696146 12 cacher.go:171] Terminating all watchers from cacher infrastructures.config.openshift.io W0313 10:43:13.697695 12 cacher.go:171] Terminating all watchers from cacher credentialsrequests.cloudcredential.openshift.io W0313 10:43:13.698053 12 cacher.go:171] Terminating all watchers from cacher consolenotifications.console.openshift.io W0313 10:43:13.698408 12 cacher.go:171] Terminating all watchers from cacher ingresscontrollers.operator.openshift.io W0313 10:43:13.699392 12 cacher.go:171] Terminating all watchers from cacher securitycontextconstraints.security.openshift.io W0313 10:43:13.699939 12 cacher.go:171] Terminating all watchers from cacher ippools.whereabouts.cni.cncf.io W0313 10:43:13.700619 12 cacher.go:171] Terminating all watchers from cacher clusteruserdefinednetworks.k8s.ovn.org W0313 10:43:13.701335 12 cacher.go:171] Terminating all watchers from cacher featuregates.config.openshift.io W0313 10:43:13.703347 12 cacher.go:171] Terminating all watchers from cacher consolesamples.console.openshift.io W0313 10:43:13.703791 12 cacher.go:171] Terminating all watchers from cacher egressfirewalls.k8s.ovn.org W0313 10:43:13.704949 12 cacher.go:171] Terminating all watchers from cacher clusterserviceversions.operators.coreos.com W0313 10:43:13.705784 12 cacher.go:171] Terminating all watchers from cacher schedulers.config.openshift.io W0313 10:43:13.706462 12 cacher.go:171] Terminating all watchers from cacher imagecontentsourcepolicies.operator.openshift.io W0313 10:43:13.706945 12 cacher.go:171] Terminating all watchers from cacher projecthelmchartrepositories.helm.openshift.io W0313 10:43:13.707491 12 cacher.go:171] Terminating all watchers from cacher dnsrecords.ingress.operator.openshift.io W0313 10:43:13.708868 12 cacher.go:171] Terminating all watchers from cacher consoleplugins.console.openshift.io W0313 10:43:13.709993 12 cacher.go:171] Terminating all watchers from cacher ipaddressclaims.ipam.cluster.x-k8s.io W0313 10:43:13.710525 12 cacher.go:171] Terminating all watchers from cacher rolebindingrestrictions.authorization.openshift.io W0313 10:43:13.710872 12 cacher.go:171] Terminating all watchers from cacher oauths.config.openshift.io W0313 10:43:13.711259 12 cacher.go:171] Terminating all watchers from cacher olms.operator.openshift.io W0313 10:43:13.711571 12 cacher.go:171] Terminating all watchers from cacher prometheuses.monitoring.coreos.com W0313 10:43:13.711959 12 cacher.go:171] Terminating all watchers from cacher kubecontrollermanagers.operator.openshift.io W0313 10:43:13.712462 12 cacher.go:171] Terminating all watchers from cacher volumesnapshotclasses.snapshot.storage.k8s.io W0313 10:43:13.713253 12 cacher.go:171] Terminating all watchers from cacher images.config.openshift.io W0313 10:43:13.714193 12 cacher.go:171] Terminating all watchers from cacher machinesets.machine.openshift.io W0313 10:43:13.714601 12 cacher.go:171] Terminating all watchers from cacher prometheusrules.monitoring.coreos.com W0313 10:43:13.715177 12 cacher.go:171] Terminating all watchers from cacher preprovisioningimages.metal3.io W0313 10:43:13.715992 12 cacher.go:171] Terminating all watchers from cacher machinehealthchecks.machine.openshift.io W0313 10:43:13.716476 12 cacher.go:171] Terminating all watchers from cacher csisnapshotcontrollers.operator.openshift.io W0313 10:43:13.717186 12 cacher.go:171] Terminating all watchers from cacher consolelinks.console.openshift.io W0313 10:43:13.717578 12 cacher.go:171] Terminating all watchers from cacher nodeslicepools.whereabouts.cni.cncf.io W0313 10:43:13.717840 12 cacher.go:171] Terminating all watchers from cacher operatorgroups.operators.coreos.com W0313 10:43:13.718519 12 cacher.go:171] Terminating all watchers from cacher authentications.config.openshift.io W0313 10:43:13.718985 12 cacher.go:171] Terminating all watchers from cacher servicecas.operator.openshift.io W0313 10:43:13.719310 12 cacher.go:171] Terminating all watchers from cacher consoleexternalloglinks.console.openshift.io W0313 10:43:13.720037 12 cacher.go:171] Terminating all watchers from cacher operatorconditions.operators.coreos.com W0313 10:43:13.720604 12 cacher.go:171] Terminating all watchers from cacher profiles.tuned.openshift.io W0313 10:43:13.721157 12 cacher.go:171] Terminating all watchers from cacher catalogsources.operators.coreos.com W0313 10:43:13.721756 12 cacher.go:171] Terminating all watchers from cacher egressqoses.k8s.ovn.org W0313 10:43:13.722264 12 cacher.go:171] Terminating all watchers from cacher apirequestcounts.apiserver.openshift.io W0313 10:43:13.722716 12 cacher.go:171] Terminating all watchers from cacher kubeletconfigs.machineconfiguration.openshift.io W0313 10:43:13.723211 12 cacher.go:171] Terminating all watchers from cacher controlplanemachinesets.machine.openshift.io W0313 10:43:13.723675 12 cacher.go:171] Terminating all watchers from cacher installplans.operators.coreos.com W0313 10:43:13.724089 12 cacher.go:171] Terminating all watchers from cacher alertmanagerconfigs.monitoring.coreos.com W0313 10:43:13.724520 12 cacher.go:171] Terminating all watchers from cacher alertmanagerconfigs.monitoring.coreos.com W0313 10:43:13.724956 12 cacher.go:171] Terminating all watchers from cacher clusterextensions.olm.operatorframework.io I0313 10:43:13.754562 1 main.go:235] Termination finished with exit code 0 I0313 10:43:13.754918 1 main.go:188] Deleting termination lock file "/var/log/kube-apiserver/.terminating" --- I0313 10:46:57.830033 1 main.go:213] Received signal terminated. Forwarding to sub-process "hyperkube". I0313 10:46:57.830810 12 controller.go:128] Shutting down kubernetes service endpoint reconciler I0313 10:46:57.830750 12 genericapiserver.go:554] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0313 10:46:57.831029 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ShutdownInitiated' Received signal to terminate, becoming unready, but keeping serving I0313 10:46:57.839151 12 genericapiserver.go:544] "[graceful-termination] shutdown event" name="AfterShutdownDelayDuration" I0313 10:46:57.839197 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'AfterShutdownDelayDuration' The minimal shutdown duration of 0s finished W0313 10:46:57.842762 12 lease.go:265] Resetting endpoints for master service "kubernetes" to [] I0313 10:46:57.858501 12 genericapiserver.go:721] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0313 10:46:57.858539 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TerminationPreShutdownHooksFinished' All pre-shutdown hooks have been finished I0313 10:46:57.858562 12 genericapiserver.go:633] "[graceful-termination] shutdown event" name="NotAcceptingNewRequest" I0313 10:46:57.858614 12 genericapiserver.go:679] "[graceful-termination] not going to wait for active watch request(s) to drain" I0313 10:46:57.858608 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'InFlightRequestsDrained' All non long-running request(s) in-flight have drained I0313 10:46:57.879137 12 genericapiserver.go:670] [graceful-termination] in-flight non long-running request(s) have drained I0313 10:46:57.879187 12 genericapiserver.go:711] "[graceful-termination] shutdown event" name="InFlightRequestsDrained" I0313 10:46:57.879258 12 genericapiserver.go:735] "[graceful-termination] audit backend shutdown completed" I0313 10:46:57.879257 12 controller.go:157] Shutting down quota evaluator I0313 10:46:57.879289 12 controller.go:176] quota evaluator worker shutdown I0313 10:46:57.879303 12 controller.go:176] quota evaluator worker shutdown I0313 10:46:57.879311 12 controller.go:176] quota evaluator worker shutdown I0313 10:46:57.879316 12 controller.go:176] quota evaluator worker shutdown I0313 10:46:57.879320 12 controller.go:176] quota evaluator worker shutdown I0313 10:46:57.879347 12 controller.go:84] Shutting down OpenAPI AggregationController I0313 10:46:57.879741 12 remote_available_controller.go:449] Shutting down RemoteAvailability controller I0313 10:46:57.879883 12 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt" I0313 10:46:57.879935 12 apiaccess_count_controller.go:95] Shutting down APIRequestCount controller. I0313 10:46:57.880117 12 dynamic_cafile_content.go:175] "Shutting down controller" name="request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt" I0313 10:46:57.880176 12 object_count_tracker.go:151] "StorageObjectCountTracker pruner is exiting" I0313 10:46:57.880268 12 dynamic_cafile_content.go:175] "Shutting down controller" name="request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt" I0313 10:46:57.880332 12 controller.go:86] Shutting down OpenAPI V3 AggregationController I0313 10:46:57.880349 12 dynamic_serving_content.go:149] "Shutting down controller" name="aggregator-proxy-cert::/etc/kubernetes/static-pod-certs/secrets/aggregator-client/tls.crt::/etc/kubernetes/static-pod-certs/secrets/aggregator-client/tls.key" I0313 10:46:57.880524 12 secure_serving.go:258] Stopped listening on [::]:6443 I0313 10:46:57.880550 12 genericapiserver.go:624] "[graceful-termination] shutdown event" name="HTTPServerStoppedListening" I0313 10:46:57.880567 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'HTTPServerStoppedListening' HTTP Server has stopped listening I0313 10:46:57.880696 12 cluster_authentication_trust_controller.go:466] Shutting down cluster_authentication_trust_controller controller I0313 10:46:57.880735 12 storage_flowcontrol.go:186] APF bootstrap ensurer is exiting I0313 10:46:57.880853 12 tlsconfig.go:258] "Shutting down DynamicServingCertificateController" I0313 10:46:57.881119 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.key" E0313 10:46:57.883242 12 wrap.go:53] "Timeout or abort while handling" logger="UnhandledError" method="GET" URI="/apis/user.openshift.io/v1/groups?allowWatchBookmarks=true&resourceVersion=13338&timeout=5m41s&timeoutSeconds=341&watch=true" auditID="20f3c4f7-54b4-49da-b354-5e0a09bb6fff" I0313 10:46:57.883974 12 nonstructuralschema_controller.go:207] Shutting down NonStructuralSchemaConditionController I0313 10:46:57.884295 12 autoregister_controller.go:168] Shutting down autoregister controller I0313 10:46:57.884340 12 crdregistration_controller.go:146] Shutting down crd-autoregister controller I0313 10:46:57.884392 12 establishing_controller.go:92] Shutting down EstablishingController I0313 10:46:57.884421 12 controller.go:170] Shutting down OpenAPI controller I0313 10:46:57.884534 12 dynamic_serving_content.go:149] "Shutting down controller" name="serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key" I0313 10:46:57.884560 12 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt" I0313 10:46:57.884597 12 clusterquotamapping.go:142] Shutting down ClusterQuotaMappingController controller I0313 10:46:57.884629 12 naming_controller.go:305] Shutting down NamingConditionController I0313 10:46:57.884656 12 controller.go:120] Shutting down OpenAPI V3 controller I0313 10:46:57.884680 12 customresource_discovery_controller.go:328] Shutting down DiscoveryController I0313 10:46:57.884711 12 apiservice_controller.go:134] Shutting down APIServiceRegistrationController I0313 10:46:57.884741 12 local_available_controller.go:172] Shutting down LocalAvailability controller I0313 10:46:57.884599 12 apiapproval_controller.go:201] Shutting down KubernetesAPIApprovalPolicyConformantConditionController I0313 10:46:57.884790 12 gc_controller.go:91] Shutting down apiserver lease garbage collector I0313 10:46:57.884809 12 apf_controller.go:389] Shutting down API Priority and Fairness config worker I0313 10:46:57.884656 12 crd_finalizer.go:281] Shutting down CRDFinalizer I0313 10:46:57.884852 12 controller.go:132] Ending legacy_token_tracking_controller I0313 10:46:57.884859 12 controller.go:133] Shutting down legacy_token_tracking_controller I0313 10:46:57.884877 12 system_namespaces_controller.go:76] Shutting down system namespaces controller I0313 10:46:57.885317 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.key" I0313 10:46:57.885359 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.crt::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.key" I0313 10:46:57.885382 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" I0313 10:46:57.885548 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key" I0313 10:46:59.881165 12 genericapiserver.go:742] [graceful-termination] apiserver is exiting I0313 10:46:59.881241 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TerminationGracefulTerminationFinished' All pending requests processed W0313 10:46:59.887540 12 cacher.go:171] Terminating all watchers from cacher apiservices.apiregistration.k8s.io W0313 10:46:59.888972 12 cacher.go:171] Terminating all watchers from cacher namespaces W0313 10:46:59.893412 12 cacher.go:171] Terminating all watchers from cacher pods W0313 10:46:59.898461 12 cacher.go:171] Terminating all watchers from cacher podtemplates W0313 10:46:59.899034 12 cacher.go:171] Terminating all watchers from cacher endpoints W0313 10:46:59.903781 12 cacher.go:171] Terminating all watchers from cacher serviceaccounts W0313 10:46:59.904815 12 cacher.go:171] Terminating all watchers from cacher replicationcontrollers W0313 10:46:59.905449 12 cacher.go:171] Terminating all watchers from cacher limitranges W0313 10:46:59.907568 12 cacher.go:171] Terminating all watchers from cacher secrets W0313 10:46:59.913955 12 cacher.go:171] Terminating all watchers from cacher configmaps W0313 10:46:59.925916 12 cacher.go:171] Terminating all watchers from cacher nodes W0313 10:46:59.927737 12 cacher.go:171] Terminating all watchers from cacher resourcequotas W0313 10:46:59.928434 12 cacher.go:171] Terminating all watchers from cacher services W0313 10:46:59.932694 12 cacher.go:171] Terminating all watchers from cacher persistentvolumeclaims W0313 10:46:59.933175 12 cacher.go:171] Terminating all watchers from cacher persistentvolumes W0313 10:46:59.933657 12 cacher.go:171] Terminating all watchers from cacher horizontalpodautoscalers.autoscaling W0313 10:46:59.934633 12 cacher.go:171] Terminating all watchers from cacher jobs.batch W0313 10:46:59.935777 12 cacher.go:171] Terminating all watchers from cacher cronjobs.batch W0313 10:46:59.936749 12 cacher.go:171] Terminating all watchers from cacher certificatesigningrequests.certificates.k8s.io W0313 10:46:59.937946 12 cacher.go:171] Terminating all watchers from cacher leases.coordination.k8s.io W0313 10:46:59.938688 12 cacher.go:171] Terminating all watchers from cacher endpointslices.discovery.k8s.io W0313 10:46:59.943309 12 cacher.go:171] Terminating all watchers from cacher networkpolicies.networking.k8s.io W0313 10:46:59.948653 12 cacher.go:171] Terminating all watchers from cacher ingresses.networking.k8s.io W0313 10:46:59.949630 12 cacher.go:171] Terminating all watchers from cacher ingressclasses.networking.k8s.io W0313 10:46:59.950277 12 cacher.go:171] Terminating all watchers from cacher runtimeclasses.node.k8s.io W0313 10:46:59.950858 12 cacher.go:171] Terminating all watchers from cacher poddisruptionbudgets.policy W0313 10:46:59.955048 12 cacher.go:171] Terminating all watchers from cacher roles.rbac.authorization.k8s.io W0313 10:46:59.956001 12 cacher.go:171] Terminating all watchers from cacher rolebindings.rbac.authorization.k8s.io W0313 10:46:59.958389 12 cacher.go:171] Terminating all watchers from cacher clusterroles.rbac.authorization.k8s.io W0313 10:47:00.051831 12 cacher.go:171] Terminating all watchers from cacher clusterrolebindings.rbac.authorization.k8s.io W0313 10:47:00.053470 12 cacher.go:171] Terminating all watchers from cacher volumeattachments.storage.k8s.io W0313 10:47:00.054354 12 cacher.go:171] Terminating all watchers from cacher csinodes.storage.k8s.io W0313 10:47:00.054729 12 cacher.go:171] Terminating all watchers from cacher csidrivers.storage.k8s.io W0313 10:47:00.055598 12 cacher.go:171] Terminating all watchers from cacher csistoragecapacities.storage.k8s.io W0313 10:47:00.056178 12 cacher.go:171] Terminating all watchers from cacher storageclasses.storage.k8s.io W0313 10:47:00.058217 12 cacher.go:171] Terminating all watchers from cacher deployments.apps W0313 10:47:00.062085 12 cacher.go:171] Terminating all watchers from cacher replicasets.apps W0313 10:47:00.062542 12 cacher.go:171] Terminating all watchers from cacher daemonsets.apps W0313 10:47:00.064360 12 cacher.go:171] Terminating all watchers from cacher controllerrevisions.apps W0313 10:47:00.064989 12 cacher.go:171] Terminating all watchers from cacher statefulsets.apps W0313 10:47:00.065822 12 cacher.go:171] Terminating all watchers from cacher validatingadmissionpolicybindings.admissionregistration.k8s.io W0313 10:47:00.067135 12 cacher.go:171] Terminating all watchers from cacher validatingwebhookconfigurations.admissionregistration.k8s.io W0313 10:47:00.067853 12 cacher.go:171] Terminating all watchers from cacher mutatingwebhookconfigurations.admissionregistration.k8s.io W0313 10:47:00.068397 12 cacher.go:171] Terminating all watchers from cacher validatingadmissionpolicies.admissionregistration.k8s.io W0313 10:47:00.070350 12 cacher.go:171] Terminating all watchers from cacher customresourcedefinitions.apiextensions.k8s.io W0313 10:47:00.071806 12 cacher.go:171] Terminating all watchers from cacher networks.config.openshift.io W0313 10:47:00.073224 12 cacher.go:171] Terminating all watchers from cacher clusterversions.config.openshift.io W0313 10:47:00.074808 12 cacher.go:171] Terminating all watchers from cacher ipamclaims.k8s.cni.cncf.io W0313 10:47:00.075757 12 cacher.go:171] Terminating all watchers from cacher catalogsources.operators.coreos.com W0313 10:47:00.078950 12 cacher.go:171] Terminating all watchers from cacher machineautoscalers.autoscaling.openshift.io W0313 10:47:00.079613 12 cacher.go:171] Terminating all watchers from cacher configs.samples.operator.openshift.io W0313 10:47:00.080078 12 cacher.go:171] Terminating all watchers from cacher imagepruners.imageregistry.operator.openshift.io W0313 10:47:00.080526 12 cacher.go:171] Terminating all watchers from cacher clusterserviceversions.operators.coreos.com W0313 10:47:00.081133 12 cacher.go:171] Terminating all watchers from cacher etcds.operator.openshift.io W0313 10:47:00.081467 12 cacher.go:171] Terminating all watchers from cacher machineconfigpools.machineconfiguration.openshift.io W0313 10:47:00.081949 12 cacher.go:171] Terminating all watchers from cacher storages.operator.openshift.io W0313 10:47:00.082390 12 cacher.go:171] Terminating all watchers from cacher podmonitors.monitoring.coreos.com W0313 10:47:00.083001 12 cacher.go:171] Terminating all watchers from cacher alertingrules.monitoring.openshift.io W0313 10:47:00.083897 12 cacher.go:171] Terminating all watchers from cacher authentications.operator.openshift.io W0313 10:47:00.084553 12 cacher.go:171] Terminating all watchers from cacher alertmanagerconfigs.monitoring.coreos.com W0313 10:47:00.085295 12 cacher.go:171] Terminating all watchers from cacher alertmanagerconfigs.monitoring.coreos.com W0313 10:47:00.085684 12 cacher.go:171] Terminating all watchers from cacher machineconfigs.machineconfiguration.openshift.io W0313 10:47:00.086566 12 cacher.go:171] Terminating all watchers from cacher rolebindingrestrictions.authorization.openshift.io W0313 10:47:00.087402 12 cacher.go:171] Terminating all watchers from cacher clusteruserdefinednetworks.k8s.ovn.org W0313 10:47:00.088126 12 cacher.go:171] Terminating all watchers from cacher firmwareschemas.metal3.io W0313 10:47:00.088621 12 cacher.go:171] Terminating all watchers from cacher machineconfigurations.operator.openshift.io W0313 10:47:00.090343 12 cacher.go:171] Terminating all watchers from cacher machines.machine.openshift.io W0313 10:47:00.090886 12 cacher.go:171] Terminating all watchers from cacher controlplanemachinesets.machine.openshift.io W0313 10:47:00.091316 12 cacher.go:171] Terminating all watchers from cacher machinehealthchecks.machine.openshift.io W0313 10:47:00.092096 12 cacher.go:171] Terminating all watchers from cacher metal3remediationtemplates.infrastructure.cluster.x-k8s.io W0313 10:47:00.092375 12 cacher.go:171] Terminating all watchers from cacher imagedigestmirrorsets.config.openshift.io W0313 10:47:00.092767 12 cacher.go:171] Terminating all watchers from cacher proxies.config.openshift.io W0313 10:47:00.094318 12 cacher.go:171] Terminating all watchers from cacher subscriptions.operators.coreos.com W0313 10:47:00.094882 12 cacher.go:171] Terminating all watchers from cacher overlappingrangeipreservations.whereabouts.cni.cncf.io W0313 10:47:00.095443 12 cacher.go:171] Terminating all watchers from cacher infrastructures.config.openshift.io W0313 10:47:00.096899 12 cacher.go:171] Terminating all watchers from cacher olmconfigs.operators.coreos.com W0313 10:47:00.097465 12 cacher.go:171] Terminating all watchers from cacher userdefinednetworks.k8s.ovn.org W0313 10:47:00.098327 12 cacher.go:171] Terminating all watchers from cacher servicecas.operator.openshift.io W0313 10:47:00.098834 12 cacher.go:171] Terminating all watchers from cacher containerruntimeconfigs.machineconfiguration.openshift.io W0313 10:47:00.099397 12 cacher.go:171] Terminating all watchers from cacher dnses.operator.openshift.io W0313 10:47:00.099751 12 cacher.go:171] Terminating all watchers from cacher alertrelabelconfigs.monitoring.openshift.io W0313 10:47:00.100421 12 cacher.go:171] Terminating all watchers from cacher openshiftapiservers.operator.openshift.io W0313 10:47:00.100803 12 cacher.go:171] Terminating all watchers from cacher imagecontentsourcepolicies.operator.openshift.io W0313 10:47:00.101395 12 cacher.go:171] Terminating all watchers from cacher kubeapiservers.operator.openshift.io W0313 10:47:00.101810 12 cacher.go:171] Terminating all watchers from cacher clustercatalogs.olm.operatorframework.io W0313 10:47:00.102664 12 cacher.go:171] Terminating all watchers from cacher storageversionmigrations.migration.k8s.io W0313 10:47:00.103899 12 cacher.go:171] Terminating all watchers from cacher projecthelmchartrepositories.helm.openshift.io W0313 10:47:00.104367 12 cacher.go:171] Terminating all watchers from cacher clusterextensions.olm.operatorframework.io W0313 10:47:00.105205 12 cacher.go:171] Terminating all watchers from cacher profiles.tuned.openshift.io W0313 10:47:00.105595 12 cacher.go:171] Terminating all watchers from cacher egressqoses.k8s.ovn.org W0313 10:47:00.106070 12 cacher.go:171] Terminating all watchers from cacher hardwaredata.metal3.io W0313 10:47:00.106536 12 cacher.go:171] Terminating all watchers from cacher projects.config.openshift.io W0313 10:47:00.107091 12 cacher.go:171] Terminating all watchers from cacher installplans.operators.coreos.com W0313 10:47:00.107828 12 cacher.go:171] Terminating all watchers from cacher ipaddresses.ipam.cluster.x-k8s.io W0313 10:47:00.108196 12 cacher.go:171] Terminating all watchers from cacher operatorpkis.network.operator.openshift.io W0313 10:47:00.109397 12 cacher.go:171] Terminating all watchers from cacher configs.imageregistry.operator.openshift.io W0313 10:47:00.109736 12 cacher.go:171] Terminating all watchers from cacher featuregates.config.openshift.io W0313 10:47:00.113440 12 cacher.go:171] Terminating all watchers from cacher tuneds.tuned.openshift.io W0313 10:47:00.113867 12 cacher.go:171] Terminating all watchers from cacher clusterautoscalers.autoscaling.openshift.io W0313 10:47:00.114427 12 cacher.go:171] Terminating all watchers from cacher metal3remediations.infrastructure.cluster.x-k8s.io W0313 10:47:00.114846 12 cacher.go:171] Terminating all watchers from cacher builds.config.openshift.io W0313 10:47:00.115417 12 cacher.go:171] Terminating all watchers from cacher credentialsrequests.cloudcredential.openshift.io W0313 10:47:00.115925 12 cacher.go:171] Terminating all watchers from cacher clusteroperators.config.openshift.io W0313 10:47:00.116483 12 cacher.go:171] Terminating all watchers from cacher ingresses.config.openshift.io W0313 10:47:00.116882 12 cacher.go:171] Terminating all watchers from cacher adminpolicybasedexternalroutes.k8s.ovn.org W0313 10:47:00.117476 12 cacher.go:171] Terminating all watchers from cacher egressfirewalls.k8s.ovn.org W0313 10:47:00.118764 12 cacher.go:171] Terminating all watchers from cacher securitycontextconstraints.security.openshift.io W0313 10:47:00.120124 12 cacher.go:171] Terminating all watchers from cacher operatorconditions.operators.coreos.com W0313 10:47:00.121236 12 cacher.go:171] Terminating all watchers from cacher prometheusrules.monitoring.coreos.com W0313 10:47:00.122445 12 cacher.go:171] Terminating all watchers from cacher authentications.config.openshift.io W0313 10:47:00.123201 12 cacher.go:171] Terminating all watchers from cacher provisionings.metal3.io W0313 10:47:00.123604 12 cacher.go:171] Terminating all watchers from cacher oauths.config.openshift.io W0313 10:47:00.123942 12 cacher.go:171] Terminating all watchers from cacher nodeslicepools.whereabouts.cni.cncf.io W0313 10:47:00.124279 12 cacher.go:171] Terminating all watchers from cacher network-attachment-definitions.k8s.cni.cncf.io W0313 10:47:00.124951 12 cacher.go:171] Terminating all watchers from cacher hostfirmwarecomponents.metal3.io W0313 10:47:00.125409 12 cacher.go:171] Terminating all watchers from cacher consoles.operator.openshift.io W0313 10:47:00.125803 12 cacher.go:171] Terminating all watchers from cacher configs.operator.openshift.io W0313 10:47:00.126197 12 cacher.go:171] Terminating all watchers from cacher preprovisioningimages.metal3.io W0313 10:47:00.126639 12 cacher.go:171] Terminating all watchers from cacher csisnapshotcontrollers.operator.openshift.io W0313 10:47:00.126977 12 cacher.go:171] Terminating all watchers from cacher machinesets.machine.openshift.io W0313 10:47:00.127854 12 cacher.go:171] Terminating all watchers from cacher ipaddressclaims.ipam.cluster.x-k8s.io W0313 10:47:00.128766 12 cacher.go:171] Terminating all watchers from cacher baremetalhosts.metal3.io W0313 10:47:00.128971 12 cacher.go:171] Terminating all watchers from cacher hostfirmwaresettings.metal3.io W0313 10:47:00.129457 12 cacher.go:171] Terminating all watchers from cacher dnses.config.openshift.io W0313 10:47:00.130325 12 cacher.go:171] Terminating all watchers from cacher performanceprofiles.performance.openshift.io W0313 10:47:00.130696 12 cacher.go:171] Terminating all watchers from cacher operators.operators.coreos.com W0313 10:47:00.131136 12 cacher.go:171] Terminating all watchers from cacher olms.operator.openshift.io W0313 10:47:00.131413 12 cacher.go:171] Terminating all watchers from cacher networks.operator.openshift.io W0313 10:47:00.131897 12 cacher.go:171] Terminating all watchers from cacher insightsoperators.operator.openshift.io W0313 10:47:00.133006 12 cacher.go:171] Terminating all watchers from cacher ippools.whereabouts.cni.cncf.io W0313 10:47:00.133387 12 cacher.go:171] Terminating all watchers from cacher consoleplugins.console.openshift.io W0313 10:47:00.134106 12 cacher.go:171] Terminating all watchers from cacher kubeletconfigs.machineconfiguration.openshift.io W0313 10:47:00.134521 12 cacher.go:171] Terminating all watchers from cacher baselineadminnetworkpolicies.policy.networking.k8s.io W0313 10:47:00.134878 12 cacher.go:171] Terminating all watchers from cacher images.config.openshift.io W0313 10:47:00.135537 12 cacher.go:171] Terminating all watchers from cacher operatorhubs.config.openshift.io W0313 10:47:00.135842 12 cacher.go:171] Terminating all watchers from cacher prometheuses.monitoring.coreos.com W0313 10:47:00.136236 12 cacher.go:171] Terminating all watchers from cacher consoleclidownloads.console.openshift.io W0313 10:47:00.136540 12 cacher.go:171] Terminating all watchers from cacher egressrouters.network.operator.openshift.io W0313 10:47:00.137179 12 cacher.go:171] Terminating all watchers from cacher openshiftcontrollermanagers.operator.openshift.io W0313 10:47:00.137571 12 cacher.go:171] Terminating all watchers from cacher dnsrecords.ingress.operator.openshift.io W0313 10:47:00.138559 12 cacher.go:171] Terminating all watchers from cacher hostupdatepolicies.metal3.io W0313 10:47:00.139236 12 cacher.go:171] Terminating all watchers from cacher ingresscontrollers.operator.openshift.io W0313 10:47:00.140168 12 cacher.go:171] Terminating all watchers from cacher apiservers.config.openshift.io W0313 10:47:00.140693 12 cacher.go:171] Terminating all watchers from cacher egressservices.k8s.ovn.org W0313 10:47:00.141569 12 cacher.go:171] Terminating all watchers from cacher thanosrulers.monitoring.coreos.com W0313 10:47:00.141997 12 cacher.go:171] Terminating all watchers from cacher machineosconfigs.machineconfiguration.openshift.io W0313 10:47:00.142445 12 cacher.go:171] Terminating all watchers from cacher clustercsidrivers.operator.openshift.io W0313 10:47:00.142881 12 cacher.go:171] Terminating all watchers from cacher kubestorageversionmigrators.operator.openshift.io W0313 10:47:00.143306 12 cacher.go:171] Terminating all watchers from cacher clusterresourcequotas.quota.openshift.io W0313 10:47:00.143675 12 cacher.go:171] Terminating all watchers from cacher adminnetworkpolicies.policy.networking.k8s.io W0313 10:47:00.144087 12 cacher.go:171] Terminating all watchers from cacher dataimages.metal3.io W0313 10:47:00.144427 12 cacher.go:171] Terminating all watchers from cacher operatorgroups.operators.coreos.com W0313 10:47:00.145518 12 cacher.go:171] Terminating all watchers from cacher nodes.config.openshift.io W0313 10:47:00.145874 12 cacher.go:171] Terminating all watchers from cacher volumesnapshots.snapshot.storage.k8s.io W0313 10:47:00.146513 12 cacher.go:171] Terminating all watchers from cacher imagetagmirrorsets.config.openshift.io W0313 10:47:00.146952 12 cacher.go:171] Terminating all watchers from cacher controllerconfigs.machineconfiguration.openshift.io W0313 10:47:00.147644 12 cacher.go:171] Terminating all watchers from cacher servicemonitors.monitoring.coreos.com W0313 10:47:00.148214 12 cacher.go:171] Terminating all watchers from cacher consoles.config.openshift.io W0313 10:47:00.148789 12 cacher.go:171] Terminating all watchers from cacher alertmanagers.monitoring.coreos.com W0313 10:47:00.149304 12 cacher.go:171] Terminating all watchers from cacher egressips.k8s.ovn.org W0313 10:47:00.149947 12 cacher.go:171] Terminating all watchers from cacher bmceventsubscriptions.metal3.io W0313 10:47:00.151155 12 cacher.go:171] Terminating all watchers from cacher volumesnapshotcontents.snapshot.storage.k8s.io W0313 10:47:00.151556 12 cacher.go:171] Terminating all watchers from cacher kubecontrollermanagers.operator.openshift.io W0313 10:47:00.152132 12 cacher.go:171] Terminating all watchers from cacher kubeschedulers.operator.openshift.io W0313 10:47:00.152382 12 cacher.go:171] Terminating all watchers from cacher probes.monitoring.coreos.com W0313 10:47:00.152869 12 cacher.go:171] Terminating all watchers from cacher cloudcredentials.operator.openshift.io W0313 10:47:00.153966 12 cacher.go:171] Terminating all watchers from cacher volumesnapshotclasses.snapshot.storage.k8s.io W0313 10:47:00.154553 12 cacher.go:171] Terminating all watchers from cacher schedulers.config.openshift.io I0313 10:47:00.182570 1 main.go:235] Termination finished with exit code 0 I0313 10:47:00.182676 1 main.go:188] Deleting termination lock file "/var/log/kube-apiserver/.terminating" --- I0313 10:49:41.765438 1 main.go:213] Received signal terminated. Forwarding to sub-process "hyperkube". I0313 10:49:41.767636 11 genericapiserver.go:554] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0313 10:49:41.767872 11 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ShutdownInitiated' Received signal to terminate, becoming unready, but keeping serving I0313 10:49:41.768124 11 controller.go:128] Shutting down kubernetes service endpoint reconciler I0313 10:49:41.780258 11 genericapiserver.go:544] "[graceful-termination] shutdown event" name="AfterShutdownDelayDuration" I0313 10:49:41.780297 11 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'AfterShutdownDelayDuration' The minimal shutdown duration of 0s finished W0313 10:49:41.782635 11 lease.go:265] Resetting endpoints for master service "kubernetes" to [] I0313 10:49:41.803922 11 genericapiserver.go:721] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0313 10:49:41.803958 11 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TerminationPreShutdownHooksFinished' All pre-shutdown hooks have been finished I0313 10:49:41.804188 11 genericapiserver.go:633] "[graceful-termination] shutdown event" name="NotAcceptingNewRequest" I0313 10:49:41.804210 11 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'InFlightRequestsDrained' All non long-running request(s) in-flight have drained I0313 10:49:41.804333 11 genericapiserver.go:679] "[graceful-termination] not going to wait for active watch request(s) to drain" I0313 10:49:41.838326 11 genericapiserver.go:670] [graceful-termination] in-flight non long-running request(s) have drained I0313 10:49:41.838380 11 genericapiserver.go:711] "[graceful-termination] shutdown event" name="InFlightRequestsDrained" I0313 10:49:41.838463 11 genericapiserver.go:735] "[graceful-termination] audit backend shutdown completed" I0313 10:49:41.838523 11 object_count_tracker.go:151] "StorageObjectCountTracker pruner is exiting" I0313 10:49:41.838560 11 controller.go:157] Shutting down quota evaluator I0313 10:49:41.838574 11 controller.go:176] quota evaluator worker shutdown I0313 10:49:41.839251 11 dynamic_serving_content.go:149] "Shutting down controller" name="aggregator-proxy-cert::/etc/kubernetes/static-pod-certs/secrets/aggregator-client/tls.crt::/etc/kubernetes/static-pod-certs/secrets/aggregator-client/tls.key" I0313 10:49:41.839305 11 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt" I0313 10:49:41.839310 11 dynamic_cafile_content.go:175] "Shutting down controller" name="request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt" I0313 10:49:41.839323 11 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt" I0313 10:49:41.839348 11 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.key" I0313 10:49:41.839429 11 apiaccess_count_controller.go:95] Shutting down APIRequestCount controller. I0313 10:49:41.839459 11 controller.go:86] Shutting down OpenAPI V3 AggregationController I0313 10:49:41.839498 11 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key" I0313 10:49:41.839518 11 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.key" I0313 10:49:41.839481 11 controller.go:84] Shutting down OpenAPI AggregationController I0313 10:49:41.839553 11 dynamic_serving_content.go:149] "Shutting down controller" name="serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key" I0313 10:49:41.839566 11 dynamic_cafile_content.go:175] "Shutting down controller" name="request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt" I0313 10:49:41.839490 11 controller.go:176] quota evaluator worker shutdown I0313 10:49:41.839494 11 controller.go:176] quota evaluator worker shutdown I0313 10:49:41.839498 11 controller.go:176] quota evaluator worker shutdown I0313 10:49:41.839502 11 controller.go:176] quota evaluator worker shutdown I0313 10:49:41.839529 11 clusterquotamapping.go:142] Shutting down ClusterQuotaMappingController controller I0313 10:49:41.839564 11 controller.go:120] Shutting down OpenAPI V3 controller I0313 10:49:41.839591 11 customresource_discovery_controller.go:328] Shutting down DiscoveryController I0313 10:49:41.839600 11 system_namespaces_controller.go:76] Shutting down system namespaces controller I0313 10:49:41.839620 11 gc_controller.go:91] Shutting down apiserver lease garbage collector I0313 10:49:41.839625 11 autoregister_controller.go:168] Shutting down autoregister controller I0313 10:49:41.839628 11 remote_available_controller.go:449] Shutting down RemoteAvailability controller I0313 10:49:41.839632 11 secure_serving.go:258] Stopped listening on [::]:6443 I0313 10:49:41.840319 11 cluster_authentication_trust_controller.go:466] Shutting down cluster_authentication_trust_controller controller I0313 10:49:41.840400 11 storage_flowcontrol.go:186] APF bootstrap ensurer is exiting I0313 10:49:41.840031 11 genericapiserver.go:624] "[graceful-termination] shutdown event" name="HTTPServerStoppedListening" I0313 10:49:41.842281 11 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'HTTPServerStoppedListening' HTTP Server has stopped listening I0313 10:49:41.839643 11 apf_controller.go:389] Shutting down API Priority and Fairness config worker I0313 10:49:41.839655 11 crdregistration_controller.go:146] Shutting down crd-autoregister controller I0313 10:49:41.839675 11 apiapproval_controller.go:201] Shutting down KubernetesAPIApprovalPolicyConformantConditionController I0313 10:49:41.839690 11 nonstructuralschema_controller.go:207] Shutting down NonStructuralSchemaConditionController I0313 10:49:41.839699 11 naming_controller.go:305] Shutting down NamingConditionController I0313 10:49:41.839705 11 establishing_controller.go:92] Shutting down EstablishingController I0313 10:49:41.839717 11 crd_finalizer.go:281] Shutting down CRDFinalizer I0313 10:49:41.839725 11 controller.go:170] Shutting down OpenAPI controller I0313 10:49:41.839744 11 apiservice_controller.go:134] Shutting down APIServiceRegistrationController I0313 10:49:41.839759 11 local_available_controller.go:172] Shutting down LocalAvailability controller I0313 10:49:41.839783 11 controller.go:132] Ending legacy_token_tracking_controller I0313 10:49:41.843127 11 controller.go:133] Shutting down legacy_token_tracking_controller I0313 10:49:41.844292 11 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.crt::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.key" I0313 10:49:41.844325 11 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" I0313 10:49:41.844678 11 tlsconfig.go:258] "Shutting down DynamicServingCertificateController" E0313 10:49:41.904800 11 wrap.go:53] "Timeout or abort while handling" logger="UnhandledError" method="GET" URI="/apis/user.openshift.io/v1/groups?allowWatchBookmarks=true&resourceVersion=14613&timeout=7m50s&timeoutSeconds=470&watch=true" auditID="4fbfd8c0-37c0-4d98-b542-9e6ac7bb111d" I0313 10:49:43.840046 11 genericapiserver.go:742] [graceful-termination] apiserver is exiting I0313 10:49:43.840145 11 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TerminationGracefulTerminationFinished' All pending requests processed W0313 10:49:43.845189 11 cacher.go:171] Terminating all watchers from cacher apiservices.apiregistration.k8s.io W0313 10:49:43.847310 11 cacher.go:171] Terminating all watchers from cacher serviceaccounts W0313 10:49:43.849981 11 cacher.go:171] Terminating all watchers from cacher endpoints W0313 10:49:43.853151 11 cacher.go:171] Terminating all watchers from cacher persistentvolumeclaims W0313 10:49:43.853877 11 cacher.go:171] Terminating all watchers from cacher podtemplates W0313 10:49:43.854480 11 cacher.go:171] Terminating all watchers from cacher replicationcontrollers W0313 10:49:43.855230 11 cacher.go:171] Terminating all watchers from cacher persistentvolumes W0313 10:49:43.856280 11 cacher.go:171] Terminating all watchers from cacher configmaps W0313 10:49:43.872585 11 cacher.go:171] Terminating all watchers from cacher pods W0313 10:49:43.875463 11 cacher.go:171] Terminating all watchers from cacher resourcequotas W0313 10:49:43.876338 11 cacher.go:171] Terminating all watchers from cacher namespaces W0313 10:49:43.876946 11 cacher.go:171] Terminating all watchers from cacher nodes W0313 10:49:43.877548 11 cacher.go:171] Terminating all watchers from cacher secrets W0313 10:49:43.884985 11 cacher.go:171] Terminating all watchers from cacher services W0313 10:49:43.909544 11 cacher.go:171] Terminating all watchers from cacher limitranges W0313 10:49:43.910914 11 cacher.go:171] Terminating all watchers from cacher horizontalpodautoscalers.autoscaling W0313 10:49:43.911632 11 cacher.go:171] Terminating all watchers from cacher cronjobs.batch W0313 10:49:43.912674 11 cacher.go:171] Terminating all watchers from cacher jobs.batch W0313 10:49:43.913952 11 cacher.go:171] Terminating all watchers from cacher certificatesigningrequests.certificates.k8s.io W0313 10:49:43.915645 11 cacher.go:171] Terminating all watchers from cacher leases.coordination.k8s.io W0313 10:49:43.916578 11 cacher.go:171] Terminating all watchers from cacher endpointslices.discovery.k8s.io W0313 10:49:43.917350 11 cacher.go:171] Terminating all watchers from cacher ingresses.networking.k8s.io W0313 10:49:43.918412 11 cacher.go:171] Terminating all watchers from cacher ingressclasses.networking.k8s.io W0313 10:49:43.919555 11 cacher.go:171] Terminating all watchers from cacher networkpolicies.networking.k8s.io W0313 10:49:43.921817 11 cacher.go:171] Terminating all watchers from cacher runtimeclasses.node.k8s.io W0313 10:49:43.925496 11 cacher.go:171] Terminating all watchers from cacher poddisruptionbudgets.policy W0313 10:49:43.926740 11 cacher.go:171] Terminating all watchers from cacher roles.rbac.authorization.k8s.io W0313 10:49:43.928854 11 cacher.go:171] Terminating all watchers from cacher rolebindings.rbac.authorization.k8s.io W0313 10:49:43.929985 11 cacher.go:171] Terminating all watchers from cacher clusterroles.rbac.authorization.k8s.io W0313 10:49:43.930987 11 cacher.go:171] Terminating all watchers from cacher clusterrolebindings.rbac.authorization.k8s.io W0313 10:49:43.931780 11 cacher.go:171] Terminating all watchers from cacher priorityclasses.scheduling.k8s.io W0313 10:49:43.932201 11 cacher.go:171] Terminating all watchers from cacher storageclasses.storage.k8s.io W0313 10:49:43.932592 11 cacher.go:171] Terminating all watchers from cacher volumeattachments.storage.k8s.io W0313 10:49:43.933019 11 cacher.go:171] Terminating all watchers from cacher csinodes.storage.k8s.io W0313 10:49:43.933682 11 cacher.go:171] Terminating all watchers from cacher csidrivers.storage.k8s.io W0313 10:49:43.935084 11 cacher.go:171] Terminating all watchers from cacher csistoragecapacities.storage.k8s.io W0313 10:49:43.942413 11 cacher.go:171] Terminating all watchers from cacher flowschemas.flowcontrol.apiserver.k8s.io W0313 10:49:43.942998 11 cacher.go:171] Terminating all watchers from cacher prioritylevelconfigurations.flowcontrol.apiserver.k8s.io W0313 10:49:43.943462 11 cacher.go:171] Terminating all watchers from cacher deployments.apps W0313 10:49:43.945524 11 cacher.go:171] Terminating all watchers from cacher daemonsets.apps W0313 10:49:43.948025 11 cacher.go:171] Terminating all watchers from cacher statefulsets.apps W0313 10:49:43.948965 11 cacher.go:171] Terminating all watchers from cacher replicasets.apps W0313 10:49:43.949325 11 cacher.go:171] Terminating all watchers from cacher controllerrevisions.apps W0313 10:49:43.949906 11 cacher.go:171] Terminating all watchers from cacher validatingwebhookconfigurations.admissionregistration.k8s.io W0313 10:49:43.952362 11 cacher.go:171] Terminating all watchers from cacher mutatingwebhookconfigurations.admissionregistration.k8s.io W0313 10:49:43.954299 11 cacher.go:171] Terminating all watchers from cacher validatingadmissionpolicies.admissionregistration.k8s.io W0313 10:49:43.956074 11 cacher.go:171] Terminating all watchers from cacher validatingadmissionpolicybindings.admissionregistration.k8s.io W0313 10:49:43.957843 11 cacher.go:171] Terminating all watchers from cacher customresourcedefinitions.apiextensions.k8s.io W0313 10:49:43.959136 11 cacher.go:171] Terminating all watchers from cacher egressqoses.k8s.ovn.org W0313 10:49:43.959559 11 cacher.go:171] Terminating all watchers from cacher egressfirewalls.k8s.ovn.org W0313 10:49:43.959978 11 cacher.go:171] Terminating all watchers from cacher ingresscontrollers.operator.openshift.io W0313 10:49:43.960603 11 cacher.go:171] Terminating all watchers from cacher securitycontextconstraints.security.openshift.io W0313 10:49:43.960843 11 cacher.go:171] Terminating all watchers from cacher projects.config.openshift.io W0313 10:49:43.961305 11 cacher.go:171] Terminating all watchers from cacher controlplanemachinesets.machine.openshift.io W0313 10:49:43.961687 11 cacher.go:171] Terminating all watchers from cacher machineconfigs.machineconfiguration.openshift.io W0313 10:49:43.962587 11 cacher.go:171] Terminating all watchers from cacher consoleexternalloglinks.console.openshift.io W0313 10:49:43.962975 11 cacher.go:171] Terminating all watchers from cacher consoles.config.openshift.io W0313 10:49:43.963560 11 cacher.go:171] Terminating all watchers from cacher machinesets.machine.openshift.io W0313 10:49:43.964009 11 cacher.go:171] Terminating all watchers from cacher dnsrecords.ingress.operator.openshift.io W0313 10:49:43.964631 11 cacher.go:171] Terminating all watchers from cacher apiservers.config.openshift.io W0313 10:49:43.966254 11 cacher.go:171] Terminating all watchers from cacher alertmanagerconfigs.monitoring.coreos.com W0313 10:49:43.966641 11 cacher.go:171] Terminating all watchers from cacher alertmanagerconfigs.monitoring.coreos.com W0313 10:49:43.967419 11 cacher.go:171] Terminating all watchers from cacher storageversionmigrations.migration.k8s.io W0313 10:49:43.968459 11 cacher.go:171] Terminating all watchers from cacher rangeallocations.security.internal.openshift.io W0313 10:49:43.969155 11 cacher.go:171] Terminating all watchers from cacher alertmanagers.monitoring.coreos.com W0313 10:49:43.969723 11 cacher.go:171] Terminating all watchers from cacher openshiftcontrollermanagers.operator.openshift.io W0313 10:49:43.970215 11 cacher.go:171] Terminating all watchers from cacher consolelinks.console.openshift.io W0313 10:49:43.970781 11 cacher.go:171] Terminating all watchers from cacher consoleyamlsamples.console.openshift.io W0313 10:49:43.971234 11 cacher.go:171] Terminating all watchers from cacher bmceventsubscriptions.metal3.io W0313 10:49:43.971826 11 cacher.go:171] Terminating all watchers from cacher clusterextensions.olm.operatorframework.io W0313 10:49:43.972698 11 cacher.go:171] Terminating all watchers from cacher clusteroperators.config.openshift.io W0313 10:49:43.974339 11 cacher.go:171] Terminating all watchers from cacher storages.operator.openshift.io W0313 10:49:43.974931 11 cacher.go:171] Terminating all watchers from cacher schedulers.config.openshift.io W0313 10:49:43.975499 11 cacher.go:171] Terminating all watchers from cacher imagecontentsourcepolicies.operator.openshift.io W0313 10:49:43.976082 11 cacher.go:171] Terminating all watchers from cacher configs.operator.openshift.io W0313 10:49:43.976569 11 cacher.go:171] Terminating all watchers from cacher rolebindingrestrictions.authorization.openshift.io W0313 10:49:43.977256 11 cacher.go:171] Terminating all watchers from cacher egressips.k8s.ovn.org W0313 10:49:43.977853 11 cacher.go:171] Terminating all watchers from cacher kubestorageversionmigrators.operator.openshift.io W0313 10:49:43.978357 11 cacher.go:171] Terminating all watchers from cacher etcds.operator.openshift.io W0313 10:49:43.978899 11 cacher.go:171] Terminating all watchers from cacher kubeapiservers.operator.openshift.io W0313 10:49:43.979227 11 cacher.go:171] Terminating all watchers from cacher machineosbuilds.machineconfiguration.openshift.io W0313 10:49:43.979623 11 cacher.go:171] Terminating all watchers from cacher thanosrulers.monitoring.coreos.com W0313 10:49:43.980218 11 cacher.go:171] Terminating all watchers from cacher machinehealthchecks.machine.openshift.io W0313 10:49:43.980787 11 cacher.go:171] Terminating all watchers from cacher clustercatalogs.olm.operatorframework.io W0313 10:49:43.981284 11 cacher.go:171] Terminating all watchers from cacher ingresses.config.openshift.io W0313 10:49:43.981560 11 cacher.go:171] Terminating all watchers from cacher tuneds.tuned.openshift.io W0313 10:49:43.981971 11 cacher.go:171] Terminating all watchers from cacher projecthelmchartrepositories.helm.openshift.io W0313 10:49:43.982519 11 cacher.go:171] Terminating all watchers from cacher servicecas.operator.openshift.io W0313 10:49:43.982952 11 cacher.go:171] Terminating all watchers from cacher kubecontrollermanagers.operator.openshift.io W0313 10:49:43.983586 11 cacher.go:171] Terminating all watchers from cacher probes.monitoring.coreos.com W0313 10:49:43.985292 11 cacher.go:171] Terminating all watchers from cacher adminpolicybasedexternalroutes.k8s.ovn.org W0313 10:49:43.985709 11 cacher.go:171] Terminating all watchers from cacher machineautoscalers.autoscaling.openshift.io W0313 10:49:43.985969 11 cacher.go:171] Terminating all watchers from cacher machineconfigurations.operator.openshift.io W0313 10:49:43.986250 11 cacher.go:171] Terminating all watchers from cacher overlappingrangeipreservations.whereabouts.cni.cncf.io W0313 10:49:43.986652 11 cacher.go:171] Terminating all watchers from cacher installplans.operators.coreos.com W0313 10:49:43.987048 11 cacher.go:171] Terminating all watchers from cacher proxies.config.openshift.io W0313 10:49:43.987494 11 cacher.go:171] Terminating all watchers from cacher baremetalhosts.metal3.io W0313 10:49:43.987876 11 cacher.go:171] Terminating all watchers from cacher operatorgroups.operators.coreos.com W0313 10:49:43.988879 11 cacher.go:171] Terminating all watchers from cacher egressrouters.network.operator.openshift.io W0313 10:49:43.989508 11 cacher.go:171] Terminating all watchers from cacher kubeschedulers.operator.openshift.io W0313 10:49:43.990056 11 cacher.go:171] Terminating all watchers from cacher imagecontentpolicies.config.openshift.io W0313 10:49:43.990593 11 cacher.go:171] Terminating all watchers from cacher volumesnapshotcontents.snapshot.storage.k8s.io W0313 10:49:43.991354 11 cacher.go:171] Terminating all watchers from cacher consolenotifications.console.openshift.io W0313 10:49:43.991732 11 cacher.go:171] Terminating all watchers from cacher kubeletconfigs.machineconfiguration.openshift.io W0313 10:49:43.992062 11 cacher.go:171] Terminating all watchers from cacher firmwareschemas.metal3.io W0313 10:49:43.992371 11 cacher.go:171] Terminating all watchers from cacher hostupdatepolicies.metal3.io W0313 10:49:43.992624 11 cacher.go:171] Terminating all watchers from cacher machineconfigpools.machineconfiguration.openshift.io W0313 10:49:43.993182 11 cacher.go:171] Terminating all watchers from cacher olmconfigs.operators.coreos.com W0313 10:49:43.993486 11 cacher.go:171] Terminating all watchers from cacher alertrelabelconfigs.monitoring.openshift.io W0313 10:49:43.993807 11 cacher.go:171] Terminating all watchers from cacher operators.operators.coreos.com W0313 10:49:43.994190 11 cacher.go:171] Terminating all watchers from cacher builds.config.openshift.io W0313 10:49:43.994603 11 cacher.go:171] Terminating all watchers from cacher preprovisioningimages.metal3.io W0313 10:49:43.995117 11 cacher.go:171] Terminating all watchers from cacher imagetagmirrorsets.config.openshift.io W0313 10:49:43.995828 11 cacher.go:171] Terminating all watchers from cacher volumesnapshots.snapshot.storage.k8s.io W0313 10:49:43.996768 11 cacher.go:171] Terminating all watchers from cacher subscriptions.operators.coreos.com W0313 10:49:43.997362 11 cacher.go:171] Terminating all watchers from cacher dnses.config.openshift.io W0313 10:49:43.998289 11 cacher.go:171] Terminating all watchers from cacher operatorhubs.config.openshift.io W0313 10:49:43.998675 11 cacher.go:171] Terminating all watchers from cacher nodeslicepools.whereabouts.cni.cncf.io W0313 10:49:43.998979 11 cacher.go:171] Terminating all watchers from cacher storagestates.migration.k8s.io W0313 10:49:43.999276 11 cacher.go:171] Terminating all watchers from cacher consolequickstarts.console.openshift.io W0313 10:49:43.999729 11 cacher.go:171] Terminating all watchers from cacher machines.machine.openshift.io W0313 10:49:44.000572 11 cacher.go:171] Terminating all watchers from cacher operatorconditions.operators.coreos.com W0313 10:49:44.001184 11 cacher.go:171] Terminating all watchers from cacher configs.imageregistry.operator.openshift.io W0313 10:49:44.002163 11 cacher.go:171] Terminating all watchers from cacher podmonitors.monitoring.coreos.com W0313 10:49:44.003560 11 cacher.go:171] Terminating all watchers from cacher performanceprofiles.performance.openshift.io W0313 10:49:44.004502 11 cacher.go:171] Terminating all watchers from cacher hardwaredata.metal3.io W0313 10:49:44.005522 11 cacher.go:171] Terminating all watchers from cacher ipaddressclaims.ipam.cluster.x-k8s.io W0313 10:49:44.005949 11 cacher.go:171] Terminating all watchers from cacher prometheuses.monitoring.coreos.com W0313 10:49:44.006427 11 cacher.go:171] Terminating all watchers from cacher apirequestcounts.apiserver.openshift.io W0313 10:49:44.006842 11 cacher.go:171] Terminating all watchers from cacher network-attachment-definitions.k8s.cni.cncf.io W0313 10:49:44.007323 11 cacher.go:171] Terminating all watchers from cacher ippools.whereabouts.cni.cncf.io W0313 10:49:44.007735 11 cacher.go:171] Terminating all watchers from cacher networks.operator.openshift.io W0313 10:49:44.008336 11 cacher.go:171] Terminating all watchers from cacher operatorpkis.network.operator.openshift.io W0313 10:49:44.008610 11 cacher.go:171] Terminating all watchers from cacher volumesnapshotclasses.snapshot.storage.k8s.io W0313 10:49:44.009475 11 cacher.go:171] Terminating all watchers from cacher catalogsources.operators.coreos.com W0313 10:49:44.010006 11 cacher.go:171] Terminating all watchers from cacher clustercsidrivers.operator.openshift.io W0313 10:49:44.010648 11 cacher.go:171] Terminating all watchers from cacher metal3remediations.infrastructure.cluster.x-k8s.io W0313 10:49:44.011083 11 cacher.go:171] Terminating all watchers from cacher imagepruners.imageregistry.operator.openshift.io W0313 10:49:44.011812 11 cacher.go:171] Terminating all watchers from cacher featuregates.config.openshift.io W0313 10:49:44.012878 11 cacher.go:171] Terminating all watchers from cacher authentications.config.openshift.io W0313 10:49:44.013582 11 cacher.go:171] Terminating all watchers from cacher credentialsrequests.cloudcredential.openshift.io W0313 10:49:44.014040 11 cacher.go:171] Terminating all watchers from cacher clusterautoscalers.autoscaling.openshift.io W0313 10:49:44.014668 11 cacher.go:171] Terminating all watchers from cacher clusterversions.config.openshift.io W0313 10:49:44.016545 11 cacher.go:171] Terminating all watchers from cacher consoles.operator.openshift.io W0313 10:49:44.017378 11 cacher.go:171] Terminating all watchers from cacher nodes.config.openshift.io W0313 10:49:44.017867 11 cacher.go:171] Terminating all watchers from cacher dnses.operator.openshift.io W0313 10:49:44.019027 11 cacher.go:171] Terminating all watchers from cacher adminnetworkpolicies.policy.networking.k8s.io W0313 10:49:44.019361 11 cacher.go:171] Terminating all watchers from cacher cloudcredentials.operator.openshift.io W0313 10:49:44.020264 11 cacher.go:171] Terminating all watchers from cacher ipaddresses.ipam.cluster.x-k8s.io W0313 10:49:44.020767 11 cacher.go:171] Terminating all watchers from cacher prometheusrules.monitoring.coreos.com W0313 10:49:44.021642 11 cacher.go:171] Terminating all watchers from cacher helmchartrepositories.helm.openshift.io W0313 10:49:44.022034 11 cacher.go:171] Terminating all watchers from cacher containerruntimeconfigs.machineconfiguration.openshift.io W0313 10:49:44.022528 11 cacher.go:171] Terminating all watchers from cacher consolesamples.console.openshift.io W0313 10:49:44.023023 11 cacher.go:171] Terminating all watchers from cacher servicemonitors.monitoring.coreos.com W0313 10:49:44.023716 11 cacher.go:171] Terminating all watchers from cacher networks.config.openshift.io W0313 10:49:44.024210 11 cacher.go:171] Terminating all watchers from cacher hostfirmwarecomponents.metal3.io W0313 10:49:44.024770 11 cacher.go:171] Terminating all watchers from cacher userdefinednetworks.k8s.ovn.org W0313 10:49:44.025520 11 cacher.go:171] Terminating all watchers from cacher consoleplugins.console.openshift.io W0313 10:49:44.026081 11 cacher.go:171] Terminating all watchers from cacher profiles.tuned.openshift.io W0313 10:49:44.027138 11 cacher.go:171] Terminating all watchers from cacher insightsoperators.operator.openshift.io W0313 10:49:44.027561 11 cacher.go:171] Terminating all watchers from cacher egressservices.k8s.ovn.org W0313 10:49:44.027819 11 cacher.go:171] Terminating all watchers from cacher provisionings.metal3.io W0313 10:49:44.028265 11 cacher.go:171] Terminating all watchers from cacher hostfirmwaresettings.metal3.io W0313 10:49:44.028573 11 cacher.go:171] Terminating all watchers from cacher clusteruserdefinednetworks.k8s.ovn.org W0313 10:49:44.029081 11 cacher.go:171] Terminating all watchers from cacher consoleclidownloads.console.openshift.io W0313 10:49:44.029679 11 cacher.go:171] Terminating all watchers from cacher infrastructures.config.openshift.io W0313 10:49:44.031578 11 cacher.go:171] Terminating all watchers from cacher clusterserviceversions.operators.coreos.com W0313 10:49:44.032132 11 cacher.go:171] Terminating all watchers from cacher configs.samples.operator.openshift.io W0313 10:49:44.032727 11 cacher.go:171] Terminating all watchers from cacher imagedigestmirrorsets.config.openshift.io W0313 10:49:44.033263 11 cacher.go:171] Terminating all watchers from cacher baselineadminnetworkpolicies.policy.networking.k8s.io W0313 10:49:44.033727 11 cacher.go:171] Terminating all watchers from cacher alertingrules.monitoring.openshift.io W0313 10:49:44.034860 11 cacher.go:171] Terminating all watchers from cacher olms.operator.openshift.io W0313 10:49:44.035349 11 cacher.go:171] Terminating all watchers from cacher images.config.openshift.io W0313 10:49:44.036191 11 cacher.go:171] Terminating all watchers from cacher openshiftapiservers.operator.openshift.io W0313 10:49:44.036822 11 cacher.go:171] Terminating all watchers from cacher dataimages.metal3.io W0313 10:49:44.037976 11 cacher.go:171] Terminating all watchers from cacher csisnapshotcontrollers.operator.openshift.io W0313 10:49:44.038565 11 cacher.go:171] Terminating all watchers from cacher machineosconfigs.machineconfiguration.openshift.io W0313 10:49:44.038847 11 cacher.go:171] Terminating all watchers from cacher clusterresourcequotas.quota.openshift.io W0313 10:49:44.039373 11 cacher.go:171] Terminating all watchers from cacher ipamclaims.k8s.cni.cncf.io W0313 10:49:44.039939 11 cacher.go:171] Terminating all watchers from cacher podnetworkconnectivitychecks.controlplane.operator.openshift.io W0313 10:49:44.040429 11 cacher.go:171] Terminating all watchers from cacher oauths.config.openshift.io W0313 10:49:44.040868 11 cacher.go:171] Terminating all watchers from cacher authentications.operator.openshift.io W0313 10:49:44.041261 11 cacher.go:171] Terminating all watchers from cacher controllerconfigs.machineconfiguration.openshift.io W0313 10:49:44.042254 11 cacher.go:171] Terminating all watchers from cacher metal3remediationtemplates.infrastructure.cluster.x-k8s.io I0313 10:49:44.076526 1 main.go:235] Termination finished with exit code 0 I0313 10:49:44.076618 1 main.go:188] Deleting termination lock file "/var/log/kube-apiserver/.terminating" ---