=ERROR REPORT==== 11-Oct-2025::10:53:45.477566 === inet_config: syntax error in /etc/rabbitmq/erl_inetrc =ERROR REPORT==== 11-Oct-2025::10:53:45.477566 === inet_config: syntax error in /etc/rabbitmq/erl_inetrc 2025-10-11 10:53:46.700997+00:00 [warning] <0.157.0> TLS distribution: unexpected message: {'EXIT',<0.156.0>,shutdown} 2025-10-11 10:53:46.700997+00:00 [warning] <0.157.0> 2025-10-11 10:53:47.875288+00:00 [info] <0.250.0> Feature flags: list of feature flags found: 2025-10-11 10:53:47.875360+00:00 [info] <0.250.0> Feature flags: [ ] implicit_default_bindings 2025-10-11 10:53:47.875378+00:00 [info] <0.250.0> Feature flags: [ ] maintenance_mode_status 2025-10-11 10:53:47.875392+00:00 [info] <0.250.0> Feature flags: [ ] quorum_queue 2025-10-11 10:53:47.875460+00:00 [info] <0.250.0> Feature flags: [ ] stream_queue 2025-10-11 10:53:47.875482+00:00 [info] <0.250.0> Feature flags: [ ] user_limits 2025-10-11 10:53:47.875502+00:00 [info] <0.250.0> Feature flags: [ ] virtual_host_metadata 2025-10-11 10:53:47.875554+00:00 [info] <0.250.0> Feature flags: feature flag states written to disk: yes 2025-10-11 10:53:48.216381+00:00 [notice] <0.44.0> Application syslog exited with reason: stopped 2025-10-11 10:53:48.216543+00:00 [notice] <0.250.0> Logging: switching to configured handler(s); following messages may not be visible in this log output 2025-10-11 10:53:48.237864+00:00 [notice] <0.250.0> Logging: configured log handlers are now ACTIVE 2025-10-11 10:53:48.491548+00:00 [info] <0.250.0> ra: starting system quorum_queues 2025-10-11 10:53:48.491626+00:00 [info] <0.250.0> starting Ra system: quorum_queues in directory: /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cell1-server-1.rabbitmq-cell1-nodes.openstack/quorum/rabbit@rabbitmq-cell1-server-1.rabbitmq-cell1-nodes.openstack 2025-10-11 10:53:48.546295+00:00 [info] <0.286.0> ra system 'quorum_queues' running pre init for 0 registered servers 2025-10-11 10:53:48.557668+00:00 [info] <0.287.0> ra: meta data store initialised for system quorum_queues. 0 record(s) recovered 2025-10-11 10:53:48.572089+00:00 [notice] <0.292.0> WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables 2025-10-11 10:53:48.582917+00:00 [info] <0.250.0> ra: starting system coordination 2025-10-11 10:53:48.582954+00:00 [info] <0.250.0> starting Ra system: coordination in directory: /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cell1-server-1.rabbitmq-cell1-nodes.openstack/coordination/rabbit@rabbitmq-cell1-server-1.rabbitmq-cell1-nodes.openstack 2025-10-11 10:53:48.591261+00:00 [info] <0.299.0> ra system 'coordination' running pre init for 0 registered servers 2025-10-11 10:53:48.591848+00:00 [info] <0.300.0> ra: meta data store initialised for system coordination. 0 record(s) recovered 2025-10-11 10:53:48.591973+00:00 [notice] <0.305.0> WAL: ra_coordination_log_wal init, open tbls: ra_coordination_log_open_mem_tables, closed tbls: ra_coordination_log_closed_mem_tables 2025-10-11 10:53:48.597427+00:00 [info] <0.250.0>  2025-10-11 10:53:48.597427+00:00 [info] <0.250.0> Starting RabbitMQ 3.9.21 on Erlang 24.3.4.2 [jit] 2025-10-11 10:53:48.597427+00:00 [info] <0.250.0> Copyright (c) 2007-2022 VMware, Inc. or its affiliates. 2025-10-11 10:53:48.597427+00:00 [info] <0.250.0> Licensed under the MPL 2.0. Website: https://rabbitmq.com ## ## RabbitMQ 3.9.21 ## ## ########## Copyright (c) 2007-2022 VMware, Inc. or its affiliates. ###### ## ########## Licensed under the MPL 2.0. Website: https://rabbitmq.com Erlang: 24.3.4.2 [jit] TLS Library: OpenSSL - OpenSSL 3.5.1 1 Jul 2025 Doc guides: https://rabbitmq.com/documentation.html Support: https://rabbitmq.com/contact.html Tutorials: https://rabbitmq.com/getstarted.html Monitoring: https://rabbitmq.com/monitoring.html Logs: /var/lib/rabbitmq/rabbitmq_upgrade.log Config file(s): /etc/rabbitmq/advanced.config /etc/rabbitmq/conf.d/10-operatorDefaults.conf /etc/rabbitmq/conf.d/11-default_user.conf /etc/rabbitmq/conf.d/90-userDefinedConfiguration.conf Starting broker...2025-10-11 10:53:48.598901+00:00 [info] <0.250.0>  2025-10-11 10:53:48.598901+00:00 [info] <0.250.0> node : rabbit@rabbitmq-cell1-server-1.rabbitmq-cell1-nodes.openstack 2025-10-11 10:53:48.598901+00:00 [info] <0.250.0> home dir : /var/lib/rabbitmq 2025-10-11 10:53:48.598901+00:00 [info] <0.250.0> config file(s) : /etc/rabbitmq/advanced.config 2025-10-11 10:53:48.598901+00:00 [info] <0.250.0> : /etc/rabbitmq/conf.d/10-operatorDefaults.conf 2025-10-11 10:53:48.598901+00:00 [info] <0.250.0> : /etc/rabbitmq/conf.d/11-default_user.conf 2025-10-11 10:53:48.598901+00:00 [info] <0.250.0> : /etc/rabbitmq/conf.d/90-userDefinedConfiguration.conf 2025-10-11 10:53:48.598901+00:00 [info] <0.250.0> cookie hash : 4ervU2rn7if45GXtXZuXkA== 2025-10-11 10:53:48.598901+00:00 [info] <0.250.0> log(s) : /var/lib/rabbitmq/rabbitmq_upgrade.log 2025-10-11 10:53:48.598901+00:00 [info] <0.250.0> :  2025-10-11 10:53:48.598901+00:00 [info] <0.250.0> database dir : /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cell1-server-1.rabbitmq-cell1-nodes.openstack 2025-10-11 10:53:48.854452+00:00 [info] <0.250.0> Feature flags: list of feature flags found: 2025-10-11 10:53:48.854519+00:00 [info] <0.250.0> Feature flags: [ ] drop_unroutable_metric 2025-10-11 10:53:48.854544+00:00 [info] <0.250.0> Feature flags: [ ] empty_basic_get_metric 2025-10-11 10:53:48.854582+00:00 [info] <0.250.0> Feature flags: [ ] implicit_default_bindings 2025-10-11 10:53:48.854597+00:00 [info] <0.250.0> Feature flags: [ ] maintenance_mode_status 2025-10-11 10:53:48.854610+00:00 [info] <0.250.0> Feature flags: [ ] quorum_queue 2025-10-11 10:53:48.854627+00:00 [info] <0.250.0> Feature flags: [ ] stream_queue 2025-10-11 10:53:48.854698+00:00 [info] <0.250.0> Feature flags: [ ] user_limits 2025-10-11 10:53:48.854729+00:00 [info] <0.250.0> Feature flags: [ ] virtual_host_metadata 2025-10-11 10:53:48.854747+00:00 [info] <0.250.0> Feature flags: feature flag states written to disk: yes 2025-10-11 10:53:51.884758+00:00 [info] <0.250.0> Running boot step pre_boot defined by app rabbit 2025-10-11 10:53:51.884873+00:00 [info] <0.250.0> Running boot step rabbit_global_counters defined by app rabbit 2025-10-11 10:53:51.885204+00:00 [info] <0.250.0> Running boot step rabbit_osiris_metrics defined by app rabbit 2025-10-11 10:53:51.885324+00:00 [info] <0.250.0> Running boot step rabbit_core_metrics defined by app rabbit 2025-10-11 10:53:51.886579+00:00 [info] <0.250.0> Running boot step rabbit_alarm defined by app rabbit 2025-10-11 10:53:51.898327+00:00 [info] <0.339.0> Memory high watermark set to 655 MiB (687194767 bytes) of 1638 MiB (1717986919 bytes) total 2025-10-11 10:53:51.916605+00:00 [info] <0.341.0> Enabling free disk space monitoring 2025-10-11 10:53:51.916810+00:00 [info] <0.341.0> Disk free limit set to 2000MB 2025-10-11 10:53:51.929601+00:00 [info] <0.250.0> Running boot step code_server_cache defined by app rabbit 2025-10-11 10:53:51.929819+00:00 [info] <0.250.0> Running boot step file_handle_cache defined by app rabbit 2025-10-11 10:53:51.930159+00:00 [info] <0.346.0> Limiting to approx 1048479 file handles (943629 sockets) 2025-10-11 10:53:51.930287+00:00 [info] <0.347.0> FHC read buffering: OFF 2025-10-11 10:53:51.930349+00:00 [info] <0.347.0> FHC write buffering: ON 2025-10-11 10:53:51.930940+00:00 [info] <0.250.0> Running boot step worker_pool defined by app rabbit 2025-10-11 10:53:51.931016+00:00 [info] <0.307.0> Will use 16 processes for default worker pool 2025-10-11 10:53:51.931063+00:00 [info] <0.307.0> Starting worker pool 'worker_pool' with 16 processes in it 2025-10-11 10:53:51.931979+00:00 [info] <0.250.0> Running boot step database defined by app rabbit 2025-10-11 10:53:51.932277+00:00 [info] <0.250.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cell1-server-1.rabbitmq-cell1-nodes.openstack is empty. Assuming we need to join an existing cluster or initialise from scratch... 2025-10-11 10:53:51.932336+00:00 [info] <0.250.0> Configured peer discovery backend: rabbit_peer_discovery_k8s 2025-10-11 10:53:51.932449+00:00 [info] <0.250.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s 2025-10-11 10:53:51.933294+00:00 [warning] <0.366.0> Description: "Authenticity is not established by certificate path validation" 2025-10-11 10:53:51.933294+00:00 [warning] <0.366.0> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing" 2025-10-11 10:53:51.933294+00:00 [warning] <0.366.0>  2025-10-11 10:53:51.960601+00:00 [info] <0.250.0> Will try to lock connecting to nodes ['rabbit@rabbitmq-cell1-server-2.rabbitmq-cell1-nodes.openstack', 2025-10-11 10:53:51.960601+00:00 [info] <0.250.0> 'rabbit@rabbitmq-cell1-server-1.rabbitmq-cell1-nodes.openstack', 2025-10-11 10:53:51.960601+00:00 [info] <0.250.0> 'rabbit@rabbitmq-cell1-server-0.rabbitmq-cell1-nodes.openstack'] 2025-10-11 10:53:57.606438+00:00 [info] <0.250.0> All discovered existing cluster peers: rabbit@rabbitmq-cell1-server-0.rabbitmq-cell1-nodes.openstack, rabbit@rabbitmq-cell1-server-1.rabbitmq-cell1-nodes.openstack, rabbit@rabbitmq-cell1-server-2.rabbitmq-cell1-nodes.openstack 2025-10-11 10:53:57.606542+00:00 [info] <0.250.0> Peer nodes we can cluster with: rabbit@rabbitmq-cell1-server-0.rabbitmq-cell1-nodes.openstack, rabbit@rabbitmq-cell1-server-2.rabbitmq-cell1-nodes.openstack 2025-10-11 10:53:57.612614+00:00 [info] <0.250.0> Node 'rabbit@rabbitmq-cell1-server-0.rabbitmq-cell1-nodes.openstack' selected for auto-clustering 2025-10-11 10:53:57.642696+00:00 [info] <0.250.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-11 10:53:57.643036+00:00 [info] <0.250.0> Successfully synced tables from a peer 2025-10-11 10:53:58.111291+00:00 [info] <0.250.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-11 10:53:58.111411+00:00 [info] <0.250.0> Successfully synced tables from a peer 2025-10-11 10:53:58.111460+00:00 [warning] <0.250.0> Feature flags: the previous instance of this node must have failed to write the `feature_flags` file at `/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cell1-server-1.rabbitmq-cell1-nodes.openstack-feature_flags`: 2025-10-11 10:53:58.111502+00:00 [warning] <0.250.0> Feature flags: - list of previously enabled feature flags now marked as such: [maintenance_mode_status, 2025-10-11 10:53:58.111502+00:00 [warning] <0.250.0> quorum_queue, 2025-10-11 10:53:58.111502+00:00 [warning] <0.250.0> user_limits, 2025-10-11 10:53:58.111502+00:00 [warning] <0.250.0> virtual_host_metadata] 2025-10-11 10:53:58.113389+00:00 [info] <0.250.0> Feature flags: list of feature flags found: 2025-10-11 10:53:58.113441+00:00 [info] <0.250.0> Feature flags: [ ] drop_unroutable_metric 2025-10-11 10:53:58.113461+00:00 [info] <0.250.0> Feature flags: [ ] empty_basic_get_metric 2025-10-11 10:53:58.113524+00:00 [info] <0.250.0> Feature flags: [ ] implicit_default_bindings 2025-10-11 10:53:58.113543+00:00 [info] <0.250.0> Feature flags: [x] maintenance_mode_status 2025-10-11 10:53:58.113559+00:00 [info] <0.250.0> Feature flags: [x] quorum_queue 2025-10-11 10:53:58.113621+00:00 [info] <0.250.0> Feature flags: [ ] stream_queue 2025-10-11 10:53:58.113638+00:00 [info] <0.250.0> Feature flags: [x] user_limits 2025-10-11 10:53:58.113650+00:00 [info] <0.250.0> Feature flags: [x] virtual_host_metadata 2025-10-11 10:53:58.113665+00:00 [info] <0.250.0> Feature flags: feature flag states written to disk: yes 2025-10-11 10:53:58.123066+00:00 [info] <0.250.0> Feature flag `drop_unroutable_metric`: mark as enabled=true 2025-10-11 10:53:58.126353+00:00 [info] <0.250.0> Feature flags: list of feature flags found: 2025-10-11 10:53:58.126407+00:00 [info] <0.250.0> Feature flags: [x] drop_unroutable_metric 2025-10-11 10:53:58.126428+00:00 [info] <0.250.0> Feature flags: [ ] empty_basic_get_metric 2025-10-11 10:53:58.126475+00:00 [info] <0.250.0> Feature flags: [ ] implicit_default_bindings 2025-10-11 10:53:58.126492+00:00 [info] <0.250.0> Feature flags: [x] maintenance_mode_status 2025-10-11 10:53:58.126505+00:00 [info] <0.250.0> Feature flags: [x] quorum_queue 2025-10-11 10:53:58.126542+00:00 [info] <0.250.0> Feature flags: [ ] stream_queue 2025-10-11 10:53:58.126555+00:00 [info] <0.250.0> Feature flags: [x] user_limits 2025-10-11 10:53:58.126566+00:00 [info] <0.250.0> Feature flags: [x] virtual_host_metadata 2025-10-11 10:53:58.126581+00:00 [info] <0.250.0> Feature flags: feature flag states written to disk: yes 2025-10-11 10:53:58.135763+00:00 [info] <0.250.0> Feature flag `empty_basic_get_metric`: mark as enabled=true 2025-10-11 10:53:58.139313+00:00 [info] <0.250.0> Feature flags: list of feature flags found: 2025-10-11 10:53:58.139368+00:00 [info] <0.250.0> Feature flags: [x] drop_unroutable_metric 2025-10-11 10:53:58.139384+00:00 [info] <0.250.0> Feature flags: [x] empty_basic_get_metric 2025-10-11 10:53:58.139407+00:00 [info] <0.250.0> Feature flags: [ ] implicit_default_bindings 2025-10-11 10:53:58.139421+00:00 [info] <0.250.0> Feature flags: [x] maintenance_mode_status 2025-10-11 10:53:58.139478+00:00 [info] <0.250.0> Feature flags: [x] quorum_queue 2025-10-11 10:53:58.139492+00:00 [info] <0.250.0> Feature flags: [ ] stream_queue 2025-10-11 10:53:58.139513+00:00 [info] <0.250.0> Feature flags: [x] user_limits 2025-10-11 10:53:58.139525+00:00 [info] <0.250.0> Feature flags: [x] virtual_host_metadata 2025-10-11 10:53:58.139540+00:00 [info] <0.250.0> Feature flags: feature flag states written to disk: yes 2025-10-11 10:53:58.149452+00:00 [info] <0.250.0> Waiting for Mnesia tables for 30000 ms, 0 retries left 2025-10-11 10:53:58.150996+00:00 [info] <0.250.0> Successfully synced tables from a peer 2025-10-11 10:53:58.151051+00:00 [info] <0.250.0> Feature flag `implicit_default_bindings`: mark as enabled=true 2025-10-11 10:53:58.154090+00:00 [info] <0.250.0> Feature flags: list of feature flags found: 2025-10-11 10:53:58.154167+00:00 [info] <0.250.0> Feature flags: [x] drop_unroutable_metric 2025-10-11 10:53:58.154196+00:00 [info] <0.250.0> Feature flags: [x] empty_basic_get_metric 2025-10-11 10:53:58.154241+00:00 [info] <0.250.0> Feature flags: [x] implicit_default_bindings 2025-10-11 10:53:58.154366+00:00 [info] <0.250.0> Feature flags: [x] maintenance_mode_status 2025-10-11 10:53:58.154392+00:00 [info] <0.250.0> Feature flags: [x] quorum_queue 2025-10-11 10:53:58.154446+00:00 [info] <0.250.0> Feature flags: [ ] stream_queue 2025-10-11 10:53:58.154471+00:00 [info] <0.250.0> Feature flags: [x] user_limits 2025-10-11 10:53:58.154492+00:00 [info] <0.250.0> Feature flags: [x] virtual_host_metadata 2025-10-11 10:53:58.154509+00:00 [info] <0.250.0> Feature flags: feature flag states written to disk: yes 2025-10-11 10:53:58.163049+00:00 [info] <0.250.0> Feature flag `stream_queue`: mark as enabled=true 2025-10-11 10:53:58.165504+00:00 [info] <0.250.0> Feature flags: list of feature flags found: 2025-10-11 10:53:58.165568+00:00 [info] <0.250.0> Feature flags: [x] drop_unroutable_metric 2025-10-11 10:53:58.165599+00:00 [info] <0.250.0> Feature flags: [x] empty_basic_get_metric 2025-10-11 10:53:58.165642+00:00 [info] <0.250.0> Feature flags: [x] implicit_default_bindings 2025-10-11 10:53:58.165669+00:00 [info] <0.250.0> Feature flags: [x] maintenance_mode_status 2025-10-11 10:53:58.165685+00:00 [info] <0.250.0> Feature flags: [x] quorum_queue 2025-10-11 10:53:58.165726+00:00 [info] <0.250.0> Feature flags: [x] stream_queue 2025-10-11 10:53:58.165747+00:00 [info] <0.250.0> Feature flags: [x] user_limits 2025-10-11 10:53:58.165760+00:00 [info] <0.250.0> Feature flags: [x] virtual_host_metadata 2025-10-11 10:53:58.165778+00:00 [info] <0.250.0> Feature flags: feature flag states written to disk: yes 2025-10-11 10:53:58.175320+00:00 [info] <0.250.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-11 10:53:58.175500+00:00 [info] <0.250.0> Successfully synced tables from a peer 2025-10-11 10:53:58.201919+00:00 [info] <0.250.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-11 10:53:58.202112+00:00 [info] <0.250.0> Successfully synced tables from a peer 2025-10-11 10:53:58.214065+00:00 [info] <0.250.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_rabbit@rabbitmq-cell1-server-1.rabbitmq-cell1-nodes.openstack' 2025-10-11 10:53:58.226829+00:00 [info] <0.250.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_rabbit@rabbitmq-cell1-server-1.rabbitmq-cell1-nodes.openstack' 2025-10-11 10:53:58.236579+00:00 [info] <0.250.0> Setting up a table for per-user connection counting on this node: 'tracked_connection_table_per_user_on_node_rabbit@rabbitmq-cell1-server-1.rabbitmq-cell1-nodes.openstack' 2025-10-11 10:53:58.237433+00:00 [info] <0.250.0> Will register with peer discovery backend rabbit_peer_discovery_k8s 2025-10-11 10:53:58.244498+00:00 [info] <0.250.0> Will try to unlock with peer discovery backend rabbit_peer_discovery_k8s 2025-10-11 10:53:58.245730+00:00 [info] <0.250.0> Running boot step database_sync defined by app rabbit 2025-10-11 10:53:58.245899+00:00 [info] <0.250.0> Running boot step feature_flags defined by app rabbit 2025-10-11 10:53:58.246139+00:00 [info] <0.250.0> Running boot step codec_correctness_check defined by app rabbit 2025-10-11 10:53:58.246252+00:00 [info] <0.250.0> Running boot step external_infrastructure defined by app rabbit 2025-10-11 10:53:58.246302+00:00 [info] <0.250.0> Running boot step rabbit_registry defined by app rabbit 2025-10-11 10:53:58.246393+00:00 [info] <0.250.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit 2025-10-11 10:53:58.246483+00:00 [info] <0.250.0> Running boot step rabbit_queue_location_random defined by app rabbit 2025-10-11 10:53:58.246601+00:00 [info] <0.250.0> Running boot step rabbit_event defined by app rabbit 2025-10-11 10:53:58.246813+00:00 [info] <0.250.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit 2025-10-11 10:53:58.246901+00:00 [info] <0.250.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit 2025-10-11 10:53:58.246959+00:00 [info] <0.250.0> Running boot step rabbit_exchange_type_direct defined by app rabbit 2025-10-11 10:53:58.247023+00:00 [info] <0.250.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit 2025-10-11 10:53:58.247071+00:00 [info] <0.250.0> Running boot step rabbit_exchange_type_headers defined by app rabbit 2025-10-11 10:53:58.247121+00:00 [info] <0.250.0> Running boot step rabbit_exchange_type_topic defined by app rabbit 2025-10-11 10:53:58.247173+00:00 [info] <0.250.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit 2025-10-11 10:53:58.247214+00:00 [info] <0.250.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit 2025-10-11 10:53:58.247246+00:00 [info] <0.250.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit 2025-10-11 10:53:58.247277+00:00 [info] <0.250.0> Running boot step rabbit_priority_queue defined by app rabbit 2025-10-11 10:53:58.247336+00:00 [info] <0.250.0> Priority queues enabled, real BQ is rabbit_variable_queue 2025-10-11 10:53:58.247385+00:00 [info] <0.250.0> Running boot step rabbit_queue_location_client_local defined by app rabbit 2025-10-11 10:53:58.247436+00:00 [info] <0.250.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit 2025-10-11 10:53:58.247476+00:00 [info] <0.250.0> Running boot step kernel_ready defined by app rabbit 2025-10-11 10:53:58.247496+00:00 [info] <0.250.0> Running boot step rabbit_sysmon_minder defined by app rabbit 2025-10-11 10:53:58.247652+00:00 [info] <0.250.0> Running boot step rabbit_epmd_monitor defined by app rabbit 2025-10-11 10:53:58.252392+00:00 [info] <0.555.0> epmd monitor knows us, inter-node communication (distribution) port: 25672 2025-10-11 10:53:58.252475+00:00 [info] <0.250.0> Running boot step guid_generator defined by app rabbit 2025-10-11 10:53:58.259605+00:00 [info] <0.250.0> Running boot step rabbit_node_monitor defined by app rabbit 2025-10-11 10:53:58.261802+00:00 [info] <0.559.0> Starting rabbit_node_monitor 2025-10-11 10:53:58.261951+00:00 [info] <0.250.0> Running boot step delegate_sup defined by app rabbit 2025-10-11 10:53:58.262490+00:00 [info] <0.250.0> Running boot step rabbit_memory_monitor defined by app rabbit 2025-10-11 10:53:58.262688+00:00 [info] <0.250.0> Running boot step core_initialized defined by app rabbit 2025-10-11 10:53:58.262728+00:00 [info] <0.250.0> Running boot step upgrade_queues defined by app rabbit 2025-10-11 10:53:58.271970+00:00 [info] <0.250.0> message_store upgrades: 1 to apply 2025-10-11 10:53:58.272171+00:00 [info] <0.250.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store 2025-10-11 10:53:58.272405+00:00 [info] <0.250.0> message_store upgrades: No durable queues found. Skipping message store migration 2025-10-11 10:53:58.272521+00:00 [info] <0.250.0> message_store upgrades: Removing the old message store data 2025-10-11 10:53:58.273078+00:00 [info] <0.250.0> message_store upgrades: All upgrades applied successfully 2025-10-11 10:53:58.286760+00:00 [info] <0.250.0> Running boot step channel_tracking defined by app rabbit 2025-10-11 10:53:58.299045+00:00 [info] <0.250.0> Setting up a table for channel tracking on this node: 'tracked_channel_on_node_rabbit@rabbitmq-cell1-server-1.rabbitmq-cell1-nodes.openstack' 2025-10-11 10:53:58.310140+00:00 [info] <0.250.0> Setting up a table for channel tracking on this node: 'tracked_channel_table_per_user_on_node_rabbit@rabbitmq-cell1-server-1.rabbitmq-cell1-nodes.openstack' 2025-10-11 10:53:58.310474+00:00 [info] <0.250.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit 2025-10-11 10:53:58.310540+00:00 [info] <0.250.0> Running boot step connection_tracking defined by app rabbit 2025-10-11 10:53:58.311418+00:00 [info] <0.250.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_rabbit@rabbitmq-cell1-server-1.rabbitmq-cell1-nodes.openstack' 2025-10-11 10:53:58.312185+00:00 [info] <0.250.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_rabbit@rabbitmq-cell1-server-1.rabbitmq-cell1-nodes.openstack' 2025-10-11 10:53:58.314246+00:00 [info] <0.250.0> Setting up a table for per-user connection counting on this node: 'tracked_connection_table_per_user_on_node_rabbit@rabbitmq-cell1-server-1.rabbitmq-cell1-nodes.openstack' 2025-10-11 10:53:58.314514+00:00 [info] <0.250.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit 2025-10-11 10:53:58.314583+00:00 [info] <0.250.0> Running boot step rabbit_exchange_parameters defined by app rabbit 2025-10-11 10:53:58.314689+00:00 [info] <0.250.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit 2025-10-11 10:53:58.315024+00:00 [info] <0.250.0> Running boot step rabbit_policies defined by app rabbit 2025-10-11 10:53:58.315385+00:00 [info] <0.250.0> Running boot step rabbit_policy defined by app rabbit 2025-10-11 10:53:58.315456+00:00 [info] <0.250.0> Running boot step rabbit_queue_location_validator defined by app rabbit 2025-10-11 10:53:58.315523+00:00 [info] <0.250.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit 2025-10-11 10:53:58.315586+00:00 [info] <0.250.0> Running boot step rabbit_stream_coordinator defined by app rabbit 2025-10-11 10:53:58.315796+00:00 [info] <0.250.0> Running boot step rabbit_vhost_limit defined by app rabbit 2025-10-11 10:53:58.315853+00:00 [info] <0.250.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management 2025-10-11 10:53:58.315929+00:00 [info] <0.250.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent 2025-10-11 10:53:58.315957+00:00 [info] <0.250.0> Management plugin: using rates mode 'basic' 2025-10-11 10:53:58.318685+00:00 [info] <0.250.0> Running boot step recovery defined by app rabbit 2025-10-11 10:53:58.320822+00:00 [info] <0.603.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cell1-server-1.rabbitmq-cell1-nodes.openstack/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists 2025-10-11 10:53:58.331142+00:00 [info] <0.603.0> Setting segment_entry_count for vhost '/' with 0 queues to '2048' 2025-10-11 10:53:58.338628+00:00 [info] <0.603.0> Starting message stores for vhost '/' 2025-10-11 10:53:58.338968+00:00 [info] <0.607.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index 2025-10-11 10:53:58.343512+00:00 [info] <0.603.0> Started message store of type transient for vhost '/' 2025-10-11 10:53:58.343815+00:00 [info] <0.611.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index 2025-10-11 10:53:58.347896+00:00 [warning] <0.611.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch 2025-10-11 10:53:58.349491+00:00 [info] <0.603.0> Started message store of type persistent for vhost '/' 2025-10-11 10:53:58.349973+00:00 [info] <0.603.0> Recovering 0 queues of type rabbit_classic_queue took 18ms 2025-10-11 10:53:58.350082+00:00 [info] <0.603.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms 2025-10-11 10:53:58.350156+00:00 [info] <0.603.0> Recovering 0 queues of type rabbit_stream_queue took 0ms 2025-10-11 10:53:58.351767+00:00 [info] <0.250.0> Running boot step empty_db_check defined by app rabbit 2025-10-11 10:53:58.351835+00:00 [info] <0.250.0> Will not seed default virtual host and user: have definitions to load... 2025-10-11 10:53:58.351856+00:00 [info] <0.250.0> Running boot step rabbit_looking_glass defined by app rabbit 2025-10-11 10:53:58.351912+00:00 [info] <0.250.0> Running boot step rabbit_core_metrics_gc defined by app rabbit 2025-10-11 10:53:58.352183+00:00 [info] <0.250.0> Running boot step background_gc defined by app rabbit 2025-10-11 10:53:58.352390+00:00 [info] <0.250.0> Running boot step routing_ready defined by app rabbit 2025-10-11 10:53:58.352414+00:00 [info] <0.250.0> Running boot step pre_flight defined by app rabbit 2025-10-11 10:53:58.352427+00:00 [info] <0.250.0> Running boot step notify_cluster defined by app rabbit 2025-10-11 10:53:58.352448+00:00 [info] <0.250.0> Running boot step networking defined by app rabbit 2025-10-11 10:53:58.352567+00:00 [info] <0.250.0> Running boot step rabbit_maintenance_mode_state defined by app rabbit 2025-10-11 10:53:58.352590+00:00 [info] <0.250.0> Creating table rabbit_node_maintenance_states for feature flag `maintenance_mode_status` 2025-10-11 10:53:58.353408+00:00 [info] <0.559.0> rabbit on node 'rabbit@rabbitmq-cell1-server-0.rabbitmq-cell1-nodes.openstack' up 2025-10-11 10:53:58.356993+00:00 [info] <0.250.0> Running boot step definition_import_worker_pool defined by app rabbit 2025-10-11 10:53:58.357047+00:00 [info] <0.307.0> Starting worker pool 'definition_import_pool' with 16 processes in it 2025-10-11 10:53:58.358039+00:00 [info] <0.250.0> Running boot step cluster_name defined by app rabbit 2025-10-11 10:53:58.358119+00:00 [info] <0.250.0> Setting cluster name to 'rabbitmq-cell1' as configured 2025-10-11 10:53:58.365358+00:00 [info] <0.250.0> Running boot step direct_client defined by app rabbit 2025-10-11 10:53:58.365603+00:00 [info] <0.250.0> Running boot step rabbit_management_load_definitions defined by app rabbitmq_management 2025-10-11 10:53:58.365782+00:00 [info] <0.655.0> Resetting node maintenance status 2025-10-11 10:53:58.412020+00:00 [info] <0.714.0> Management plugin: HTTP (non-TLS) listener started on port 15672 2025-10-11 10:53:58.426173+00:00 [info] <0.714.0> Management plugin: HTTPS listener started on port 15671 2025-10-11 10:53:58.426306+00:00 [info] <0.769.0> Statistics database started. 2025-10-11 10:53:58.426380+00:00 [info] <0.768.0> Starting worker pool 'management_worker_pool' with 3 processes in it 2025-10-11 10:53:58.436847+00:00 [info] <0.785.0> Peer discovery: node cleanup is disabled 2025-10-11 10:53:58.440469+00:00 [info] <0.793.0> Prometheus metrics: HTTP (non-TLS) listener started on port 15692 2025-10-11 10:53:58.444199+00:00 [info] <0.793.0> Prometheus metrics: HTTPS listener started on port 15691 2025-10-11 10:53:58.444341+00:00 [info] <0.655.0> Ready to start client connection listeners 2025-10-11 10:53:58.449130+00:00 [info] <0.866.0> started TLS (SSL) listener on [::]:5671 completed with 6 plugins. 2025-10-11 10:53:58.567772+00:00 [info] <0.655.0> Server startup complete; 6 plugins started. 2025-10-11 10:53:58.567772+00:00 [info] <0.655.0> * rabbitmq_prometheus 2025-10-11 10:53:58.567772+00:00 [info] <0.655.0> * rabbitmq_peer_discovery_k8s 2025-10-11 10:53:58.567772+00:00 [info] <0.655.0> * rabbitmq_peer_discovery_common 2025-10-11 10:53:58.567772+00:00 [info] <0.655.0> * rabbitmq_management 2025-10-11 10:53:58.567772+00:00 [info] <0.655.0> * rabbitmq_web_dispatch 2025-10-11 10:53:58.567772+00:00 [info] <0.655.0> * rabbitmq_management_agent 2025-10-11 10:54:03.453537+00:00 [info] <0.559.0> rabbit on node 'rabbit@rabbitmq-cell1-server-2.rabbitmq-cell1-nodes.openstack' up 2025-10-11 10:56:52.641495+00:00 [info] <0.1197.0> accepting AMQP connection <0.1197.0> (10.129.0.164:53496 -> 10.130.0.67:5671) 2025-10-11 10:56:52.643023+00:00 [info] <0.1197.0> Connection <0.1197.0> (10.129.0.164:53496 -> 10.130.0.67:5671) has a client-provided name: nova-conductor:1:5ee8fbd3-814b-433f-bcc0-d394efc7800b 2025-10-11 10:56:52.644403+00:00 [info] <0.1197.0> connection <0.1197.0> (10.129.0.164:53496 -> 10.130.0.67:5671 - nova-conductor:1:5ee8fbd3-814b-433f-bcc0-d394efc7800b): user 'default_user_Ih8CBxepTS8DcAgFqxu' authenticated and granted access to vhost '/' 2025-10-11 10:56:52.729367+00:00 [notice] <0.1213.0> queue 'conductor' in vhost '/': candidate -> leader in term: 1 machine version: 1 2025-10-11 10:56:52.730417+00:00 [info] <0.1206.0> ra: started cluster %2F_conductor with 3 servers 2025-10-11 10:56:52.730417+00:00 [info] <0.1206.0> 0 servers failed to start: [] 2025-10-11 10:56:52.730417+00:00 [info] <0.1206.0> Leader: {'%2F_conductor','rabbit@rabbitmq-cell1-server-1.rabbitmq-cell1-nodes.openstack'} 2025-10-11 10:56:52.899282+00:00 [notice] <0.1229.0> queue 'conductor.nova-cell1-conductor-0' in vhost '/': candidate -> leader in term: 1 machine version: 1 2025-10-11 10:56:52.900217+00:00 [info] <0.1206.0> ra: started cluster %2F_conductor.nova-cell1-conductor-0 with 3 servers 2025-10-11 10:56:52.900217+00:00 [info] <0.1206.0> 0 servers failed to start: [] 2025-10-11 10:56:52.900217+00:00 [info] <0.1206.0> Leader: {'%2F_conductor.nova-cell1-conductor-0','rabbit@rabbitmq-cell1-server-1.rabbitmq-cell1-nodes.openstack'} 2025-10-11 10:56:56.212075+00:00 [info] <0.1260.0> queue 'compute' in vhost '/': leader call - leader not known. Command will be forwarded once leader is known. 2025-10-11 10:56:56.235397+00:00 [info] <0.1260.0> queue 'compute' in vhost '/': granting vote for {'%2F_compute','rabbit@rabbitmq-cell1-server-0.rabbitmq-cell1-nodes.openstack'} with last indexterm {0,0} for term 1 previous term was 0 2025-10-11 10:56:56.249686+00:00 [info] <0.1260.0> queue 'compute' in vhost '/': detected a new leader {'%2F_compute','rabbit@rabbitmq-cell1-server-0.rabbitmq-cell1-nodes.openstack'} in term 1 2025-10-11 10:56:56.311510+00:00 [info] <0.1268.0> queue 'compute.nova-cell1-compute-ironic-compute-0' in vhost '/': leader call - leader not known. Command will be forwarded once leader is known. 2025-10-11 10:56:56.330535+00:00 [info] <0.1268.0> queue 'compute.nova-cell1-compute-ironic-compute-0' in vhost '/': granting vote for {'%2F_compute.nova-cell1-compute-ironic-compute-0','rabbit@rabbitmq-cell1-server-0.rabbitmq-cell1-nodes.openstack'} with last indexterm {0,0} for term 1 previous term was 0 2025-10-11 10:56:56.340645+00:00 [info] <0.1268.0> queue 'compute.nova-cell1-compute-ironic-compute-0' in vhost '/': detected a new leader {'%2F_compute.nova-cell1-compute-ironic-compute-0','rabbit@rabbitmq-cell1-server-0.rabbitmq-cell1-nodes.openstack'} in term 1 2025-10-11 11:28:03.463383+00:00 [info] <0.8268.0> accepting AMQP connection <0.8268.0> (100.64.0.3:47812 -> 10.130.0.67:5671) 2025-10-11 11:28:03.465301+00:00 [info] <0.8268.0> Connection <0.8268.0> (100.64.0.3:47812 -> 10.130.0.67:5671) has a client-provided name: nova-compute:2:5e5d1613-8064-4105-8fd5-6c3754f396a9 2025-10-11 11:28:03.467700+00:00 [info] <0.8268.0> connection <0.8268.0> (100.64.0.3:47812 -> 10.130.0.67:5671 - nova-compute:2:5e5d1613-8064-4105-8fd5-6c3754f396a9): user 'default_user_Ih8CBxepTS8DcAgFqxu' authenticated and granted access to vhost '/' 2025-10-11 11:28:06.226580+00:00 [warning] <0.8268.0> closing AMQP connection <0.8268.0> (100.64.0.3:47812 -> 10.130.0.67:5671 - nova-compute:2:5e5d1613-8064-4105-8fd5-6c3754f396a9, vhost: '/', user: 'default_user_Ih8CBxepTS8DcAgFqxu'): 2025-10-11 11:28:06.226580+00:00 [warning] <0.8268.0> client unexpectedly closed TCP connection 2025-10-11 11:28:10.184580+00:00 [info] <0.8330.0> accepting AMQP connection <0.8330.0> (100.64.0.3:47988 -> 10.130.0.67:5671) 2025-10-11 11:28:10.187157+00:00 [info] <0.8330.0> Connection <0.8330.0> (100.64.0.3:47988 -> 10.130.0.67:5671) has a client-provided name: nova-compute:2:18bf3379-2e35-4c87-9d60-a958515c60af 2025-10-11 11:28:10.190543+00:00 [info] <0.8330.0> connection <0.8330.0> (100.64.0.3:47988 -> 10.130.0.67:5671 - nova-compute:2:18bf3379-2e35-4c87-9d60-a958515c60af): user 'default_user_Ih8CBxepTS8DcAgFqxu' authenticated and granted access to vhost '/' 2025-10-11 11:28:10.280209+00:00 [info] <0.8347.0> accepting AMQP connection <0.8347.0> (100.64.0.3:50102 -> 10.130.0.67:5671) 2025-10-11 11:28:10.282350+00:00 [info] <0.8347.0> Connection <0.8347.0> (100.64.0.3:50102 -> 10.130.0.67:5671) has a client-provided name: nova-compute:2:7d1acb36-e721-4440-9502-bb79559f8e95 2025-10-11 11:28:10.284667+00:00 [info] <0.8347.0> connection <0.8347.0> (100.64.0.3:50102 -> 10.130.0.67:5671 - nova-compute:2:7d1acb36-e721-4440-9502-bb79559f8e95): user 'default_user_Ih8CBxepTS8DcAgFqxu' authenticated and granted access to vhost '/' 2025-10-11 11:28:13.169075+00:00 [info] <0.8368.0> accepting AMQP connection <0.8368.0> (100.64.0.3:50118 -> 10.130.0.67:5671) 2025-10-11 11:28:13.172109+00:00 [info] <0.8368.0> Connection <0.8368.0> (100.64.0.3:50118 -> 10.130.0.67:5671) has a client-provided name: nova-compute:2:1f2e9b43-b04f-4233-82af-afba436a06b5 2025-10-11 11:28:13.177661+00:00 [info] <0.8368.0> connection <0.8368.0> (100.64.0.3:50118 -> 10.130.0.67:5671 - nova-compute:2:1f2e9b43-b04f-4233-82af-afba436a06b5): user 'default_user_Ih8CBxepTS8DcAgFqxu' authenticated and granted access to vhost '/' 2025-10-11 11:28:13.267819+00:00 [notice] <0.8391.0> queue 'compute.edpm-compute-0.ctlplane.openstack.lab' in vhost '/': candidate -> leader in term: 1 machine version: 1 2025-10-11 11:28:13.268893+00:00 [info] <0.8382.0> ra: started cluster %2F_compute.edpm-compute-0.ctlplane.openstack.lab with 3 servers 2025-10-11 11:28:13.268893+00:00 [info] <0.8382.0> 0 servers failed to start: [] 2025-10-11 11:28:13.268893+00:00 [info] <0.8382.0> Leader: {'%2F_compute.edpm-compute-0.ctlplane.openstack.lab','rabbit@rabbitmq-cell1-server-1.rabbitmq-cell1-nodes.openstack'} 2025-10-11 11:28:13.464921+00:00 [info] <0.8418.0> queue 'compute.edpm-compute-1.ctlplane.openstack.lab' in vhost '/': granting vote for {'%2F_compute.edpm-compute-1.ctlplane.openstack.lab','rabbit@rabbitmq-cell1-server-2.rabbitmq-cell1-nodes.openstack'} with last indexterm {0,0} for term 1 previous term was 0 2025-10-11 11:28:13.476209+00:00 [info] <0.8418.0> queue 'compute.edpm-compute-1.ctlplane.openstack.lab' in vhost '/': detected a new leader {'%2F_compute.edpm-compute-1.ctlplane.openstack.lab','rabbit@rabbitmq-cell1-server-2.rabbitmq-cell1-nodes.openstack'} in term 1