:_mod-docs-content-type: PROCEDURE [id="relocating-one-instance-of-a-monitoring-stack-to-migrate-daemons-to-target-nodes_{context}"] = Scenario 2: Relocating one instance of a monitoring stack to migrate daemons to target nodes [role="_abstract"] Instead of adding a single `monitoring` label to all the target nodes, it is possible to relocate one instance of each monitoring stack daemon on a particular node. .Procedure . Set each of your nodes to host a particular daemon instance, for example, if you have three target nodes: + ---- [tripleo-admin@controller-0 ~]$ sudo cephadm shell -- ceph orch host ls | grep -i cephstorage HOST ADDR LABELS cephstorage-0.redhat.local 192.168.24.11 osd ---> grafana cephstorage-1.redhat.local 192.168.24.12 osd ---> prometheus cephstorage-2.redhat.local 192.168.24.47 osd ---> alertmanager ---- . Add the appropriate labels to the target nodes: + ---- declare -A target_nodes target_nodes[grafana]=cephstorage-0 target_nodes[prometheus]=cephstorage-1 target_nodes[alertmanager]=cephstorage-2 for label in "${!target_nodes[@]}"; do ceph orch host label add ${target_nodes[$label]} $label done ---- . Verify that the labels are properly applied to the target nodes: + ---- [tripleo-admin@controller-0 ~]$ sudo cephadm shell -- ceph orch host ls | grep -i cephstorage HOST ADDR LABELS STATUS cephstorage-0.redhat.local 192.168.24.11 osd grafana cephstorage-1.redhat.local 192.168.24.12 osd prometheus cephstorage-2.redhat.local 192.168.24.47 osd alertmanager ---- . Dump the current monitoring stack spec: + ---- function export_spec { local component="$1" local target_dir="$2" sudo cephadm shell -- ceph orch ls --export "$component" > "$target_dir/$component" } SPEC_DIR=${SPEC_DIR:-"$PWD/ceph_specs"} for m in grafana prometheus alertmanager; do export_spec "$m" "$SPEC_DIR" done ---- . For each daemon, edit the current spec and replace the placement/hosts section with the placement/label section, for example: + [source,yaml] ---- service_type: grafana service_name: grafana placement: label: grafana networks: - 172.17.3.0/24 spec: port: 3100 ---- + The same procedure applies to Prometheus and Alertmanager specs. . Apply the new monitoring spec to relocate the monitoring stack daemons: + ---- SPEC_DIR=${SPEC_DIR:-"$PWD/ceph_specs"} function migrate_daemon { local component="$1" local target_dir="$2" sudo cephadm shell -m "$target_dir" -- ceph orch apply -i /mnt/ceph_specs/$component } for m in grafana prometheus alertmanager; do migrate_daemon "$m" "$SPEC_DIR" done ---- . Verify that the daemons are deployed on the expected nodes: + ---- [ceph: root@controller-0 /]# ceph orch ps | grep -iE "(prome|alert|grafa)" alertmanager.cephstorage-2 cephstorage-2.redhat.local 172.17.3.144:9093,9094 grafana.cephstorage-0 cephstorage-0.redhat.local 172.17.3.83:3100 prometheus.cephstorage-1 cephstorage-1.redhat.local 172.17.3.53:9092 ---- + [NOTE] With the procedure described above we lose High Availability: the monitoring stack daemons have no VIP and haproxy anymore; Node exporters are still running on all the nodes: instead of using labels we keep the current approach as we want to not reduce the monitoring space covered. + //kgilliga: What does "the procedure described above" refer to? . Update the Ceph Dashboard Manager configuration. An important aspect that should be considered at this point is to replace and verify that the {Ceph} configuration is aligned with the relocation you just made. Run the `ceph config dump` command and review the current config. In particular, focus on the following configuration entries: //kgilliga: If we're saying that customers should consider this aspect, is this step optional? + ---- [ceph: root@controller-0 /]# ceph config dump ... mgr advanced mgr/dashboard/ALERTMANAGER_API_HOST http://172.17.3.83:9093 mgr advanced mgr/dashboard/GRAFANA_API_URL https://172.17.3.144:3100 mgr advanced mgr/dashboard/PROMETHEUS_API_HOST http://172.17.3.83:9092 mgr advanced mgr/dashboard/controller-0.ycokob/server_addr 172.17.3.33 mgr advanced mgr/dashboard/controller-1.lmzpuc/server_addr 172.17.3.147 mgr advanced mgr/dashboard/controller-2.xpdgfl/server_addr 172.17.3.138 ---- + . Verify that `grafana`, `alertmanager` and `prometheus` `API_HOST/URL` point to the IP addresses (on the storage network) of the node where each daemon has been relocated. This should be automatically addressed by cephadm and it shouldn’t require any manual action. + ---- [ceph: root@controller-0 /]# ceph orch ps | grep -iE "(prome|alert|grafa)" alertmanager.cephstorage-0 cephstorage-0.redhat.local 172.17.3.83:9093,9094 alertmanager.cephstorage-1 cephstorage-1.redhat.local 172.17.3.53:9093,9094 alertmanager.cephstorage-2 cephstorage-2.redhat.local 172.17.3.144:9093,9094 grafana.cephstorage-0 cephstorage-0.redhat.local 172.17.3.83:3100 grafana.cephstorage-1 cephstorage-1.redhat.local 172.17.3.53:3100 grafana.cephstorage-2 cephstorage-2.redhat.local 172.17.3.144:3100 prometheus.cephstorage-0 cephstorage-0.redhat.local 172.17.3.83:9092 prometheus.cephstorage-1 cephstorage-1.redhat.local 172.17.3.53:9092 prometheus.cephstorage-2 cephstorage-2.redhat.local 172.17.3.144:9092 ---- + ---- [ceph: root@controller-0 /]# ceph config dump ... ... mgr advanced mgr/dashboard/ALERTMANAGER_API_HOST http://172.17.3.83:9093 mgr advanced mgr/dashboard/PROMETHEUS_API_HOST http://172.17.3.83:9092 mgr advanced mgr/dashboard/GRAFANA_API_URL https://172.17.3.144:3100 ---- + . The Ceph Dashboard (mgr module plugin) has not been impacted at all by this relocation. The service is provided by the Ceph Manager daemon, hence we might experience an impact when the active mgr is migrated or is force-failed. However, having three replicas definition allows to redirect requests to a different instance (it’s still an A/P model), hence the impact should be limited. .. When the RBD migration is over, the following {Ceph} config keys must be regenerated to point to the right mgr container: //kgilliga: Is the assumption that customers have already done the RBD migration before doing this procedure? RBD migration comes before this chapter in the adoption guide. + ---- mgr advanced mgr/dashboard/controller-0.ycokob/server_addr 172.17.3.33 mgr advanced mgr/dashboard/controller-1.lmzpuc/server_addr 172.17.3.147 mgr advanced mgr/dashboard/controller-2.xpdgfl/server_addr 172.17.3.138 ---- + ---- $ sudo cephadm shell $ ceph orch ps | awk '/mgr./ {print $1}' ---- .. For each retrieved mgr, update the entry in the {Ceph} configuration: + ---- $ ceph config set mgr mgr/dashboard/<>/server_addr/ ----