:_mod-docs-content-type: PROCEDURE
[id="relocating-one-instance-of-a-monitoring-stack-to-migrate-daemons-to-target-nodes_{context}"]

= Scenario 2: Relocating one instance of a monitoring stack to migrate daemons to target nodes

[role="_abstract"]
Instead of adding a single `monitoring` label to all the target nodes, it is
possible to relocate one instance of each monitoring stack daemon on a
particular node.

.Procedure

. Set each of your nodes to host a particular daemon instance, for example, if you have three target nodes:
+
----
[tripleo-admin@controller-0 ~]$ sudo cephadm shell -- ceph orch host ls | grep -i cephstorage

HOST                        ADDR           LABELS
cephstorage-0.redhat.local  192.168.24.11  osd ---> grafana
cephstorage-1.redhat.local  192.168.24.12  osd ---> prometheus
cephstorage-2.redhat.local  192.168.24.47  osd ---> alertmanager
----

. Add the appropriate labels to the target nodes:
+
----
declare -A target_nodes

target_nodes[grafana]=cephstorage-0
target_nodes[prometheus]=cephstorage-1
target_nodes[alertmanager]=cephstorage-2

for label in "${!target_nodes[@]}"; do
    ceph orch host label add ${target_nodes[$label]} $label
done
----

. Verify that the labels are properly applied to the target nodes:
+
----
[tripleo-admin@controller-0 ~]$ sudo cephadm shell -- ceph orch host ls | grep -i cephstorage

HOST                    	ADDR       	LABELS          	STATUS
cephstorage-0.redhat.local  192.168.24.11  osd grafana
cephstorage-1.redhat.local  192.168.24.12  osd prometheus
cephstorage-2.redhat.local  192.168.24.47  osd alertmanager
----

. Dump the current monitoring stack spec:
+
----
function export_spec {
    local component="$1"
    local target_dir="$2"
    sudo cephadm shell -- ceph orch ls --export "$component" > "$target_dir/$component"
}

SPEC_DIR=${SPEC_DIR:-"$PWD/ceph_specs"}
for m in grafana prometheus alertmanager; do
    export_spec "$m" "$SPEC_DIR"
done
----

. For each daemon, edit the current spec and replace the placement/hosts section
with the placement/label section, for example:
+
[source,yaml]
----
service_type: grafana
service_name: grafana
placement:
  label: grafana
networks:
- 172.17.3.0/24
spec:
  port: 3100
----
+
The same procedure applies to Prometheus and Alertmanager specs.

. Apply the new monitoring spec to relocate the monitoring stack daemons:
+
----
SPEC_DIR=${SPEC_DIR:-"$PWD/ceph_specs"}
function migrate_daemon {
    local component="$1"
    local target_dir="$2"
    sudo cephadm shell -m "$target_dir" -- ceph orch apply -i /mnt/ceph_specs/$component
}
for m in grafana prometheus alertmanager; do
    migrate_daemon  "$m" "$SPEC_DIR"
done
----

. Verify that the daemons are deployed on the expected nodes:
+
----
[ceph: root@controller-0 /]# ceph orch ps | grep -iE "(prome|alert|grafa)"
alertmanager.cephstorage-2  cephstorage-2.redhat.local  172.17.3.144:9093,9094
grafana.cephstorage-0       cephstorage-0.redhat.local  172.17.3.83:3100
prometheus.cephstorage-1    cephstorage-1.redhat.local  172.17.3.53:9092
----
+
[NOTE]
With the procedure described above we lose High Availability: the monitoring
stack daemons have no VIP and haproxy anymore; Node exporters are still
running on all the nodes: instead of using labels we keep the current approach
as we want to not reduce the monitoring space covered.
+
//kgilliga: What does "the procedure described above" refer to?
. Update the Ceph Dashboard Manager configuration. An important aspect that should be considered at this point is to replace and
verify that the {Ceph} configuration is aligned with the relocation you just made. Run the `ceph config dump` command and review the current config.
In particular, focus on the following configuration entries:
//kgilliga: If we're saying that customers should consider this aspect, is this step optional?
+
----
[ceph: root@controller-0 /]# ceph config dump
...
mgr  advanced  mgr/dashboard/ALERTMANAGER_API_HOST  http://172.17.3.83:9093
mgr  advanced  mgr/dashboard/GRAFANA_API_URL        https://172.17.3.144:3100
mgr  advanced  mgr/dashboard/PROMETHEUS_API_HOST    http://172.17.3.83:9092
mgr  advanced  mgr/dashboard/controller-0.ycokob/server_addr  172.17.3.33
mgr  advanced  mgr/dashboard/controller-1.lmzpuc/server_addr  172.17.3.147
mgr  advanced  mgr/dashboard/controller-2.xpdgfl/server_addr  172.17.3.138
----
+
. Verify that `grafana`, `alertmanager` and `prometheus` `API_HOST/URL` point to
the IP addresses (on the storage network) of the node where each daemon has been
relocated. This should be automatically addressed by cephadm and it shouldn’t
require any manual action.
+
----
[ceph: root@controller-0 /]# ceph orch ps | grep -iE "(prome|alert|grafa)"
alertmanager.cephstorage-0  cephstorage-0.redhat.local  172.17.3.83:9093,9094
alertmanager.cephstorage-1  cephstorage-1.redhat.local  172.17.3.53:9093,9094
alertmanager.cephstorage-2  cephstorage-2.redhat.local  172.17.3.144:9093,9094
grafana.cephstorage-0       cephstorage-0.redhat.local  172.17.3.83:3100
grafana.cephstorage-1       cephstorage-1.redhat.local  172.17.3.53:3100
grafana.cephstorage-2       cephstorage-2.redhat.local  172.17.3.144:3100
prometheus.cephstorage-0    cephstorage-0.redhat.local  172.17.3.83:9092
prometheus.cephstorage-1    cephstorage-1.redhat.local  172.17.3.53:9092
prometheus.cephstorage-2    cephstorage-2.redhat.local  172.17.3.144:9092
----
+
----
[ceph: root@controller-0 /]# ceph config dump
...
...
mgr  advanced  mgr/dashboard/ALERTMANAGER_API_HOST   http://172.17.3.83:9093
mgr  advanced  mgr/dashboard/PROMETHEUS_API_HOST     http://172.17.3.83:9092
mgr  advanced  mgr/dashboard/GRAFANA_API_URL         https://172.17.3.144:3100
----
+
. The Ceph Dashboard (mgr module plugin) has not been impacted at all by this
relocation. The service is provided by the Ceph Manager daemon, hence we might
experience an impact when the active mgr is migrated or is force-failed.
However, having three replicas definition allows to redirect requests to a
different instance (it’s still an A/P model), hence the impact should be
limited.
.. When the RBD migration is over, the following {Ceph} config keys must
be regenerated to point to the right mgr container:
//kgilliga: Is the assumption that customers have already done the RBD migration before doing this procedure? RBD migration comes before this chapter in the adoption guide.
+
----
mgr    advanced  mgr/dashboard/controller-0.ycokob/server_addr  172.17.3.33
mgr    advanced  mgr/dashboard/controller-1.lmzpuc/server_addr  172.17.3.147
mgr    advanced  mgr/dashboard/controller-2.xpdgfl/server_addr  172.17.3.138
----
+
----
$ sudo cephadm shell
$ ceph orch ps | awk '/mgr./ {print $1}'
----

.. For each retrieved mgr, update the entry in the {Ceph} configuration:
+
----
$ ceph config set mgr mgr/dashboard/<>/server_addr/<ip addr>
----
