:_mod-docs-content-type: PROCEDURE
[id="migrating-object-storage-data-to-rhoso-nodes_{context}"]

= Migrating the {object_storage} data from {OpenStackShort} to {rhos_acro} nodes

[role="_abstract"]
The {object_storage_first_ref} migration involves the following steps:

. Add new nodes to the {object_storage} rings.
. Set weights of existing nodes to 0.
. Rebalance rings by moving one replica.
. Copy rings to old nodes and restart services.
. Check replication status and repeat the previous two steps until the old nodes are drained.
. Remove the old nodes from the rings.

.Prerequisites

* Adopt the {object_storage}. For more information, see xref:adopting-the-object-storage-service_adopt-control-plane[Adopting the {object_storage}].
* For DNS servers, ensure that all existing nodes are able to resolve the hostnames of the {rhocp_long} pods, for example, by using the external IP of the DNSMasq service as the nameserver in `/etc/resolv.conf`:
+
----
$ oc get service dnsmasq-dns -o jsonpath="{.status.loadBalancer.ingress[0].ip}" | $CONTROLLER1_SSH sudo tee /etc/resolv.conf
----
* Track the current status of the replication by using the `swift-dispersion` tool:
+
----
$ oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c 'swift-ring-tool get && swift-dispersion-populate'
----
+
The command might need a few minutes to complete. It creates 0-byte objects that are distributed across the {object_storage} deployment, and you can use the `swift-dispersion-report` afterward to show the current replication status:
+
----
$ oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c 'swift-ring-tool get && swift-dispersion-report'
----
+
The output of the `swift-dispersion-report` command looks similar to the following:
+
----
Queried 1024 containers for dispersion reporting, 5s, 0 retries
100.00% of container copies found (3072 of 3072)
Sample represents 100.00% of the container partition space
Queried 1024 objects for dispersion reporting, 4s, 0 retries
There were 1024 partitions missing 0 copies.
100.00% of object copies found (3072 of 3072)
Sample represents 100.00% of the object partition space
----

.Procedure

. Add new nodes by scaling up the SwiftStorage resource from 0 to 3:
// TODO add paragraph / link on EDPM node usage for Swift
+
----
$ oc patch openstackcontrolplane openstack --type=merge -p='{"spec":{"swift":{"template":{"swiftStorage":{"replicas": 3}}}}}'
----
+
This command creates three storage instances on the {rhocp_long} cluster that use Persistent Volume Claims.

. Wait until all three pods are running and the rings include the new devices:
+
----
$ oc wait pods --for condition=Ready -l component=swift-storage
$ oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c 'swift-ring-tool get && swift-ring-builder object.builder search --device pv'
----

. From the current rings, get the storage management IP addresses of the nodes to drain:
+
----
$ oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c 'swift-ring-tool get && swift-ring-builder object.builder search _' | tail -n +2 | awk '{print $4}' | sort -u

----
+
The output looks similar to the following:
+
----
172.20.0.100
swift-storage-0.swift-storage.openstack.svc
swift-storage-1.swift-storage.openstack.svc
swift-storage-2.swift-storage.openstack.svc
----

. Drain the old nodes. In the following example, the old node `172.20.0.100` is drained:
+
----
$ oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c '
> swift-ring-tool get
> swift-ring-tool drain 172.20.0.100
> swift-ring-tool rebalance
> swift-ring-tool push'
----
+
Depending on your deployment, you might have more nodes to include in the command.

. Copy and apply the updated rings to the original nodes. Run the
ssh commands for your existing nodes that store the {object_storage} data:
+
----
$ oc extract --confirm cm/swift-ring-files
$ $CONTROLLER1_SSH "tar -C /var/lib/config-data/puppet-generated/swift/etc/swift/ -xzf -" < swiftrings.tar.gz
$ $CONTROLLER1_SSH "systemctl restart tripleo_swift_*"
----

. Track the replication progress by using the `swift-dispersion-report` tool:
+
----
$ oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c "swift-ring-tool get && swift-dispersion-report"
----
+
The output shows less than 100% of copies found. Repeat the command until all container and object copies are found:
+
----
Queried 1024 containers for dispersion reporting, 6s, 0 retries
There were 5 partitions missing 1 copy.
99.84% of container copies found (3067 of 3072)
Sample represents 100.00% of the container partition space
Queried 1024 objects for dispersion reporting, 7s, 0 retries
There were 739 partitions missing 1 copy.
There were 285 partitions missing 0 copies.
75.94% of object copies found (2333 of 3072)
Sample represents 100.00% of the object partition space
----

+
[NOTE]
The rebalance command moves only one replica at a time to ensure that data is available continuously. This requires running the rebalance command multiple times to complete the full rebalance operation.
Additionally, a minimum wait time of one hour between consecutive rebalance commands is
enforced to prevent moving multiple replicas at the same time. Running the
rebalance again before this period elapses has no effect.
+

. Move the next replica to the new nodes by rebalancing and distributing the rings:
+
----
$ oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c '
> swift-ring-tool get
> swift-ring-tool rebalance
> swift-ring-tool push'

$ oc extract --confirm cm/swift-ring-files
$ $CONTROLLER1_SSH "tar -C /var/lib/config-data/puppet-generated/swift/etc/swift/ -xzf -" < swiftrings.tar.gz
$ $CONTROLLER1_SSH "systemctl restart tripleo_swift_*"
----
+
Monitor the `swift-dispersion-report` output again, wait until all copies are found, and then repeat this step until all your replicas are moved to the new nodes.

. Remove the nodes from the rings:
+
----
$ oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c '
> swift-ring-tool get
> swift-ring-tool remove 172.20.0.100
> swift-ring-tool rebalance
> swift-ring-tool push'
----
+
[NOTE]
Even if all replicas are on the new nodes and the `swift-dispersion-report` command reports 100% of the copies found, there might still be data on the old nodes. The replicators remove this data, but it might take more time.

.Verification

. Check the disk usage of all disks in the cluster:
+
----
$ oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c 'swift-ring-tool get && swift-recon -d'
----

. Confirm that there are no more `\*.db` or `*.data` files in the `/srv/node` directory on the nodes:
+
----
$CONTROLLER1_SSH "find /srv/node/ -type f -name '*.db' -o -name '*.data' | wc -l"
----