Nov 24 14:26:46.404717 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:26:46.404717 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:27:46.399345 np0005533992.novalocal cinder-backup[122429]: DEBUG dbcounter [-] [122429] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=122429) stat_writer /usr/local/lib/python3.12/site-packages/dbcounter.py:115}} Nov 24 14:27:46.408522 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:27:46.408793 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:28:46.415934 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:28:46.415934 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:29:46.405792 np0005533992.novalocal cinder-backup[122429]: DEBUG dbcounter [-] [122429] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=122429) stat_writer /usr/local/lib/python3.12/site-packages/dbcounter.py:115}} Nov 24 14:29:46.413232 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:29:46.413461 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:30:46.415540 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:30:46.415994 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:31:46.413325 np0005533992.novalocal cinder-backup[122429]: DEBUG dbcounter [-] [122429] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=122429) stat_writer /usr/local/lib/python3.12/site-packages/dbcounter.py:115}} Nov 24 14:31:46.415530 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:31:46.415789 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:32:46.419469 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:32:46.419942 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:33:46.414174 np0005533992.novalocal cinder-backup[122429]: DEBUG dbcounter [-] [122429] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=122429) stat_writer /usr/local/lib/python3.12/site-packages/dbcounter.py:115}} Nov 24 14:33:46.415549 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:33:46.415809 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:34:46.415881 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:34:46.416311 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:35:46.420683 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:35:46.421066 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:35:46.432105 np0005533992.novalocal cinder-backup[122429]: DEBUG dbcounter [-] [122429] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=122429) stat_writer /usr/local/lib/python3.12/site-packages/dbcounter.py:115}} Nov 24 14:36:46.429260 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:36:46.429260 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:37:46.426144 np0005533992.novalocal cinder-backup[122429]: DEBUG dbcounter [-] [122429] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=122429) stat_writer /usr/local/lib/python3.12/site-packages/dbcounter.py:115}} Nov 24 14:37:46.428862 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:37:46.429081 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:37:58.616908 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Create backup started, backup: 053a65ac-3a09-4a4b-80b8-5a9d4de11a74 volume: 5410a40c-b38e-41f4-907e-6d228834ea7d. Nov 24 14:37:58.621970 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Call Volume Manager to get_backup_device for Backup(availability_zone='nova',container='tempest-volumesbackupstest-backup-container-241182277',created_at=2025-11-24T19:37:58Z,data_timestamp=2025-11-24T19:37:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-1557939192',encryption_key_id=None,fail_reason=None,host='np0005533992.novalocal',id=053a65ac-3a09-4a4b-80b8-5a9d4de11a74,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='c642367dafaa45dd89634e7ec9822ef2',restore_volume_id=None,service='cinder.backup.drivers.swift.SwiftBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='0e48df3b35f84ee88136fd47ee531c1f',volume_id=5410a40c-b38e-41f4-907e-6d228834ea7d) Nov 24 14:38:00.097683 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Using swift URL https://38.102.83.12:8080/v1/AUTH_c642367dafaa45dd89634e7ec9822ef2 {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:286}} Nov 24 14:38:00.097683 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Connect to https://38.102.83.12:8080/v1/AUTH_ in "per_user" mode {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:287}} Nov 24 14:38:00.098253 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '38.102.83.12', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:38:00.098977 np0005533992.novalocal cinder-backup[122429]: INFO oslo.privsep.daemon [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Running privsep helper: ['sudo', 'cinder-rootwrap', '/etc/cinder/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/cinder/cinder.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmp465as7qf/privsep.sock'] Nov 24 14:38:00.113391 np0005533992.novalocal sudo[159734]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf privsep-helper --config-file /etc/cinder/cinder.conf --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmp465as7qf/privsep.sock Nov 24 14:38:00.825520 np0005533992.novalocal cinder-backup[122429]: INFO oslo.privsep.daemon [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Spawned new privsep daemon via rootwrap Nov 24 14:38:00.826028 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Accepted privsep connection to /tmp/tmp465as7qf/privsep.sock {{(pid=122429) __init__ /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:377}} Nov 24 14:38:00.826932 np0005533992.novalocal cinder-backup[122429]: INFO oslo.privsep.daemon [-] privsep daemon starting Nov 24 14:38:00.827142 np0005533992.novalocal cinder-backup[122429]: INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 Nov 24 14:38:00.827322 np0005533992.novalocal cinder-backup[122429]: INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_DAC_READ_SEARCH|CAP_SYS_ADMIN/CAP_DAC_READ_SEARCH|CAP_SYS_ADMIN/none Nov 24 14:38:00.827495 np0005533992.novalocal cinder-backup[122429]: INFO oslo.privsep.daemon [-] privsep daemon running as pid 159746 Nov 24 14:38:00.829057 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[c53d26d2-74bf-4b59-ab0c-2ba1174fb635]: (2,) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:00.902394 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:00.909097 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.007s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:00.909444 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[c9834ca0-fe6a-445c-9516-e95358a3f4a7]: (4, ('InitiatorName=iqn.1994-05.com.redhat:5b9c7bcf3f78\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:00.912060 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[1bb2fcdf-7826-4a67-a724-87df3f6b26da]: [Errno 2] No such file or directory: '/dev/scini' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:38:00.912060 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:38:00.912060 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:38:00.912060 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:38:00.912060 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:00.912060 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:38:00.912060 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:38:00.912060 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:00.912060 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 57, in get_guid Nov 24 14:38:00.912060 np0005533992.novalocal cinder-backup[122429]: with open_scini_device() as fd: Nov 24 14:38:00.912060 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:00.912060 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/contextlib.py", line 137, in __enter__ Nov 24 14:38:00.912060 np0005533992.novalocal cinder-backup[122429]: return next(self.gen) Nov 24 14:38:00.912060 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^ Nov 24 14:38:00.912060 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 40, in open_scini_device Nov 24 14:38:00.912060 np0005533992.novalocal cinder-backup[122429]: fd = os.open(SCINI_DEVICE_PATH, os.O_RDWR) Nov 24 14:38:00.912060 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:00.912060 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: '/dev/scini' Nov 24 14:38:00.912666 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[1bb2fcdf-7826-4a67-a724-87df3f6b26da]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 57, in get_guid\n with open_scini_device() as fd:\n ^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/contextlib.py", line 137, in __enter__\n return next(self.gen)\n ^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 40, in open_scini_device\n fd = os.open(SCINI_DEVICE_PATH, os.O_RDWR)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory: \'/dev/scini\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:00.913457 np0005533992.novalocal cinder-backup[122429]: ERROR os_brick.initiator.connectors.scaleio [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Error querying sdc guid: [Errno 2] No such file or directory: FileNotFoundError: [Errno 2] No such file or directory Nov 24 14:38:00.914062 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.scaleio [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Unable to find SDC guid: Error querying sdc guid: [Errno 2] No such file or directory Nov 24 14:38:00.914530 np0005533992.novalocal cinder-backup[122429]: WARNING os_brick.initiator.connectors.nvmeof [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Could not find nvme_core/parameters/multipath: FileNotFoundError: [Errno 2] No such file or directory: '/sys/module/nvme_core/parameters/multipath' Nov 24 14:38:00.915576 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt -v / -n -o SOURCE {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:00.924515 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt -v / -n -o SOURCE" returned: 0 in 0.009s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:00.924799 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[d7dec9a1-a401-47ff-b15b-77695a757653]: (4, ('/dev/vda2\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:00.926421 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/vda2 -s UUID -o value {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:00.931506 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/vda2 -s UUID -o value" returned: 0 in 0.006s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:00.931773 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[c24d15e8-e32e-4f4b-877d-a2784ca1e113]: (4, ('3fc68afb-d7e4-475c-9164-ea34f338d965\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:00.933016 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[2a8d9368-a6b2-4842-a21d-97c0aa7e1707]: (4, 'e2c36df9-c117-4156-b18b-d355565d8e8e') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:00.933514 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Running cmd (subprocess): nvme version {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:00.941016 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] 'nvme version' failed. Not Retrying. {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:38:00.941237 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.nvmeof [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] nvme not present on system {{(pid=122429) nvme_present /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/nvmeof.py:784}} Nov 24 14:38:00.942789 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=122429) find_dsc /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:132}} Nov 24 14:38:00.943472 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.lightos [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Current host hostNQN nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e and IP(s) are ['38.102.83.12', 'fe80::f816:3eff:fe17:2fa8', '192.168.122.1', '172.24.5.1', '2001:db8::2', 'fe80::64fc:4aff:fe7b:de4b', 'fe80::fc16:3eff:feae:621f', 'fe80::58e0:b7ff:fe2e:5631'] Nov 24 14:38:00.943617 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=122429) get_connector_properties /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:109}} Nov 24 14:38:00.944088 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e dsc: {{(pid=122429) get_connector_properties /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:112}} Nov 24 14:38:00.944088 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] <== get_connector_properties: return (846ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '38.102.83.12', 'host': 'np0005533992.novalocal', 'multipath': False, 'enforce_multipath': False, 'initiator': 'iqn.1994-05.com.redhat:5b9c7bcf3f78', 'do_local_attach': False, 'uuid': '3fc68afb-d7e4-475c-9164-ea34f338d965', 'system uuid': 'e2c36df9-c117-4156-b18b-d355565d8e8e', 'nvme_native_multipath': False, 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e', 'found_dsc': '', 'host_ips': ['38.102.83.12', 'fe80::f816:3eff:fe17:2fa8', '192.168.122.1', '172.24.5.1', '2001:db8::2', 'fe80::64fc:4aff:fe7b:de4b', 'fe80::fc16:3eff:feae:621f', 'fe80::58e0:b7ff:fe2e:5631']} {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:38:04.458125 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connector [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Factory for iscsi on None {{(pid=122429) factory /usr/local/lib/python3.12/site-packages/os_brick/initiator/connector.py:281}} Nov 24 14:38:04.459792 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] ==> connect_volume: call "{'self': , 'connection_properties': {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d', 'target_lun': 0, 'volume_id': 'e265de99-6228-4929-987d-8aca0e16408d', 'auth_method': 'CHAP', 'auth_username': 'HnDVF5qbNQAZ7m366TEn', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False}}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:38:04.460200 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Acquiring lock "connect_volume" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:38:04.460997 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:38:04.461317 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Connection properties {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d', 'target_lun': 0, 'volume_id': 'e265de99-6228-4929-987d-8aca0e16408d', 'auth_method': 'CHAP', 'auth_username': 'HnDVF5qbNQAZ7m366TEn', 'auth_password': 'HtYXD2NJytR7k2xo', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False, 'enforce_multipath': False} {{(pid=122429) check_multipath /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:133}} Nov 24 14:38:04.463964 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd show status {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:04.468490 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] 'multipathd show status' failed. Not Retrying. {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:38:04.477666 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[7e631181-bdb9-4719-b8cb-5958ec7ee918]: [Errno 2] No such file or directory: 'multipathd' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:38:04.477666 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:38:04.477666 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:38:04.477666 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:38:04.477666 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:04.477666 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:38:04.477666 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:38:04.477666 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:04.477666 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root Nov 24 14:38:04.477666 np0005533992.novalocal cinder-backup[122429]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Nov 24 14:38:04.477666 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:04.477666 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute Nov 24 14:38:04.477666 np0005533992.novalocal cinder-backup[122429]: return putils.execute(on_execute=on_execute, Nov 24 14:38:04.477666 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:04.477666 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute Nov 24 14:38:04.477666 np0005533992.novalocal cinder-backup[122429]: obj = subprocess.Popen(cmd, Nov 24 14:38:04.477666 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:04.477666 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__ Nov 24 14:38:04.477666 np0005533992.novalocal cinder-backup[122429]: self._execute_child(args, executable, preexec_fn, close_fds, Nov 24 14:38:04.477666 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child Nov 24 14:38:04.477666 np0005533992.novalocal cinder-backup[122429]: raise child_exception_type(errno_num, err_msg, err_filename) Nov 24 14:38:04.477666 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: 'multipathd' Nov 24 14:38:04.478596 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[7e631181-bdb9-4719-b8cb-5958ec7ee918]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root\n return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute\n return putils.execute(on_execute=on_execute,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute\n obj = subprocess.Popen(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__\n self._execute_child(args, executable, preexec_fn, close_fds,\n File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child\n raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: \'multipathd\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:04.479783 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Acquiring lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:38:04.480411 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:38:04.480870 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Trying to connect to iSCSI portal 38.102.83.12:3260 Nov 24 14:38:04.482596 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d -p 38.102.83.12:3260 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:04.490895 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d -p 38.102.83.12:3260" returned: 21 in 0.008s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:04.490895 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[ed888b86-7226-462e-a577-8bd5aa2437c4]: (4, ('', 'iscsiadm: No records found\n')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:04.491172 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] iscsiadm (): stdout= stderr=iscsiadm: No records found Nov 24 14:38:04.491172 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:04.492044 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d -p 38.102.83.12:3260 --interface default --op new {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:04.500671 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d -p 38.102.83.12:3260 --interface default --op new" returned: 0 in 0.009s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:04.501083 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[1c0a213a-5ee7-4a3a-bd49-83f14f5274e8]: (4, ('New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 38.102.83.12,3260,-1 iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d] added\n', 'iscsiadm: Config file line 381 does not have value\niscsiadm: Config file line 381 does not have value\niscsiadm: Config file line 381 does not have value\n')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:04.501426 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--interface', 'default', '--op', 'new'): stdout=New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 38.102.83.12,3260,-1 iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d] added Nov 24 14:38:04.501426 np0005533992.novalocal cinder-backup[122429]: stderr=iscsiadm: Config file line 381 does not have value Nov 24 14:38:04.501426 np0005533992.novalocal cinder-backup[122429]: iscsiadm: Config file line 381 does not have value Nov 24 14:38:04.501426 np0005533992.novalocal cinder-backup[122429]: iscsiadm: Config file line 381 does not have value Nov 24 14:38:04.501426 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:04.501622 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Retrying to connect to iSCSI portal 38.102.83.12:3260 {{(pid=122429) _connect_to_iscsi_portal_unsafe /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1133}} Nov 24 14:38:04.502020 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Finished call to 'os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe' after 0.021(s), this was the 1st time calling it. {{(pid=122429) log_it /usr/local/lib/python3.12/site-packages/tenacity/after.py:44}} Nov 24 14:38:04.502212 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Retrying os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe in 1.0 seconds as it raised BrickException: An unknown exception occurred.. {{(pid=122429) log_it /usr/local/lib/python3.12/site-packages/tenacity/before_sleep.py:65}} Nov 24 14:38:05.504407 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Trying to connect to iSCSI portal 38.102.83.12:3260 Nov 24 14:38:05.505910 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d -p 38.102.83.12:3260 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:05.517605 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d -p 38.102.83.12:3260" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:05.518444 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[200a1601-f005-480e-a500-5b3009ff24cc]: (4, ('# BEGIN RECORD 6.2.1.11\nnode.name = iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d\nnode.tpgt = -1\nnode.startup = automatic\nnode.leading_login = No\niface.iscsi_ifacename = default\niface.net_ifacename = \niface.ipaddress = \niface.prefix_len = 0\niface.hwaddress = \niface.transport_name = tcp\niface.initiatorname = \niface.state = \niface.vlan_id = 0\niface.vlan_priority = 0\niface.vlan_state = \niface.iface_num = 0\niface.mtu = 0\niface.port = 0\niface.bootproto = \niface.subnet_mask = \niface.gateway = \niface.dhcp_alt_client_id_state = \niface.dhcp_alt_client_id = \niface.dhcp_dns = \niface.dhcp_learn_iqn = \niface.dhcp_req_vendor_id_state = \niface.dhcp_vendor_id_state = \niface.dhcp_vendor_id = \niface.dhcp_slp_da = \niface.fragmentation = \niface.gratuitous_arp = \niface.incoming_forwarding = \niface.tos_state = \niface.tos = 0\niface.ttl = 0\niface.delayed_ack = \niface.tcp_nagle = \niface.tcp_wsf_state = \niface.tcp_wsf = 0\niface.tcp_timer_scale = 0\niface.tcp_timestamp = \niface.redirect = \niface.def_task_mgmt_timeout = 0\niface.header_digest = \niface.data_digest = \niface.immediate_data = \niface.initial_r2t = \niface.data_seq_inorder = \niface.data_pdu_inorder = \niface.erl = 0\niface.max_receive_data_len = 0\niface.first_burst_len = 0\niface.max_outstanding_r2t = 0\niface.max_burst_len = 0\niface.chap_auth = \niface.bidi_chap = \niface.strict_login_compliance = \niface.discovery_auth = \niface.discovery_logout = \nnode.discovery_address = \nnode.discovery_port = 0\nnode.discovery_type = fw\nnode.session.initial_cmdsn = 0\nnode.session.initial_login_retry_max = 8\nnode.session.xmit_thread_priority = 0\nnode.session.cmds_max = 128\nnode.session.queue_depth = 32\nnode.session.nr_sessions = 1\nnode.session.auth.authmethod = None\nnode.session.auth.username = \nnode.session.auth.password = \nnode.session.auth.username_in = \nnode.session.auth.password_in = \nnode.session.auth.chap_algs = SHA3-256,SHA256\nnode.session.timeo.replacement_timeout = 120\nnode.session.err_timeo.abort_timeout = 15\nnode.session.err_timeo.lu_reset_timeout = 30\nnode.session.err_timeo.tgt_reset_timeout = 30\nnode.session.err_timeo.host_reset_timeout = 60\nnode.session.iscsi.FastAbort = Yes\nnode.session.iscsi.InitialR2T = No\nnode.session.iscsi.ImmediateData = Yes\nnode.session.iscsi.FirstBurstLength = 262144\nnode.session.iscsi.MaxBurstLength = 16776192\nnode.session.iscsi.DefaultTime2Retain = 0\nnode.session.iscsi.DefaultTime2Wait = 2\nnode.session.iscsi.MaxConnections = 1\nnode.session.iscsi.MaxOutstandingR2T = 1\nnode.session.iscsi.ERL = 0\nnode.session.scan = auto\nnode.session.reopen_max = 0\nnode.session.conn_reopen_log_freq = 1\nnode.conn[0].address = 38.102.83.12\nnode.conn[0].port = 3260\nnode.conn[0].startup = manual\nnode.conn[0].tcp.window_size = 524288\nnode.conn[0].tcp.type_of_service = 0\nnode.conn[0].timeo.logout_timeout = 15\nnode.conn[0].timeo.login_timeout = 15\nnode.conn[0].timeo.auth_timeout = 45\nnode.conn[0].timeo.noop_out_interval = 5\nnode.conn[0].timeo.noop_out_timeout = 5\nnode.conn[0].iscsi.MaxXmitDataSegmentLength = 0\nnode.conn[0].iscsi.MaxRecvDataSegmentLength = 262144\nnode.conn[0].iscsi.HeaderDigest = None\nnode.conn[0].iscsi.DataDigest = None\nnode.conn[0].iscsi.IFMarker = No\nnode.conn[0].iscsi.OFMarker = No\n# END RECORD\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] iscsiadm (): stdout=# BEGIN RECORD 6.2.1.11 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.name = iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.tpgt = -1 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.startup = automatic Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.leading_login = No Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.iscsi_ifacename = default Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.net_ifacename = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.ipaddress = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.prefix_len = 0 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.hwaddress = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.transport_name = tcp Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.initiatorname = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.state = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.vlan_id = 0 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.vlan_priority = 0 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.vlan_state = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.iface_num = 0 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.mtu = 0 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.port = 0 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.bootproto = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.subnet_mask = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.gateway = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_alt_client_id_state = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_alt_client_id = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_dns = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_learn_iqn = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_req_vendor_id_state = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_vendor_id_state = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_vendor_id = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_slp_da = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.fragmentation = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.gratuitous_arp = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.incoming_forwarding = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.tos_state = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.tos = 0 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.ttl = 0 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.delayed_ack = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.tcp_nagle = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.tcp_wsf_state = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.tcp_wsf = 0 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.tcp_timer_scale = 0 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.tcp_timestamp = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.redirect = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.def_task_mgmt_timeout = 0 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.header_digest = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.data_digest = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.immediate_data = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.initial_r2t = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.data_seq_inorder = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.data_pdu_inorder = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.erl = 0 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.max_receive_data_len = 0 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.first_burst_len = 0 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.max_outstanding_r2t = 0 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.max_burst_len = 0 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.chap_auth = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.bidi_chap = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.strict_login_compliance = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.discovery_auth = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: iface.discovery_logout = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.discovery_address = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.discovery_port = 0 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.discovery_type = fw Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.initial_cmdsn = 0 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.initial_login_retry_max = 8 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.xmit_thread_priority = 0 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.cmds_max = 128 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.queue_depth = 32 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.nr_sessions = 1 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.auth.authmethod = None Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.auth.username = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.auth.password = *** Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.auth.username_in = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.auth.password_in = Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.auth.chap_algs = SHA3-256,SHA256 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.timeo.replacement_timeout = 120 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.abort_timeout = 15 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.lu_reset_timeout = 30 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.tgt_reset_timeout = 30 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.host_reset_timeout = 60 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.FastAbort = Yes Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.InitialR2T = No Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.ImmediateData = Yes Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.FirstBurstLength = 262144 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxBurstLength = 16776192 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.DefaultTime2Retain = 0 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.DefaultTime2Wait = 2 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxConnections = 1 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxOutstandingR2T = 1 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.ERL = 0 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.scan = auto Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.reopen_max = 0 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.session.conn_reopen_log_freq = 1 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.conn[0].address = 38.102.83.12 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.conn[0].port = 3260 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.conn[0].startup = manual Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.conn[0].tcp.window_size = 524288 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.conn[0].tcp.type_of_service = 0 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.logout_timeout = 15 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.login_timeout = 15 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.auth_timeout = 45 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.noop_out_interval = 5 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.noop_out_timeout = 5 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.MaxXmitDataSegmentLength = 0 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144 Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.HeaderDigest = None Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.DataDigest = None Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.IFMarker = No Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.OFMarker = No Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: # END RECORD Nov 24 14:38:05.520275 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:05.525372 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d -p 38.102.83.12:3260 --op update -n node.session.scan -v manual {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:05.536626 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d -p 38.102.83.12:3260 --op update -n node.session.scan -v manual" returned: 0 in 0.015s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:05.537624 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[f9c25e3b-4639-4b59-9880-6d0c64142b2a]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:05.538061 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.scan', '-v', 'manual'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:05.541979 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d -p 38.102.83.12:3260 --op update -n node.session.auth.authmethod -v CHAP {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:05.548703 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d -p 38.102.83.12:3260 --op update -n node.session.auth.authmethod -v CHAP" returned: 0 in 0.009s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:05.549506 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[9d87d863-13c4-4907-8c54-522b9c99460e]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:05.550247 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.authmethod', '-v', 'CHAP'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:05.552407 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d -p 38.102.83.12:3260 --op update -n node.session.auth.username -v HnDVF5qbNQAZ7m366TEn {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:05.559582 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d -p 38.102.83.12:3260 --op update -n node.session.auth.username -v HnDVF5qbNQAZ7m366TEn" returned: 0 in 0.008s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:05.560800 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[52e4f320-09f7-45e3-baed-692d146678d7]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:05.561315 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.username', '-v', 'HnDVF5qbNQAZ7m366TEn'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:05.564195 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d -p 38.102.83.12:3260 --op update -n node.session.auth.password -v *** {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:05.572756 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d -p 38.102.83.12:3260 --op update -n node.session.auth.password -v ***" returned: 0 in 0.009s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:05.572905 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[99a998a9-c943-4857-af28-63a9094dee34]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:05.573521 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.password', '-v', '***'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:05.574502 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:05.585918 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:05.586105 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[968c671f-a98d-4401-b655-a2900f4809cc]: (4, ('tcp: [19] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-5410a40c-b38e-41f4-907e-6d228834ea7d (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:05.586667 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('-m', 'session'): stdout=tcp: [19] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-5410a40c-b38e-41f4-907e-6d228834ea7d (non-flash) Nov 24 14:38:05.586667 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:38:05.586871 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] iscsi session list stdout=tcp: [19] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-5410a40c-b38e-41f4-907e-6d228834ea7d (non-flash) Nov 24 14:38:05.586871 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:38:05.587884 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d -p 38.102.83.12:3260 --login {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:05.604652 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d -p 38.102.83.12:3260 --login" returned: 0 in 0.017s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:05.604936 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[73528261-7693-4ef1-b5d4-73be40e431ef]: (4, ('Login to [iface: default, target: iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d, portal: 38.102.83.12,3260] successful.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:05.605460 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--login',): stdout=Login to [iface: default, target: iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d, portal: 38.102.83.12,3260] successful. Nov 24 14:38:05.605460 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:05.606459 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d -p 38.102.83.12:3260 --op update -n node.startup -v automatic {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:05.617463 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d -p 38.102.83.12:3260 --op update -n node.startup -v automatic" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:05.617800 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[1deabc16-7a26-4581-b251-6543b6c00f05]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:05.618267 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:05.619323 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:05.628544 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.009s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:05.628887 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[5ee7822c-e3c8-4db3-b3fc-63ad090bb566]: (4, ('tcp: [19] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-5410a40c-b38e-41f4-907e-6d228834ea7d (non-flash)\ntcp: [20] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:05.629315 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('-m', 'session'): stdout=tcp: [19] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-5410a40c-b38e-41f4-907e-6d228834ea7d (non-flash) Nov 24 14:38:05.629315 np0005533992.novalocal cinder-backup[122429]: tcp: [20] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d (non-flash) Nov 24 14:38:05.629315 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:38:05.629638 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] iscsi session list stdout=tcp: [19] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-5410a40c-b38e-41f4-907e-6d228834ea7d (non-flash) Nov 24 14:38:05.629638 np0005533992.novalocal cinder-backup[122429]: tcp: [20] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d (non-flash) Nov 24 14:38:05.629638 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:38:05.630165 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: held 1.150s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:38:05.630430 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Connected to 38.102.83.12:3260 {{(pid=122429) _connect_vol /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:659}} Nov 24 14:38:05.632057 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] HCTL ('3', '-', '-', 0) found on session 20 with lun 0 {{(pid=122429) get_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:788}} Nov 24 14:38:05.632318 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Scanning host 3 c: -, t: -, l: 0) {{(pid=122429) scan_iscsi /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:814}} Nov 24 14:38:05.633472 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/class/scsi_host/host3/scan {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:05.657971 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/class/scsi_host/host3/scan" returned: 0 in 0.022s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:05.658246 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[0b3ad497-a3c2-4326-82e7-1446c386ca59]: (4, ('- - 0', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:05.659655 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Searching for a device in session 20 and hctl ['3', '*', '*', 0] yield: None {{(pid=122429) device_name_by_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:808}} Nov 24 14:38:06.661033 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Searching for a device in session 20 and hctl ['3', '*', '*', 0] yield: sdb {{(pid=122429) device_name_by_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:808}} Nov 24 14:38:06.661834 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Connected to sdb using {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d', 'target_lun': 0, 'volume_id': 'e265de99-6228-4929-987d-8aca0e16408d', 'auth_method': 'CHAP', 'auth_username': 'HnDVF5qbNQAZ7m366TEn', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False} {{(pid=122429) _connect_vol /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:687}} Nov 24 14:38:06.662793 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" :: held 2.202s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:38:06.663146 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] <== connect_volume: return (2203ms) {'type': 'block', 'scsi_wwn': '3600140546e7cde2639c4b5a85a22383a', 'path': '/dev/sdb'} {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:38:06.663803 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sdb {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:06.687795 np0005533992.novalocal sudo[159889]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sdb Nov 24 14:38:07.052851 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sdb" returned: 0 in 0.389s {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:07.059327 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_db.sqlalchemy.engines [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=122429) _check_effective_sql_mode /usr/local/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:397}} Nov 24 14:38:07.069433 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] _create_container started, container: tempest-volumesbackupstest-backup-container-241182277,backup: 053a65ac-3a09-4a4b-80b8-5a9d4de11a74. {{(pid=122429) _create_container /opt/stack/cinder/cinder/backup/chunkeddriver.py:252}} Nov 24 14:38:07.275605 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] generate_object_name_prefix: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74 {{(pid=122429) _generate_object_name_prefix /opt/stack/cinder/cinder/backup/drivers/swift.py:433}} Nov 24 14:38:07.280174 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] starting backup of volume: 5410a40c-b38e-41f4-907e-6d228834ea7d, volume size: 1073741824, object names prefix volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74, availability zone: nova {{(pid=122429) _prepare_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:362}} Nov 24 14:38:07.422002 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:07.539023 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:07.539154 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:07.656225 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00001: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:07.657393 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:07.889384 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:08.004847 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:08.004847 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:08.144386 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00002: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:08.144762 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:08.323204 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:08.422476 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:08.422606 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:08.559630 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00003: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:08.559630 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:08.752579 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:08.878312 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:08.878600 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:09.022106 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00004: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:09.022626 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:09.175406 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:09.272702 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:09.272872 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:09.398285 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00005: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:09.398427 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:09.533382 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:09.656738 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:09.656738 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:09.772621 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00006: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:09.772794 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:09.997627 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:10.153127 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:10.153127 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:10.299334 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00007: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:10.299623 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:10.437835 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:10.535293 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:10.535545 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:10.680463 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00008: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:10.680816 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:10.790589 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:10.887395 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:10.887888 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:11.046188 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00009: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:11.046308 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:11.149154 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:11.245512 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:11.245512 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:11.380280 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00010: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:11.380406 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:11.473337 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Create backup started, backup: b805dbfc-103d-4564-a6c5-40e433409863 volume: 8d10e5cf-1417-4901-8774-045390352639. Nov 24 14:38:11.478449 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Call Volume Manager to get_backup_device for Backup(availability_zone='nova',container='tempest-volumesbackupsv39test-backup-container-225728091',created_at=2025-11-24T19:38:11Z,data_timestamp=2025-11-24T19:38:11Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsV39Test-Backup-1325089779',encryption_key_id=None,fail_reason=None,host='np0005533992.novalocal',id=b805dbfc-103d-4564-a6c5-40e433409863,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='107d295432b146088ae2cfd7c6d08054',restore_volume_id=None,service='cinder.backup.drivers.swift.SwiftBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='e7c9e2495e1d4506a8f41d6f3262e5ff',volume_id=8d10e5cf-1417-4901-8774-045390352639) Nov 24 14:38:11.515134 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Using swift URL https://38.102.83.12:8080/v1/AUTH_107d295432b146088ae2cfd7c6d08054 {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:286}} Nov 24 14:38:11.515444 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Connect to https://38.102.83.12:8080/v1/AUTH_ in "per_user" mode {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:287}} Nov 24 14:38:11.516095 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '38.102.83.12', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:38:11.518365 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:11.526540 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.009s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:11.526824 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[a2b09c2f-c22d-4a06-9fe4-a5646bb02d16]: (4, ('InitiatorName=iqn.1994-05.com.redhat:5b9c7bcf3f78\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:11.528998 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[631c94bc-4b11-4fe8-b31e-cd1e4da103d7]: [Errno 2] No such file or directory: '/dev/scini' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:38:11.528998 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:38:11.528998 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:38:11.528998 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:38:11.528998 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:11.528998 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:38:11.528998 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:38:11.528998 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:11.528998 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 57, in get_guid Nov 24 14:38:11.528998 np0005533992.novalocal cinder-backup[122429]: with open_scini_device() as fd: Nov 24 14:38:11.528998 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:11.528998 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/contextlib.py", line 137, in __enter__ Nov 24 14:38:11.528998 np0005533992.novalocal cinder-backup[122429]: return next(self.gen) Nov 24 14:38:11.528998 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^ Nov 24 14:38:11.528998 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 40, in open_scini_device Nov 24 14:38:11.528998 np0005533992.novalocal cinder-backup[122429]: fd = os.open(SCINI_DEVICE_PATH, os.O_RDWR) Nov 24 14:38:11.528998 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:11.528998 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: '/dev/scini' Nov 24 14:38:11.529544 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[631c94bc-4b11-4fe8-b31e-cd1e4da103d7]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 57, in get_guid\n with open_scini_device() as fd:\n ^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/contextlib.py", line 137, in __enter__\n return next(self.gen)\n ^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 40, in open_scini_device\n fd = os.open(SCINI_DEVICE_PATH, os.O_RDWR)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory: \'/dev/scini\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:11.530096 np0005533992.novalocal cinder-backup[122429]: ERROR os_brick.initiator.connectors.scaleio [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Error querying sdc guid: [Errno 2] No such file or directory: FileNotFoundError: [Errno 2] No such file or directory Nov 24 14:38:11.530579 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.scaleio [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Unable to find SDC guid: Error querying sdc guid: [Errno 2] No such file or directory Nov 24 14:38:11.531416 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt -v / -n -o SOURCE {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:11.570087 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt -v / -n -o SOURCE" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:11.571152 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[1d68edb1-cee9-4d32-9d21-b73e5741d60e]: (4, ('/dev/vda2\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:11.571575 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/vda2 -s UUID -o value {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:11.581001 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/vda2 -s UUID -o value" returned: 0 in 0.009s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:11.581276 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[b9e80420-4ade-4df3-8ed0-9feeea60154a]: (4, ('3fc68afb-d7e4-475c-9164-ea34f338d965\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:11.582375 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[f579ca81-3b48-4dfe-9a7d-ebaa64776207]: (4, 'e2c36df9-c117-4156-b18b-d355565d8e8e') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:11.583000 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Running cmd (subprocess): nvme version {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:11.601502 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] 'nvme version' failed. Not Retrying. {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:38:11.601954 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.nvmeof [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] nvme not present on system {{(pid=122429) nvme_present /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/nvmeof.py:784}} Nov 24 14:38:11.603891 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=122429) find_dsc /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:132}} Nov 24 14:38:11.604946 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.lightos [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Current host hostNQN nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e and IP(s) are ['38.102.83.12', 'fe80::f816:3eff:fe17:2fa8', '192.168.122.1', '172.24.5.1', '2001:db8::2', 'fe80::64fc:4aff:fe7b:de4b', 'fe80::fc16:3eff:feae:621f', 'fe80::58e0:b7ff:fe2e:5631'] Nov 24 14:38:11.605233 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=122429) get_connector_properties /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:109}} Nov 24 14:38:11.605535 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e dsc: {{(pid=122429) get_connector_properties /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:112}} Nov 24 14:38:11.606194 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] <== get_connector_properties: return (89ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '38.102.83.12', 'host': 'np0005533992.novalocal', 'multipath': False, 'enforce_multipath': False, 'initiator': 'iqn.1994-05.com.redhat:5b9c7bcf3f78', 'do_local_attach': False, 'uuid': '3fc68afb-d7e4-475c-9164-ea34f338d965', 'system uuid': 'e2c36df9-c117-4156-b18b-d355565d8e8e', 'nvme_native_multipath': False, 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e', 'found_dsc': '', 'host_ips': ['38.102.83.12', 'fe80::f816:3eff:fe17:2fa8', '192.168.122.1', '172.24.5.1', '2001:db8::2', 'fe80::64fc:4aff:fe7b:de4b', 'fe80::fc16:3eff:feae:621f', 'fe80::58e0:b7ff:fe2e:5631']} {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:38:11.712194 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:11.865884 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:11.866223 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:12.007026 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00011: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:12.007374 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:12.014485 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_db.sqlalchemy.engines [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=122429) _check_effective_sql_mode /usr/local/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:397}} Nov 24 14:38:12.397690 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:12.574464 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:12.574755 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:12.714717 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00012: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:12.714896 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:12.857630 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:12.971653 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:12.971931 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:13.090180 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00013: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:13.090180 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:13.278091 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:13.410595 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:13.410595 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:13.515149 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00014: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:13.515442 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:14.054681 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:14.154277 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:14.154883 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:14.312893 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00015: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:14.312893 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:14.487206 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:14.582324 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:14.582324 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:14.734043 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00016: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:14.734151 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:14.944264 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:15.065620 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:15.065951 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:15.115177 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connector [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Factory for iscsi on None {{(pid=122429) factory /usr/local/lib/python3.12/site-packages/os_brick/initiator/connector.py:281}} Nov 24 14:38:15.116544 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] ==> connect_volume: call "{'self': , 'connection_properties': {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639', 'target_lun': 0, 'volume_id': '8d10e5cf-1417-4901-8774-045390352639', 'auth_method': 'CHAP', 'auth_username': '9XkGSPtjXGatiZ24v5gy', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False}}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:38:15.116891 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Acquiring lock "connect_volume" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:38:15.117509 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:38:15.117860 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Connection properties {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639', 'target_lun': 0, 'volume_id': '8d10e5cf-1417-4901-8774-045390352639', 'auth_method': 'CHAP', 'auth_username': '9XkGSPtjXGatiZ24v5gy', 'auth_password': 'br2xHdwwaqW9oSU9', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False, 'enforce_multipath': False} {{(pid=122429) check_multipath /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:133}} Nov 24 14:38:15.119086 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd show status {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:15.125741 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] 'multipathd show status' failed. Not Retrying. {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:38:15.128105 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[24640b10-ad2b-41f6-93b9-1eb3266a42ac]: [Errno 2] No such file or directory: 'multipathd' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:38:15.128105 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:38:15.128105 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:38:15.128105 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:38:15.128105 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:15.128105 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:38:15.128105 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:38:15.128105 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:15.128105 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root Nov 24 14:38:15.128105 np0005533992.novalocal cinder-backup[122429]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Nov 24 14:38:15.128105 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:15.128105 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute Nov 24 14:38:15.128105 np0005533992.novalocal cinder-backup[122429]: return putils.execute(on_execute=on_execute, Nov 24 14:38:15.128105 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:15.128105 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute Nov 24 14:38:15.128105 np0005533992.novalocal cinder-backup[122429]: obj = subprocess.Popen(cmd, Nov 24 14:38:15.128105 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:15.128105 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__ Nov 24 14:38:15.128105 np0005533992.novalocal cinder-backup[122429]: self._execute_child(args, executable, preexec_fn, close_fds, Nov 24 14:38:15.128105 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child Nov 24 14:38:15.128105 np0005533992.novalocal cinder-backup[122429]: raise child_exception_type(errno_num, err_msg, err_filename) Nov 24 14:38:15.128105 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: 'multipathd' Nov 24 14:38:15.129318 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[24640b10-ad2b-41f6-93b9-1eb3266a42ac]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root\n return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute\n return putils.execute(on_execute=on_execute,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute\n obj = subprocess.Popen(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__\n self._execute_child(args, executable, preexec_fn, close_fds,\n File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child\n raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: \'multipathd\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:15.129593 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Acquiring lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:38:15.130088 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:38:15.130325 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Trying to connect to iSCSI portal 38.102.83.12:3260 Nov 24 14:38:15.131074 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 -p 38.102.83.12:3260 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:15.142655 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 -p 38.102.83.12:3260" returned: 21 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:15.142965 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[da92ab0c-b34d-4ff2-91d0-8386613fc104]: (4, ('', 'iscsiadm: No records found\n')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:15.143306 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] iscsiadm (): stdout= stderr=iscsiadm: No records found Nov 24 14:38:15.143306 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:15.144097 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 -p 38.102.83.12:3260 --interface default --op new {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:15.155445 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 -p 38.102.83.12:3260 --interface default --op new" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:15.155782 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[f6615d96-de0b-44e6-a88d-151668f8bce9]: (4, ('New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 38.102.83.12,3260,-1 iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639] added\n', 'iscsiadm: Config file line 381 does not have value\niscsiadm: Config file line 381 does not have value\niscsiadm: Config file line 381 does not have value\n')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:15.156088 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] iscsiadm ('--interface', 'default', '--op', 'new'): stdout=New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 38.102.83.12,3260,-1 iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639] added Nov 24 14:38:15.156088 np0005533992.novalocal cinder-backup[122429]: stderr=iscsiadm: Config file line 381 does not have value Nov 24 14:38:15.156088 np0005533992.novalocal cinder-backup[122429]: iscsiadm: Config file line 381 does not have value Nov 24 14:38:15.156088 np0005533992.novalocal cinder-backup[122429]: iscsiadm: Config file line 381 does not have value Nov 24 14:38:15.156088 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:15.156296 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Retrying to connect to iSCSI portal 38.102.83.12:3260 {{(pid=122429) _connect_to_iscsi_portal_unsafe /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1133}} Nov 24 14:38:15.156795 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Finished call to 'os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe' after 0.026(s), this was the 1st time calling it. {{(pid=122429) log_it /usr/local/lib/python3.12/site-packages/tenacity/after.py:44}} Nov 24 14:38:15.157024 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Retrying os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe in 1.0 seconds as it raised BrickException: An unknown exception occurred.. {{(pid=122429) log_it /usr/local/lib/python3.12/site-packages/tenacity/before_sleep.py:65}} Nov 24 14:38:15.217533 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00017: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:15.217737 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:15.648897 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:15.752842 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:15.752918 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:15.912089 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00018: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:15.912303 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:16.106495 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:16.158267 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Trying to connect to iSCSI portal 38.102.83.12:3260 Nov 24 14:38:16.160365 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 -p 38.102.83.12:3260 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:16.171790 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 -p 38.102.83.12:3260" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:16.172061 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[6f8e4cb4-3764-4eb9-b24f-157496db7fd9]: (4, ('# BEGIN RECORD 6.2.1.11\nnode.name = iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639\nnode.tpgt = -1\nnode.startup = automatic\nnode.leading_login = No\niface.iscsi_ifacename = default\niface.net_ifacename = \niface.ipaddress = \niface.prefix_len = 0\niface.hwaddress = \niface.transport_name = tcp\niface.initiatorname = \niface.state = \niface.vlan_id = 0\niface.vlan_priority = 0\niface.vlan_state = \niface.iface_num = 0\niface.mtu = 0\niface.port = 0\niface.bootproto = \niface.subnet_mask = \niface.gateway = \niface.dhcp_alt_client_id_state = \niface.dhcp_alt_client_id = \niface.dhcp_dns = \niface.dhcp_learn_iqn = \niface.dhcp_req_vendor_id_state = \niface.dhcp_vendor_id_state = \niface.dhcp_vendor_id = \niface.dhcp_slp_da = \niface.fragmentation = \niface.gratuitous_arp = \niface.incoming_forwarding = \niface.tos_state = \niface.tos = 0\niface.ttl = 0\niface.delayed_ack = \niface.tcp_nagle = \niface.tcp_wsf_state = \niface.tcp_wsf = 0\niface.tcp_timer_scale = 0\niface.tcp_timestamp = \niface.redirect = \niface.def_task_mgmt_timeout = 0\niface.header_digest = \niface.data_digest = \niface.immediate_data = \niface.initial_r2t = \niface.data_seq_inorder = \niface.data_pdu_inorder = \niface.erl = 0\niface.max_receive_data_len = 0\niface.first_burst_len = 0\niface.max_outstanding_r2t = 0\niface.max_burst_len = 0\niface.chap_auth = \niface.bidi_chap = \niface.strict_login_compliance = \niface.discovery_auth = \niface.discovery_logout = \nnode.discovery_address = \nnode.discovery_port = 0\nnode.discovery_type = fw\nnode.session.initial_cmdsn = 0\nnode.session.initial_login_retry_max = 8\nnode.session.xmit_thread_priority = 0\nnode.session.cmds_max = 128\nnode.session.queue_depth = 32\nnode.session.nr_sessions = 1\nnode.session.auth.authmethod = None\nnode.session.auth.username = \nnode.session.auth.password = \nnode.session.auth.username_in = \nnode.session.auth.password_in = \nnode.session.auth.chap_algs = SHA3-256,SHA256\nnode.session.timeo.replacement_timeout = 120\nnode.session.err_timeo.abort_timeout = 15\nnode.session.err_timeo.lu_reset_timeout = 30\nnode.session.err_timeo.tgt_reset_timeout = 30\nnode.session.err_timeo.host_reset_timeout = 60\nnode.session.iscsi.FastAbort = Yes\nnode.session.iscsi.InitialR2T = No\nnode.session.iscsi.ImmediateData = Yes\nnode.session.iscsi.FirstBurstLength = 262144\nnode.session.iscsi.MaxBurstLength = 16776192\nnode.session.iscsi.DefaultTime2Retain = 0\nnode.session.iscsi.DefaultTime2Wait = 2\nnode.session.iscsi.MaxConnections = 1\nnode.session.iscsi.MaxOutstandingR2T = 1\nnode.session.iscsi.ERL = 0\nnode.session.scan = auto\nnode.session.reopen_max = 0\nnode.session.conn_reopen_log_freq = 1\nnode.conn[0].address = 38.102.83.12\nnode.conn[0].port = 3260\nnode.conn[0].startup = manual\nnode.conn[0].tcp.window_size = 524288\nnode.conn[0].tcp.type_of_service = 0\nnode.conn[0].timeo.logout_timeout = 15\nnode.conn[0].timeo.login_timeout = 15\nnode.conn[0].timeo.auth_timeout = 45\nnode.conn[0].timeo.noop_out_interval = 5\nnode.conn[0].timeo.noop_out_timeout = 5\nnode.conn[0].iscsi.MaxXmitDataSegmentLength = 0\nnode.conn[0].iscsi.MaxRecvDataSegmentLength = 262144\nnode.conn[0].iscsi.HeaderDigest = None\nnode.conn[0].iscsi.DataDigest = None\nnode.conn[0].iscsi.IFMarker = No\nnode.conn[0].iscsi.OFMarker = No\n# END RECORD\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] iscsiadm (): stdout=# BEGIN RECORD 6.2.1.11 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.name = iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.tpgt = -1 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.startup = automatic Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.leading_login = No Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.iscsi_ifacename = default Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.net_ifacename = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.ipaddress = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.prefix_len = 0 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.hwaddress = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.transport_name = tcp Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.initiatorname = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.state = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.vlan_id = 0 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.vlan_priority = 0 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.vlan_state = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.iface_num = 0 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.mtu = 0 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.port = 0 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.bootproto = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.subnet_mask = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.gateway = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_alt_client_id_state = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_alt_client_id = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_dns = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_learn_iqn = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_req_vendor_id_state = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_vendor_id_state = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_vendor_id = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_slp_da = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.fragmentation = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.gratuitous_arp = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.incoming_forwarding = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.tos_state = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.tos = 0 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.ttl = 0 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.delayed_ack = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.tcp_nagle = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.tcp_wsf_state = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.tcp_wsf = 0 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.tcp_timer_scale = 0 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.tcp_timestamp = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.redirect = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.def_task_mgmt_timeout = 0 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.header_digest = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.data_digest = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.immediate_data = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.initial_r2t = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.data_seq_inorder = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.data_pdu_inorder = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.erl = 0 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.max_receive_data_len = 0 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.first_burst_len = 0 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.max_outstanding_r2t = 0 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.max_burst_len = 0 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.chap_auth = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.bidi_chap = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.strict_login_compliance = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.discovery_auth = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: iface.discovery_logout = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.discovery_address = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.discovery_port = 0 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.discovery_type = fw Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.initial_cmdsn = 0 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.initial_login_retry_max = 8 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.xmit_thread_priority = 0 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.cmds_max = 128 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.queue_depth = 32 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.nr_sessions = 1 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.auth.authmethod = None Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.auth.username = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.auth.password = *** Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.auth.username_in = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.auth.password_in = Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.auth.chap_algs = SHA3-256,SHA256 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.timeo.replacement_timeout = 120 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.abort_timeout = 15 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.lu_reset_timeout = 30 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.tgt_reset_timeout = 30 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.host_reset_timeout = 60 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.FastAbort = Yes Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.InitialR2T = No Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.ImmediateData = Yes Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.FirstBurstLength = 262144 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxBurstLength = 16776192 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.DefaultTime2Retain = 0 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.DefaultTime2Wait = 2 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxConnections = 1 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxOutstandingR2T = 1 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.ERL = 0 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.scan = auto Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.reopen_max = 0 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.session.conn_reopen_log_freq = 1 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.conn[0].address = 38.102.83.12 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.conn[0].port = 3260 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.conn[0].startup = manual Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.conn[0].tcp.window_size = 524288 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.conn[0].tcp.type_of_service = 0 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.logout_timeout = 15 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.login_timeout = 15 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.auth_timeout = 45 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.noop_out_interval = 5 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.noop_out_timeout = 5 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.MaxXmitDataSegmentLength = 0 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144 Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.HeaderDigest = None Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.DataDigest = None Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.IFMarker = No Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.OFMarker = No Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: # END RECORD Nov 24 14:38:16.173735 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:16.179019 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 -p 38.102.83.12:3260 --op update -n node.session.scan -v manual {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:16.188494 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 -p 38.102.83.12:3260 --op update -n node.session.scan -v manual" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:16.188800 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[3169a8f1-0aa4-46a5-94e9-413ebe51df34]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:16.189183 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] iscsiadm ('--op', 'update', '-n', 'node.session.scan', '-v', 'manual'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:16.190597 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 -p 38.102.83.12:3260 --op update -n node.session.auth.authmethod -v CHAP {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:16.201631 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 -p 38.102.83.12:3260 --op update -n node.session.auth.authmethod -v CHAP" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:16.202087 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[18745e0f-22b8-4606-8d35-90ef0a46a57f]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:16.202463 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.authmethod', '-v', 'CHAP'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:16.205213 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 -p 38.102.83.12:3260 --op update -n node.session.auth.username -v 9XkGSPtjXGatiZ24v5gy {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:16.213324 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 -p 38.102.83.12:3260 --op update -n node.session.auth.username -v 9XkGSPtjXGatiZ24v5gy" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:16.213549 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[a4334328-4960-4648-a2cc-4c9f49ad4246]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:16.214018 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.username', '-v', '9XkGSPtjXGatiZ24v5gy'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:16.215605 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 -p 38.102.83.12:3260 --op update -n node.session.auth.password -v *** {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:16.225495 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 -p 38.102.83.12:3260 --op update -n node.session.auth.password -v ***" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:16.225979 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[165561a5-c574-4c33-bd55-7cb10f2f1677]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:16.226498 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.password', '-v', '***'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:16.228588 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:16.239822 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:16.240967 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[ebab8cdf-9c8e-4702-8e51-b2c4f290a3bf]: (4, ('tcp: [19] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-5410a40c-b38e-41f4-907e-6d228834ea7d (non-flash)\ntcp: [20] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:16.241275 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] iscsiadm ('-m', 'session'): stdout=tcp: [19] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-5410a40c-b38e-41f4-907e-6d228834ea7d (non-flash) Nov 24 14:38:16.241275 np0005533992.novalocal cinder-backup[122429]: tcp: [20] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d (non-flash) Nov 24 14:38:16.241275 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:38:16.241529 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] iscsi session list stdout=tcp: [19] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-5410a40c-b38e-41f4-907e-6d228834ea7d (non-flash) Nov 24 14:38:16.241529 np0005533992.novalocal cinder-backup[122429]: tcp: [20] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d (non-flash) Nov 24 14:38:16.241529 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:38:16.242450 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 -p 38.102.83.12:3260 --login {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:16.244772 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:16.244772 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:16.263586 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 -p 38.102.83.12:3260 --login" returned: 0 in 0.021s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:16.263808 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[486dfb8f-5962-417e-8ed3-1c77748d54fc]: (4, ('Login to [iface: default, target: iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639, portal: 38.102.83.12,3260] successful.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:16.264497 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] iscsiadm ('--login',): stdout=Login to [iface: default, target: iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639, portal: 38.102.83.12,3260] successful. Nov 24 14:38:16.264497 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:16.265263 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 -p 38.102.83.12:3260 --op update -n node.startup -v automatic {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:16.275543 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 -p 38.102.83.12:3260 --op update -n node.startup -v automatic" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:16.277215 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[3079b824-ee85-46b7-9f44-95c026f086e1]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:16.277723 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:16.278711 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:16.291210 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:16.291471 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[958f455a-d4ad-4738-beeb-3ec96d8c3ca8]: (4, ('tcp: [19] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-5410a40c-b38e-41f4-907e-6d228834ea7d (non-flash)\ntcp: [20] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d (non-flash)\ntcp: [21] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:16.291959 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] iscsiadm ('-m', 'session'): stdout=tcp: [19] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-5410a40c-b38e-41f4-907e-6d228834ea7d (non-flash) Nov 24 14:38:16.291959 np0005533992.novalocal cinder-backup[122429]: tcp: [20] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d (non-flash) Nov 24 14:38:16.291959 np0005533992.novalocal cinder-backup[122429]: tcp: [21] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 (non-flash) Nov 24 14:38:16.291959 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:38:16.292215 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] iscsi session list stdout=tcp: [19] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-5410a40c-b38e-41f4-907e-6d228834ea7d (non-flash) Nov 24 14:38:16.292215 np0005533992.novalocal cinder-backup[122429]: tcp: [20] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d (non-flash) Nov 24 14:38:16.292215 np0005533992.novalocal cinder-backup[122429]: tcp: [21] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 (non-flash) Nov 24 14:38:16.292215 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:38:16.292755 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: held 1.163s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:38:16.293047 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Connected to 38.102.83.12:3260 {{(pid=122429) _connect_vol /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:659}} Nov 24 14:38:16.294019 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] HCTL ('4', '-', '-', 0) found on session 21 with lun 0 {{(pid=122429) get_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:788}} Nov 24 14:38:16.294227 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Scanning host 4 c: -, t: -, l: 0) {{(pid=122429) scan_iscsi /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:814}} Nov 24 14:38:16.295321 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/class/scsi_host/host4/scan {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:16.319113 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/class/scsi_host/host4/scan" returned: 0 in 0.022s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:16.319113 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[49911e01-e397-4c54-a843-d70c35665abb]: (4, ('- - 0', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:16.319891 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Searching for a device in session 21 and hctl ['4', '*', '*', 0] yield: None {{(pid=122429) device_name_by_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:808}} Nov 24 14:38:16.385915 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00019: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:16.386656 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:16.548447 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:16.672907 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:16.673411 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:16.819145 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00020: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:16.819432 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:16.931910 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:17.026802 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Compressed 25165824 bytes of data to 24470 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:17.026802 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:17.118291 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00021: 77377273b0a4b61febdbf7bbf52b9db9 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:17.118466 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:17.122676 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Getting metadata type 'volume-base-metadata' {{(pid=122429) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Nov 24 14:38:17.127859 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=122429) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Nov 24 14:38:17.127998 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Getting metadata type 'volume-metadata' {{(pid=122429) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Nov 24 14:38:17.132752 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] No metadata type 'volume-metadata' available {{(pid=122429) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Nov 24 14:38:17.132879 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Getting metadata type 'volume-glance-metadata' {{(pid=122429) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Nov 24 14:38:17.136722 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] No metadata type 'volume-glance-metadata' available {{(pid=122429) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Nov 24 14:38:17.136972 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] _write_sha256file started, container name: tempest-volumesbackupstest-backup-container-241182277, sha256file filename: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74_sha256file. {{(pid=122429) _write_sha256file /opt/stack/cinder/cinder/backup/chunkeddriver.py:302}} Nov 24 14:38:17.214417 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] _write_sha256file finished. {{(pid=122429) _write_sha256file /opt/stack/cinder/cinder/backup/chunkeddriver.py:318}} Nov 24 14:38:17.214697 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] _write_metadata started, container name: tempest-volumesbackupstest-backup-container-241182277, metadata filename: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74_metadata. {{(pid=122429) _write_metadata /opt/stack/cinder/cinder/backup/chunkeddriver.py:279}} Nov 24 14:38:17.248381 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] _write_metadata finished. Metadata: b'{\n "backup_description": null,\n "backup_id": "053a65ac-3a09-4a4b-80b8-5a9d4de11a74",\n "backup_name": "tempest-VolumesBackupsTest-Backup-1557939192",\n "created_at": "2025-11-24 19:37:58+00:00",\n "objects": [\n {\n "volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00001": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 0\n }\n },\n {\n "volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00002": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 52428800\n }\n },\n {\n "volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00003": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 104857600\n }\n },\n {\n "volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00004": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 157286400\n }\n },\n {\n "volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00005": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 209715200\n }\n },\n {\n "volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00006": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 262144000\n }\n },\n {\n "volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00007": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 314572800\n }\n },\n {\n "volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00008": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 367001600\n }\n },\n {\n "volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00009": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 419430400\n }\n },\n {\n "volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00010": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 471859200\n }\n },\n {\n "volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00011": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 524288000\n }\n },\n {\n "volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00012": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 576716800\n }\n },\n {\n "volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00013": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 629145600\n }\n },\n {\n "volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00014": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 681574400\n }\n },\n {\n "volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00015": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 734003200\n }\n },\n {\n "volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00016": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 786432000\n }\n },\n {\n "volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00017": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 838860800\n }\n },\n {\n "volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00018": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 891289600\n }\n },\n {\n "volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00019": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 943718400\n }\n },\n {\n "volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00020": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 996147200\n }\n },\n {\n "volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00021": {\n "compression": "zlib",\n "length": 25165824,\n "md5": "77377273b0a4b61febdbf7bbf52b9db9",\n "offset": 1048576000\n }\n }\n ],\n "parent_id": null,\n "version": "1.0.0",\n "volume_id": "5410a40c-b38e-41f4-907e-6d228834ea7d",\n "volume_meta": "{\\"version\\": 2, \\"volume-base-metadata\\": {\\"id\\": \\"5410a40c-b38e-41f4-907e-6d228834ea7d\\", \\"_name_id\\": null, \\"use_quota\\": true, \\"ec2_id\\": null, \\"user_id\\": \\"0e48df3b35f84ee88136fd47ee531c1f\\", \\"project_id\\": \\"c642367dafaa45dd89634e7ec9822ef2\\", \\"snapshot_id\\": null, \\"cluster_name\\": null, \\"host\\": \\"np0005533992.novalocal@lvmdriver-1#lvmdriver-1\\", \\"size\\": 1, \\"availability_zone\\": \\"nova\\", \\"status\\": \\"backing-up\\", \\"attach_status\\": \\"attached\\", \\"migration_status\\": null, \\"scheduled_at\\": \\"2025-11-24T19:37:07.000000\\", \\"launched_at\\": \\"2025-11-24T19:37:08.000000\\", \\"terminated_at\\": null, \\"display_name\\": \\"tempest-VolumesBackupsTest-Volume-41173554\\", \\"display_description\\": null, \\"provider_location\\": \\"38.102.83.12:3260,iqn.2010-10.org.openstack:volume-5410a40c-b38e-41f4-907e-6d228834ea7d iqn.2010-10.org.openstack:volume-5410a40c-b38e-41f4-907e-6d228834ea7d 0\\", \\"provider_auth\\": \\"CHAP hzeCNTAuKgVTnpGooZD4 j8ScTS2wooDfj9V3\\", \\"provider_geometry\\": null, \\"provider_id\\": null, \\"volume_type_id\\": \\"1f8776b6-c527-4643-876a-506285adfc66\\", \\"source_volid\\": null, \\"encryption_key_id\\": null, \\"consistencygroup_id\\": null, \\"group_id\\": null, \\"bootable\\": false, \\"multiattach\\": false, \\"replication_status\\": null, \\"replication_extended_status\\": null, \\"replication_driver_data\\": null, \\"previous_status\\": \\"in-use\\", \\"service_uuid\\": \\"6d0941fd-19ec-48ae-82b1-f25526175227\\", \\"shared_targets\\": false, \\"deleted_at\\": null, \\"deleted\\": false, \\"created_at\\": \\"2025-11-24T19:37:07.000000\\", \\"updated_at\\": \\"2025-11-24T19:37:58.000000\\"}}"\n}'. {{(pid=122429) _write_metadata /opt/stack/cinder/cinder/backup/chunkeddriver.py:298}} Nov 24 14:38:17.255048 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] backup 053a65ac-3a09-4a4b-80b8-5a9d4de11a74 finished. {{(pid=122429) _finalize_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:465}} Nov 24 14:38:17.358071 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sdb {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:17.362248 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Searching for a device in session 21 and hctl ['4', '*', '*', 0] yield: sdc {{(pid=122429) device_name_by_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:808}} Nov 24 14:38:17.362794 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Connected to sdc using {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639', 'target_lun': 0, 'volume_id': '8d10e5cf-1417-4901-8774-045390352639', 'auth_method': 'CHAP', 'auth_username': '9XkGSPtjXGatiZ24v5gy', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False} {{(pid=122429) _connect_vol /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:687}} Nov 24 14:38:17.363530 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Lock "connect_volume" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" :: held 2.246s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:38:17.363744 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] <== connect_volume: return (2247ms) {'type': 'block', 'scsi_wwn': '36001405dc70b46620f4471a92670c1a0', 'path': '/dev/sdc'} {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:38:17.364174 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sdc {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:17.381237 np0005533992.novalocal sudo[160104]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sdb Nov 24 14:38:17.390627 np0005533992.novalocal sudo[160105]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sdc Nov 24 14:38:17.707967 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sdb" returned: 0 in 0.350s {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:17.709698 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] ==> disconnect_volume: call "{'args': (, {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d', 'target_lun': 0, 'volume_id': 'e265de99-6228-4929-987d-8aca0e16408d', 'auth_method': 'CHAP', 'auth_username': 'HnDVF5qbNQAZ7m366TEn', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False, 'enforce_multipath': False}, {'type': 'block', 'scsi_wwn': '3600140546e7cde2639c4b5a85a22383a', 'path': '/dev/sdb'}), 'kwargs': {'force': True}}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:38:17.710028 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Acquiring lock "connect_volume" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:38:17.710768 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:38:17.712242 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sdc" returned: 0 in 0.348s {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:17.716261 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m discoverydb -o show -P 1 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:17.725634 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] _create_container started, container: tempest-volumesbackupsv39test-backup-container-225728091,backup: b805dbfc-103d-4564-a6c5-40e433409863. {{(pid=122429) _create_container /opt/stack/cinder/cinder/backup/chunkeddriver.py:252}} Nov 24 14:38:17.727216 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m discoverydb -o show -P 1" returned: 0 in 0.013s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:17.727331 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[47c48c5a-bdaa-4044-aab9-835724715b7f]: (4, ('SENDTARGETS:\nNo targets found.\niSNS:\nNo targets found.\nSTATIC:\nTarget: iqn.2010-10.org.openstack:volume-5410a40c-b38e-41f4-907e-6d228834ea7d\n\tPortal: 38.102.83.12:3260,-1\n\t\tIface Name: default\nTarget: iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d\n\tPortal: 38.102.83.12:3260,-1\n\t\tIface Name: default\nTarget: iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639\n\tPortal: 38.102.83.12:3260,-1\n\t\tIface Name: default\nFIRMWARE:\nNo targets found.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:17.729629 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] iscsiadm ['-m', 'discoverydb', '-o', 'show', '-P', 1]: stdout=SENDTARGETS: Nov 24 14:38:17.729629 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:38:17.729629 np0005533992.novalocal cinder-backup[122429]: iSNS: Nov 24 14:38:17.729629 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:38:17.729629 np0005533992.novalocal cinder-backup[122429]: STATIC: Nov 24 14:38:17.729629 np0005533992.novalocal cinder-backup[122429]: Target: iqn.2010-10.org.openstack:volume-5410a40c-b38e-41f4-907e-6d228834ea7d Nov 24 14:38:17.729629 np0005533992.novalocal cinder-backup[122429]: Portal: 38.102.83.12:3260,-1 Nov 24 14:38:17.729629 np0005533992.novalocal cinder-backup[122429]: Iface Name: default Nov 24 14:38:17.729629 np0005533992.novalocal cinder-backup[122429]: Target: iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d Nov 24 14:38:17.729629 np0005533992.novalocal cinder-backup[122429]: Portal: 38.102.83.12:3260,-1 Nov 24 14:38:17.729629 np0005533992.novalocal cinder-backup[122429]: Iface Name: default Nov 24 14:38:17.729629 np0005533992.novalocal cinder-backup[122429]: Target: iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 Nov 24 14:38:17.729629 np0005533992.novalocal cinder-backup[122429]: Portal: 38.102.83.12:3260,-1 Nov 24 14:38:17.729629 np0005533992.novalocal cinder-backup[122429]: Iface Name: default Nov 24 14:38:17.729629 np0005533992.novalocal cinder-backup[122429]: FIRMWARE: Nov 24 14:38:17.729629 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:38:17.729629 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:38:17.729629 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Regex to get portals from discoverydb: ^SENDTARGETS: Nov 24 14:38:17.729629 np0005533992.novalocal cinder-backup[122429]: .*?^DiscoveryAddress: 38.102.83.12,3260.*? Nov 24 14:38:17.729629 np0005533992.novalocal cinder-backup[122429]: (.*?)^(?:DiscoveryAddress|iSNS):.* {{(pid=122429) _get_discoverydb_portals /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:392}} Nov 24 14:38:17.729629 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Getting connected devices for (ips,iqns,luns)=[('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d', 0)] {{(pid=122429) _get_connection_devices /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:855}} Nov 24 14:38:17.730386 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:17.740838 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:17.741091 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[aae56615-0279-4a68-a565-735a2fb2b2d8]: (4, ('38.102.83.12:3260,4294967295 iqn.2010-10.org.openstack:volume-5410a40c-b38e-41f4-907e-6d228834ea7d\n38.102.83.12:3260,4294967295 iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639\n38.102.83.12:3260,4294967295 iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:17.742069 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:17.756737 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.014s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:17.756929 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[94087311-2abc-4c3e-9446-5ce8f14136b1]: (4, ('tcp: [19] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-5410a40c-b38e-41f4-907e-6d228834ea7d (non-flash)\ntcp: [20] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d (non-flash)\ntcp: [21] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:17.757318 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('-m', 'session'): stdout=tcp: [19] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-5410a40c-b38e-41f4-907e-6d228834ea7d (non-flash) Nov 24 14:38:17.757318 np0005533992.novalocal cinder-backup[122429]: tcp: [20] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d (non-flash) Nov 24 14:38:17.757318 np0005533992.novalocal cinder-backup[122429]: tcp: [21] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 (non-flash) Nov 24 14:38:17.757318 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:38:17.757517 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] iscsi session list stdout=tcp: [19] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-5410a40c-b38e-41f4-907e-6d228834ea7d (non-flash) Nov 24 14:38:17.757517 np0005533992.novalocal cinder-backup[122429]: tcp: [20] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d (non-flash) Nov 24 14:38:17.757517 np0005533992.novalocal cinder-backup[122429]: tcp: [21] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 (non-flash) Nov 24 14:38:17.757517 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:38:17.758764 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Resulting device map defaultdict(. at 0x7f5b726a6700>, {('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d'): ({'sdb'}, set())}) {{(pid=122429) _get_connection_devices /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:888}} Nov 24 14:38:17.759009 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Removing single pathed devices sdb {{(pid=122429) remove_connection /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:381}} Nov 24 14:38:17.759812 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd show status {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:17.767789 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] 'multipathd show status' failed. Not Retrying. {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:38:17.769233 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[9e281bcf-146c-42c2-b230-a0b040b950d4]: [Errno 2] No such file or directory: 'multipathd' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:38:17.769233 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:38:17.769233 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:38:17.769233 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:38:17.769233 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:17.769233 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:38:17.769233 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:38:17.769233 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:17.769233 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root Nov 24 14:38:17.769233 np0005533992.novalocal cinder-backup[122429]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Nov 24 14:38:17.769233 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:17.769233 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute Nov 24 14:38:17.769233 np0005533992.novalocal cinder-backup[122429]: return putils.execute(on_execute=on_execute, Nov 24 14:38:17.769233 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:17.769233 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute Nov 24 14:38:17.769233 np0005533992.novalocal cinder-backup[122429]: obj = subprocess.Popen(cmd, Nov 24 14:38:17.769233 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:17.769233 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__ Nov 24 14:38:17.769233 np0005533992.novalocal cinder-backup[122429]: self._execute_child(args, executable, preexec_fn, close_fds, Nov 24 14:38:17.769233 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child Nov 24 14:38:17.769233 np0005533992.novalocal cinder-backup[122429]: raise child_exception_type(errno_num, err_msg, err_filename) Nov 24 14:38:17.769233 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: 'multipathd' Nov 24 14:38:17.769993 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[9e281bcf-146c-42c2-b230-a0b040b950d4]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root\n return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute\n return putils.execute(on_execute=on_execute,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute\n obj = subprocess.Popen(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__\n self._execute_child(args, executable, preexec_fn, close_fds,\n File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child\n raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: \'multipathd\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:17.770825 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Flushing IO for device /dev/sdb {{(pid=122429) flush_device_io /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:441}} Nov 24 14:38:17.771720 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blockdev --flushbufs /dev/sdb {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:17.779028 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "blockdev --flushbufs /dev/sdb" returned: 0 in 0.007s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:17.779601 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[554a5035-69ff-4412-9b6d-658a56ffdfb1]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:17.779752 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Remove SCSI device /dev/sdb with /sys/block/sdb/device/delete {{(pid=122429) remove_scsi_device /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:151}} Nov 24 14:38:17.780803 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/block/sdb/device/delete {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:17.818136 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/block/sdb/device/delete" returned: 0 in 0.037s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:17.818405 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[75c33ca9-b2d1-421a-a546-6ccd71e86872]: (4, ('1', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:17.818964 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Checking to see if SCSI volumes sdb have been removed. {{(pid=122429) wait_for_volumes_removal /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:159}} Nov 24 14:38:17.819251 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] SCSI volumes sdb have been removed. {{(pid=122429) wait_for_volumes_removal /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:169}} Nov 24 14:38:17.820192 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Disconnecting from: [('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d')] {{(pid=122429) _disconnect_connection /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1201}} Nov 24 14:38:17.821730 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d -p 38.102.83.12:3260 --op update -n node.startup -v manual {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:17.832024 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d -p 38.102.83.12:3260 --op update -n node.startup -v manual" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:17.832479 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[9d974da6-8c00-4a46-9a09-67fe99ad2307]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:17.832900 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:17.833760 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d -p 38.102.83.12:3260 --logout {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:17.857141 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d -p 38.102.83.12:3260 --logout" returned: 0 in 0.023s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:17.857273 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[4e3b3fb6-d3ec-458a-9d5c-d95eaa0d3b5a]: (4, ('Logging out of session [sid: 20, target: iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d, portal: 38.102.83.12,3260]\nLogout of [sid: 20, target: iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d, portal: 38.102.83.12,3260] successful.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:17.857540 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--logout',): stdout=Logging out of session [sid: 20, target: iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d, portal: 38.102.83.12,3260] Nov 24 14:38:17.857540 np0005533992.novalocal cinder-backup[122429]: Logout of [sid: 20, target: iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d, portal: 38.102.83.12,3260] successful. Nov 24 14:38:17.857540 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:17.858243 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d -p 38.102.83.12:3260 --op delete {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:17.871178 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e265de99-6228-4929-987d-8aca0e16408d -p 38.102.83.12:3260 --op delete" returned: 0 in 0.013s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:17.871394 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[5e067d76-ed7f-4a67-b2b9-eb080f74f132]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:17.871823 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'delete'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:17.872220 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: held 0.162s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:38:17.872395 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] <== disconnect_volume: return (162ms) None {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:38:17.924833 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] generate_object_name_prefix: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863 {{(pid=122429) _generate_object_name_prefix /opt/stack/cinder/cinder/backup/drivers/swift.py:433}} Nov 24 14:38:17.930508 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] starting backup of volume: 8d10e5cf-1417-4901-8774-045390352639, volume size: 1073741824, object names prefix volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863, availability zone: nova {{(pid=122429) _prepare_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:362}} Nov 24 14:38:18.220583 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:18.351472 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:18.351791 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:18.485705 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] backup MD5 for volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00001: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:18.485827 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:18.590931 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:18.742059 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:18.742499 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:18.909295 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] backup MD5 for volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00002: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:18.909589 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:19.074746 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:19.174779 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:19.174882 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:19.317689 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] backup MD5 for volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00003: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:19.317971 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:19.463499 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:19.585920 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:19.586179 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:19.740312 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] backup MD5 for volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00004: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:19.740411 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:19.915461 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:20.021283 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:20.021606 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:20.162519 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] backup MD5 for volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00005: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:20.162865 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:20.357214 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:20.475703 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:20.475703 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:20.625510 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] backup MD5 for volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00006: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:20.625679 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:20.800457 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:20.908817 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:20.909086 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:21.052163 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] backup MD5 for volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00007: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:21.052444 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:21.422728 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-15c92182-3e03-4988-96bf-156d90b835fa tempest-VolumesBackupsTest-1816644175 None] Create backup finished. backup: 053a65ac-3a09-4a4b-80b8-5a9d4de11a74. Nov 24 14:38:21.490477 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:21.659012 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:21.659279 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:21.830393 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] backup MD5 for volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00008: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:21.830692 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:21.998922 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:22.124381 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:22.124865 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:22.273546 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] backup MD5 for volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00009: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:22.273546 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:22.398582 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] Delete backup started, backup: 053a65ac-3a09-4a4b-80b8-5a9d4de11a74. Nov 24 14:38:22.399821 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] Using swift URL https://38.102.83.12:8080/v1/AUTH_c642367dafaa45dd89634e7ec9822ef2 {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:286}} Nov 24 14:38:22.399973 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] Connect to https://38.102.83.12:8080/v1/AUTH_ in "per_user" mode {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:287}} Nov 24 14:38:22.400138 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] delete started, backup: 053a65ac-3a09-4a4b-80b8-5a9d4de11a74, container: tempest-volumesbackupstest-backup-container-241182277, prefix: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:844}} Nov 24 14:38:22.450164 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:22.464201 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] generated object list: ['volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00001', 'volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00002', 'volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00003', 'volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00004', 'volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00005', 'volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00006', 'volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00007', 'volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00008', 'volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00009', 'volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00010', 'volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00011', 'volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00012', 'volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00013', 'volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00014', 'volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00015', 'volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00016', 'volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00017', 'volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00018', 'volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00019', 'volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00020', 'volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00021', 'volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74_metadata', 'volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74_sha256file']. {{(pid=122429) _generate_object_names /opt/stack/cinder/cinder/backup/chunkeddriver.py:263}} Nov 24 14:38:22.491374 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00001 in container: tempest-volumesbackupstest-backup-container-241182277. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:22.516577 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00002 in container: tempest-volumesbackupstest-backup-container-241182277. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:22.542928 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00003 in container: tempest-volumesbackupstest-backup-container-241182277. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:22.569638 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00004 in container: tempest-volumesbackupstest-backup-container-241182277. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:22.598298 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00005 in container: tempest-volumesbackupstest-backup-container-241182277. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:22.623542 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00006 in container: tempest-volumesbackupstest-backup-container-241182277. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:22.641236 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:22.641236 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:22.644294 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00007 in container: tempest-volumesbackupstest-backup-container-241182277. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:22.671760 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00008 in container: tempest-volumesbackupstest-backup-container-241182277. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:22.694103 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00009 in container: tempest-volumesbackupstest-backup-container-241182277. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:22.726128 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00010 in container: tempest-volumesbackupstest-backup-container-241182277. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:22.757659 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00011 in container: tempest-volumesbackupstest-backup-container-241182277. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:22.767457 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] backup MD5 for volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00010: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:22.767457 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:22.770357 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_db.sqlalchemy.engines [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=122429) _check_effective_sql_mode /usr/local/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:397}} Nov 24 14:38:22.789647 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00012 in container: tempest-volumesbackupstest-backup-container-241182277. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:22.822143 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00013 in container: tempest-volumesbackupstest-backup-container-241182277. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:22.848419 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00014 in container: tempest-volumesbackupstest-backup-container-241182277. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:22.873152 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00015 in container: tempest-volumesbackupstest-backup-container-241182277. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:22.910108 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00016 in container: tempest-volumesbackupstest-backup-container-241182277. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:22.928948 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:22.930038 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00017 in container: tempest-volumesbackupstest-backup-container-241182277. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:22.953717 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00018 in container: tempest-volumesbackupstest-backup-container-241182277. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:22.971580 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00019 in container: tempest-volumesbackupstest-backup-container-241182277. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:22.990830 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00020 in container: tempest-volumesbackupstest-backup-container-241182277. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:23.014080 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74-00021 in container: tempest-volumesbackupstest-backup-container-241182277. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:23.034531 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74_metadata in container: tempest-volumesbackupstest-backup-container-241182277. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:23.050179 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:23.050422 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:23.059034 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_5410a40c-b38e-41f4-907e-6d228834ea7d/20251124193807/az_nova_backup_053a65ac-3a09-4a4b-80b8-5a9d4de11a74_sha256file in container: tempest-volumesbackupstest-backup-container-241182277. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:23.059368 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] delete 053a65ac-3a09-4a4b-80b8-5a9d4de11a74 finished. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:870}} Nov 24 14:38:23.102868 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.quota [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] Created reservations ['87657d25-2297-4acf-b91d-1610b04b9a7b', 'c7cf53cf-2598-48ad-a097-2ee33619c5cb'] {{(pid=122429) reserve /opt/stack/cinder/cinder/quota.py:752}} Nov 24 14:38:23.130981 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-bd25de4e-5f33-4849-9503-6f81c86d78b1 tempest-VolumesBackupsTest-1816644175 None] Delete backup finished, backup 053a65ac-3a09-4a4b-80b8-5a9d4de11a74 deleted. Nov 24 14:38:23.159079 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] backup MD5 for volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00011: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:23.159317 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:23.302266 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:23.398331 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:23.398649 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:23.553994 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] backup MD5 for volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00012: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:23.553994 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:23.838145 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:23.967460 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:23.967596 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:24.115251 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] backup MD5 for volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00013: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:24.115505 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:24.280483 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:24.410487 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:24.410799 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:24.534955 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] backup MD5 for volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00014: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:24.535171 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:24.707928 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:24.841512 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:24.841835 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:24.981756 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] backup MD5 for volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00015: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:24.981913 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:25.151290 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:25.244929 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:25.244929 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:25.433483 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] backup MD5 for volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00016: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:25.433483 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:25.629643 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:25.843587 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:25.843587 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:26.001909 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] backup MD5 for volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00017: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:26.002077 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:26.649233 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:26.811343 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:26.811655 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:26.961057 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] backup MD5 for volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00018: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:26.961317 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:27.112217 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:27.258811 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:27.258975 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:27.447679 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] backup MD5 for volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00019: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:27.447679 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:27.586751 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:27.689704 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:27.690160 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:27.836689 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] backup MD5 for volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00020: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:27.836994 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:27.944590 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:28.023653 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Compressed 25165824 bytes of data to 24470 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:28.023933 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:28.123403 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] backup MD5 for volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00021: 77377273b0a4b61febdbf7bbf52b9db9 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:28.123675 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:28.129667 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Getting metadata type 'volume-base-metadata' {{(pid=122429) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Nov 24 14:38:28.137075 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=122429) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Nov 24 14:38:28.137368 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Getting metadata type 'volume-metadata' {{(pid=122429) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Nov 24 14:38:28.142023 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] No metadata type 'volume-metadata' available {{(pid=122429) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Nov 24 14:38:28.142291 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Getting metadata type 'volume-glance-metadata' {{(pid=122429) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Nov 24 14:38:28.146141 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] No metadata type 'volume-glance-metadata' available {{(pid=122429) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Nov 24 14:38:28.146579 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] _write_sha256file started, container name: tempest-volumesbackupsv39test-backup-container-225728091, sha256file filename: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863_sha256file. {{(pid=122429) _write_sha256file /opt/stack/cinder/cinder/backup/chunkeddriver.py:302}} Nov 24 14:38:28.234664 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] _write_sha256file finished. {{(pid=122429) _write_sha256file /opt/stack/cinder/cinder/backup/chunkeddriver.py:318}} Nov 24 14:38:28.235320 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] _write_metadata started, container name: tempest-volumesbackupsv39test-backup-container-225728091, metadata filename: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863_metadata. {{(pid=122429) _write_metadata /opt/stack/cinder/cinder/backup/chunkeddriver.py:279}} Nov 24 14:38:28.277278 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] _write_metadata finished. Metadata: b'{\n "backup_description": null,\n "backup_id": "b805dbfc-103d-4564-a6c5-40e433409863",\n "backup_name": "tempest-VolumesBackupsV39Test-Backup-1325089779",\n "created_at": "2025-11-24 19:38:11+00:00",\n "objects": [\n {\n "volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00001": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 0\n }\n },\n {\n "volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00002": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 52428800\n }\n },\n {\n "volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00003": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 104857600\n }\n },\n {\n "volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00004": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 157286400\n }\n },\n {\n "volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00005": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 209715200\n }\n },\n {\n "volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00006": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 262144000\n }\n },\n {\n "volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00007": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 314572800\n }\n },\n {\n "volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00008": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 367001600\n }\n },\n {\n "volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00009": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 419430400\n }\n },\n {\n "volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00010": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 471859200\n }\n },\n {\n "volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00011": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 524288000\n }\n },\n {\n "volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00012": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 576716800\n }\n },\n {\n "volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00013": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 629145600\n }\n },\n {\n "volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00014": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 681574400\n }\n },\n {\n "volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00015": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 734003200\n }\n },\n {\n "volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00016": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 786432000\n }\n },\n {\n "volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00017": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 838860800\n }\n },\n {\n "volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00018": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 891289600\n }\n },\n {\n "volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00019": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 943718400\n }\n },\n {\n "volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00020": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 996147200\n }\n },\n {\n "volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00021": {\n "compression": "zlib",\n "length": 25165824,\n "md5": "77377273b0a4b61febdbf7bbf52b9db9",\n "offset": 1048576000\n }\n }\n ],\n "parent_id": null,\n "version": "1.0.0",\n "volume_id": "8d10e5cf-1417-4901-8774-045390352639",\n "volume_meta": "{\\"version\\": 2, \\"volume-base-metadata\\": {\\"id\\": \\"8d10e5cf-1417-4901-8774-045390352639\\", \\"_name_id\\": null, \\"use_quota\\": true, \\"ec2_id\\": null, \\"user_id\\": \\"e7c9e2495e1d4506a8f41d6f3262e5ff\\", \\"project_id\\": \\"107d295432b146088ae2cfd7c6d08054\\", \\"snapshot_id\\": null, \\"cluster_name\\": null, \\"host\\": \\"np0005533992.novalocal@lvmdriver-1#lvmdriver-1\\", \\"size\\": 1, \\"availability_zone\\": \\"nova\\", \\"status\\": \\"backing-up\\", \\"attach_status\\": \\"detached\\", \\"migration_status\\": null, \\"scheduled_at\\": \\"2025-11-24T19:38:10.000000\\", \\"launched_at\\": \\"2025-11-24T19:38:10.000000\\", \\"terminated_at\\": null, \\"display_name\\": \\"tempest-VolumesBackupsV39Test-Volume-1886132356\\", \\"display_description\\": null, \\"provider_location\\": \\"38.102.83.12:3260,iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 0\\", \\"provider_auth\\": \\"CHAP 9XkGSPtjXGatiZ24v5gy br2xHdwwaqW9oSU9\\", \\"provider_geometry\\": null, \\"provider_id\\": null, \\"volume_type_id\\": \\"1f8776b6-c527-4643-876a-506285adfc66\\", \\"source_volid\\": null, \\"encryption_key_id\\": null, \\"consistencygroup_id\\": null, \\"group_id\\": null, \\"bootable\\": false, \\"multiattach\\": false, \\"replication_status\\": null, \\"replication_extended_status\\": null, \\"replication_driver_data\\": null, \\"previous_status\\": \\"available\\", \\"service_uuid\\": \\"6d0941fd-19ec-48ae-82b1-f25526175227\\", \\"shared_targets\\": false, \\"deleted_at\\": null, \\"deleted\\": false, \\"created_at\\": \\"2025-11-24T19:38:10.000000\\", \\"updated_at\\": \\"2025-11-24T19:38:13.000000\\"}}"\n}'. {{(pid=122429) _write_metadata /opt/stack/cinder/cinder/backup/chunkeddriver.py:298}} Nov 24 14:38:28.285745 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] backup b805dbfc-103d-4564-a6c5-40e433409863 finished. {{(pid=122429) _finalize_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:465}} Nov 24 14:38:28.377199 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sdc {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:28.403942 np0005533992.novalocal sudo[160304]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sdc Nov 24 14:38:28.718931 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sdc" returned: 0 in 0.341s {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:28.721649 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] ==> disconnect_volume: call "{'args': (, {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639', 'target_lun': 0, 'volume_id': '8d10e5cf-1417-4901-8774-045390352639', 'auth_method': 'CHAP', 'auth_username': '9XkGSPtjXGatiZ24v5gy', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False, 'enforce_multipath': False}, {'type': 'block', 'scsi_wwn': '36001405dc70b46620f4471a92670c1a0', 'path': '/dev/sdc'}), 'kwargs': {'force': True}}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:38:28.722178 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Acquiring lock "connect_volume" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:38:28.723435 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:38:28.724394 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m discoverydb -o show -P 1 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:28.734372 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m discoverydb -o show -P 1" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:28.734822 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[8cbe51cd-1a66-42fa-910c-d97fef5be510]: (4, ('SENDTARGETS:\nNo targets found.\niSNS:\nNo targets found.\nSTATIC:\nTarget: iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639\n\tPortal: 38.102.83.12:3260,-1\n\t\tIface Name: default\nFIRMWARE:\nNo targets found.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:28.735305 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] iscsiadm ['-m', 'discoverydb', '-o', 'show', '-P', 1]: stdout=SENDTARGETS: Nov 24 14:38:28.735305 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:38:28.735305 np0005533992.novalocal cinder-backup[122429]: iSNS: Nov 24 14:38:28.735305 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:38:28.735305 np0005533992.novalocal cinder-backup[122429]: STATIC: Nov 24 14:38:28.735305 np0005533992.novalocal cinder-backup[122429]: Target: iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 Nov 24 14:38:28.735305 np0005533992.novalocal cinder-backup[122429]: Portal: 38.102.83.12:3260,-1 Nov 24 14:38:28.735305 np0005533992.novalocal cinder-backup[122429]: Iface Name: default Nov 24 14:38:28.735305 np0005533992.novalocal cinder-backup[122429]: FIRMWARE: Nov 24 14:38:28.735305 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:38:28.735305 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:38:28.735613 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Regex to get portals from discoverydb: ^SENDTARGETS: Nov 24 14:38:28.735613 np0005533992.novalocal cinder-backup[122429]: .*?^DiscoveryAddress: 38.102.83.12,3260.*? Nov 24 14:38:28.735613 np0005533992.novalocal cinder-backup[122429]: (.*?)^(?:DiscoveryAddress|iSNS):.* {{(pid=122429) _get_discoverydb_portals /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:392}} Nov 24 14:38:28.736314 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Getting connected devices for (ips,iqns,luns)=[('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639', 0)] {{(pid=122429) _get_connection_devices /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:855}} Nov 24 14:38:28.736923 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:28.745911 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node" returned: 0 in 0.009s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:28.746087 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[fb83efe5-027d-4414-8548-8bb4154c327f]: (4, ('38.102.83.12:3260,4294967295 iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:28.746957 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:28.757461 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:28.757805 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[b3b1ce54-4d8e-406d-982a-246a9dc809e1]: (4, ('tcp: [21] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:28.758240 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] iscsiadm ('-m', 'session'): stdout=tcp: [21] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 (non-flash) Nov 24 14:38:28.758240 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:38:28.758512 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] iscsi session list stdout=tcp: [21] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 (non-flash) Nov 24 14:38:28.758512 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:38:28.759428 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Resulting device map defaultdict(. at 0x7f5b90c9a700>, {('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639'): ({'sdc'}, set())}) {{(pid=122429) _get_connection_devices /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:888}} Nov 24 14:38:28.759872 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Removing single pathed devices sdc {{(pid=122429) remove_connection /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:381}} Nov 24 14:38:28.760673 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd show status {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:28.766343 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] 'multipathd show status' failed. Not Retrying. {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:38:28.767390 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[ab4d4ae1-c2cf-4366-a748-d5fe8a90ed23]: [Errno 2] No such file or directory: 'multipathd' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:38:28.767390 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:38:28.767390 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:38:28.767390 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:38:28.767390 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:28.767390 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:38:28.767390 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:38:28.767390 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:28.767390 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root Nov 24 14:38:28.767390 np0005533992.novalocal cinder-backup[122429]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Nov 24 14:38:28.767390 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:28.767390 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute Nov 24 14:38:28.767390 np0005533992.novalocal cinder-backup[122429]: return putils.execute(on_execute=on_execute, Nov 24 14:38:28.767390 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:28.767390 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute Nov 24 14:38:28.767390 np0005533992.novalocal cinder-backup[122429]: obj = subprocess.Popen(cmd, Nov 24 14:38:28.767390 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:28.767390 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__ Nov 24 14:38:28.767390 np0005533992.novalocal cinder-backup[122429]: self._execute_child(args, executable, preexec_fn, close_fds, Nov 24 14:38:28.767390 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child Nov 24 14:38:28.767390 np0005533992.novalocal cinder-backup[122429]: raise child_exception_type(errno_num, err_msg, err_filename) Nov 24 14:38:28.767390 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: 'multipathd' Nov 24 14:38:28.768214 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[ab4d4ae1-c2cf-4366-a748-d5fe8a90ed23]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root\n return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute\n return putils.execute(on_execute=on_execute,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute\n obj = subprocess.Popen(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__\n self._execute_child(args, executable, preexec_fn, close_fds,\n File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child\n raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: \'multipathd\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:28.769020 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Flushing IO for device /dev/sdc {{(pid=122429) flush_device_io /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:441}} Nov 24 14:38:28.769886 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blockdev --flushbufs /dev/sdc {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:28.776879 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "blockdev --flushbufs /dev/sdc" returned: 0 in 0.007s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:28.777182 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[472d8a39-f899-46b4-91f6-b226bfefa1a0]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:28.777617 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Remove SCSI device /dev/sdc with /sys/block/sdc/device/delete {{(pid=122429) remove_scsi_device /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:151}} Nov 24 14:38:28.778401 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/block/sdc/device/delete {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:28.833138 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/block/sdc/device/delete" returned: 0 in 0.054s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:28.833471 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[0ce58f6d-7e41-423d-ac2f-bacb2c117b57]: (4, ('1', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:28.834063 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Checking to see if SCSI volumes sdc have been removed. {{(pid=122429) wait_for_volumes_removal /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:159}} Nov 24 14:38:28.834390 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] SCSI volumes sdc have been removed. {{(pid=122429) wait_for_volumes_removal /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:169}} Nov 24 14:38:28.835050 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Disconnecting from: [('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639')] {{(pid=122429) _disconnect_connection /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1201}} Nov 24 14:38:28.835861 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 -p 38.102.83.12:3260 --op update -n node.startup -v manual {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:28.848063 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 -p 38.102.83.12:3260 --op update -n node.startup -v manual" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:28.848346 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[006b27f6-ef5e-4e93-9ff0-b67bdcef893e]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:28.848815 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:28.849653 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 -p 38.102.83.12:3260 --logout {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:28.877596 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 -p 38.102.83.12:3260 --logout" returned: 0 in 0.028s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:28.878237 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[58b012dd-4bf1-4f39-a599-72983d049c7a]: (4, ('Logging out of session [sid: 21, target: iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639, portal: 38.102.83.12,3260]\nLogout of [sid: 21, target: iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639, portal: 38.102.83.12,3260] successful.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:28.878646 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] iscsiadm ('--logout',): stdout=Logging out of session [sid: 21, target: iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639, portal: 38.102.83.12,3260] Nov 24 14:38:28.878646 np0005533992.novalocal cinder-backup[122429]: Logout of [sid: 21, target: iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639, portal: 38.102.83.12,3260] successful. Nov 24 14:38:28.878646 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:28.879608 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 -p 38.102.83.12:3260 --op delete {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:28.891034 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-8d10e5cf-1417-4901-8774-045390352639 -p 38.102.83.12:3260 --op delete" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:28.891296 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[8fb42762-6831-426c-81a5-09ccda3c6f97]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:28.891739 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] iscsiadm ('--op', 'delete'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:28.892145 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Lock "connect_volume" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: held 0.170s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:38:28.892452 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] <== disconnect_volume: return (170ms) None {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:38:32.429481 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-e8098b0b-a891-488c-aa8a-24e196a8288b tempest-VolumesBackupsV39Test-2049679509 None] Create backup finished. backup: b805dbfc-103d-4564-a6c5-40e433409863. Nov 24 14:38:33.317360 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] Delete backup started, backup: b805dbfc-103d-4564-a6c5-40e433409863. Nov 24 14:38:33.319435 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] Using swift URL https://38.102.83.12:8080/v1/AUTH_107d295432b146088ae2cfd7c6d08054 {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:286}} Nov 24 14:38:33.319736 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] Connect to https://38.102.83.12:8080/v1/AUTH_ in "per_user" mode {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:287}} Nov 24 14:38:33.320033 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] delete started, backup: b805dbfc-103d-4564-a6c5-40e433409863, container: tempest-volumesbackupsv39test-backup-container-225728091, prefix: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:844}} Nov 24 14:38:33.383444 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] generated object list: ['volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00001', 'volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00002', 'volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00003', 'volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00004', 'volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00005', 'volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00006', 'volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00007', 'volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00008', 'volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00009', 'volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00010', 'volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00011', 'volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00012', 'volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00013', 'volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00014', 'volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00015', 'volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00016', 'volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00017', 'volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00018', 'volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00019', 'volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00020', 'volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00021', 'volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863_metadata', 'volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863_sha256file']. {{(pid=122429) _generate_object_names /opt/stack/cinder/cinder/backup/chunkeddriver.py:263}} Nov 24 14:38:33.410428 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] deleted object: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00001 in container: tempest-volumesbackupsv39test-backup-container-225728091. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:33.435435 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] deleted object: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00002 in container: tempest-volumesbackupsv39test-backup-container-225728091. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:33.464658 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] deleted object: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00003 in container: tempest-volumesbackupsv39test-backup-container-225728091. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:33.491138 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] deleted object: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00004 in container: tempest-volumesbackupsv39test-backup-container-225728091. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:33.516167 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] deleted object: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00005 in container: tempest-volumesbackupsv39test-backup-container-225728091. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:33.540858 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] deleted object: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00006 in container: tempest-volumesbackupsv39test-backup-container-225728091. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:33.571707 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] deleted object: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00007 in container: tempest-volumesbackupsv39test-backup-container-225728091. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:33.598494 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] deleted object: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00008 in container: tempest-volumesbackupsv39test-backup-container-225728091. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:33.625037 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] deleted object: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00009 in container: tempest-volumesbackupsv39test-backup-container-225728091. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:33.657619 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] deleted object: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00010 in container: tempest-volumesbackupsv39test-backup-container-225728091. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:33.684069 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] deleted object: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00011 in container: tempest-volumesbackupsv39test-backup-container-225728091. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:33.712221 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] deleted object: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00012 in container: tempest-volumesbackupsv39test-backup-container-225728091. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:33.740279 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] deleted object: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00013 in container: tempest-volumesbackupsv39test-backup-container-225728091. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:33.769756 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] deleted object: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00014 in container: tempest-volumesbackupsv39test-backup-container-225728091. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:33.799398 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] deleted object: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00015 in container: tempest-volumesbackupsv39test-backup-container-225728091. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:33.831726 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] deleted object: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00016 in container: tempest-volumesbackupsv39test-backup-container-225728091. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:33.861078 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] deleted object: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00017 in container: tempest-volumesbackupsv39test-backup-container-225728091. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:33.893062 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] deleted object: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00018 in container: tempest-volumesbackupsv39test-backup-container-225728091. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:33.925582 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] deleted object: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00019 in container: tempest-volumesbackupsv39test-backup-container-225728091. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:33.957920 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] deleted object: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00020 in container: tempest-volumesbackupsv39test-backup-container-225728091. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:33.991830 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] deleted object: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863-00021 in container: tempest-volumesbackupsv39test-backup-container-225728091. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:34.019279 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] deleted object: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863_metadata in container: tempest-volumesbackupsv39test-backup-container-225728091. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:34.056289 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] deleted object: volume_8d10e5cf-1417-4901-8774-045390352639/20251124193817/az_nova_backup_b805dbfc-103d-4564-a6c5-40e433409863_sha256file in container: tempest-volumesbackupsv39test-backup-container-225728091. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:38:34.056663 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] delete b805dbfc-103d-4564-a6c5-40e433409863 finished. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:870}} Nov 24 14:38:34.062984 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_db.sqlalchemy.engines [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=122429) _check_effective_sql_mode /usr/local/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:397}} Nov 24 14:38:34.088423 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.quota [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] Created reservations ['6f500e4b-6fdb-4483-8d8a-c0b2d387c05b', '46324b8a-dbca-4f1e-96b5-a45f424c55a1'] {{(pid=122429) reserve /opt/stack/cinder/cinder/quota.py:752}} Nov 24 14:38:34.114827 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-f30c4e53-6515-4d50-bdc8-30a326ca1b94 tempest-VolumesBackupsV39Test-2049679509 None] Delete backup finished, backup b805dbfc-103d-4564-a6c5-40e433409863 deleted. Nov 24 14:38:41.739094 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Create backup started, backup: f5cd5608-e8db-496c-be30-dfa768060eda volume: e6f9e8be-7196-4cc0-bea9-056b2e289daa. Nov 24 14:38:41.744085 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Call Volume Manager to get_backup_device for Backup(availability_zone='nova',container='tempest-volumesbackupstest-backup-container-1248068680',created_at=2025-11-24T19:38:41Z,data_timestamp=2025-11-24T19:38:41Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-1951522908',encryption_key_id=None,fail_reason=None,host='np0005533992.novalocal',id=f5cd5608-e8db-496c-be30-dfa768060eda,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='c642367dafaa45dd89634e7ec9822ef2',restore_volume_id=None,service='cinder.backup.drivers.swift.SwiftBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='0e48df3b35f84ee88136fd47ee531c1f',volume_id=e6f9e8be-7196-4cc0-bea9-056b2e289daa) Nov 24 14:38:41.770309 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Using swift URL https://38.102.83.12:8080/v1/AUTH_c642367dafaa45dd89634e7ec9822ef2 {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:286}} Nov 24 14:38:41.770600 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Connect to https://38.102.83.12:8080/v1/AUTH_ in "per_user" mode {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:287}} Nov 24 14:38:41.771098 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '38.102.83.12', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:38:41.772482 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:41.778855 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.006s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:41.779128 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[0bdbccf8-c630-4b47-9f21-3ce2f18243ee]: (4, ('InitiatorName=iqn.1994-05.com.redhat:5b9c7bcf3f78\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:41.781415 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[e6d0d2d4-c65d-43f6-8c7d-e20fe9c70e9c]: [Errno 2] No such file or directory: '/dev/scini' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:38:41.781415 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:38:41.781415 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:38:41.781415 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:38:41.781415 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:41.781415 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:38:41.781415 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:38:41.781415 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:41.781415 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 57, in get_guid Nov 24 14:38:41.781415 np0005533992.novalocal cinder-backup[122429]: with open_scini_device() as fd: Nov 24 14:38:41.781415 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:41.781415 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/contextlib.py", line 137, in __enter__ Nov 24 14:38:41.781415 np0005533992.novalocal cinder-backup[122429]: return next(self.gen) Nov 24 14:38:41.781415 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^ Nov 24 14:38:41.781415 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 40, in open_scini_device Nov 24 14:38:41.781415 np0005533992.novalocal cinder-backup[122429]: fd = os.open(SCINI_DEVICE_PATH, os.O_RDWR) Nov 24 14:38:41.781415 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:41.781415 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: '/dev/scini' Nov 24 14:38:41.781980 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[e6d0d2d4-c65d-43f6-8c7d-e20fe9c70e9c]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 57, in get_guid\n with open_scini_device() as fd:\n ^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/contextlib.py", line 137, in __enter__\n return next(self.gen)\n ^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 40, in open_scini_device\n fd = os.open(SCINI_DEVICE_PATH, os.O_RDWR)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory: \'/dev/scini\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:41.782302 np0005533992.novalocal cinder-backup[122429]: ERROR os_brick.initiator.connectors.scaleio [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Error querying sdc guid: [Errno 2] No such file or directory: FileNotFoundError: [Errno 2] No such file or directory Nov 24 14:38:41.782671 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.scaleio [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Unable to find SDC guid: Error querying sdc guid: [Errno 2] No such file or directory Nov 24 14:38:41.783269 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt -v / -n -o SOURCE {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:41.792144 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt -v / -n -o SOURCE" returned: 0 in 0.009s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:41.792466 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[f60298ce-0acb-4d6e-97a0-067cb7ba267e]: (4, ('/dev/vda2\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:41.793161 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/vda2 -s UUID -o value {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:41.806188 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/vda2 -s UUID -o value" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:41.806188 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[409144a3-077e-4671-ba56-513ef291e061]: (4, ('3fc68afb-d7e4-475c-9164-ea34f338d965\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:41.807857 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[02fb4a50-a56a-42f1-9aac-b588d80a207f]: (4, 'e2c36df9-c117-4156-b18b-d355565d8e8e') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:41.807857 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Running cmd (subprocess): nvme version {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:41.819609 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] 'nvme version' failed. Not Retrying. {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:38:41.820000 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.nvmeof [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] nvme not present on system {{(pid=122429) nvme_present /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/nvmeof.py:784}} Nov 24 14:38:41.821951 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=122429) find_dsc /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:132}} Nov 24 14:38:41.823031 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.lightos [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Current host hostNQN nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e and IP(s) are ['38.102.83.12', 'fe80::f816:3eff:fe17:2fa8', '192.168.122.1', '172.24.5.1', '2001:db8::2', 'fe80::64fc:4aff:fe7b:de4b'] Nov 24 14:38:41.823192 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=122429) get_connector_properties /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:109}} Nov 24 14:38:41.823522 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e dsc: {{(pid=122429) get_connector_properties /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:112}} Nov 24 14:38:41.824241 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] <== get_connector_properties: return (51ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '38.102.83.12', 'host': 'np0005533992.novalocal', 'multipath': False, 'enforce_multipath': False, 'initiator': 'iqn.1994-05.com.redhat:5b9c7bcf3f78', 'do_local_attach': False, 'uuid': '3fc68afb-d7e4-475c-9164-ea34f338d965', 'system uuid': 'e2c36df9-c117-4156-b18b-d355565d8e8e', 'nvme_native_multipath': False, 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e', 'found_dsc': '', 'host_ips': ['38.102.83.12', 'fe80::f816:3eff:fe17:2fa8', '192.168.122.1', '172.24.5.1', '2001:db8::2', 'fe80::64fc:4aff:fe7b:de4b']} {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:38:45.334260 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connector [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Factory for iscsi on None {{(pid=122429) factory /usr/local/lib/python3.12/site-packages/os_brick/initiator/connector.py:281}} Nov 24 14:38:45.335739 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] ==> connect_volume: call "{'self': , 'connection_properties': {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa', 'target_lun': 0, 'volume_id': 'e6f9e8be-7196-4cc0-bea9-056b2e289daa', 'auth_method': 'CHAP', 'auth_username': '9Ckqgeps5ikyyHL3sYQC', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False}}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:38:45.335905 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Acquiring lock "connect_volume" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:38:45.336548 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:38:45.336882 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Connection properties {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa', 'target_lun': 0, 'volume_id': 'e6f9e8be-7196-4cc0-bea9-056b2e289daa', 'auth_method': 'CHAP', 'auth_username': '9Ckqgeps5ikyyHL3sYQC', 'auth_password': '3o8otLEdFarmqEwD', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False, 'enforce_multipath': False} {{(pid=122429) check_multipath /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:133}} Nov 24 14:38:45.338482 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd show status {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:45.345079 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] 'multipathd show status' failed. Not Retrying. {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:38:45.347827 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[298dc066-c2ba-463b-9247-6a69081372ee]: [Errno 2] No such file or directory: 'multipathd' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:38:45.347827 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:38:45.347827 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:38:45.347827 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:38:45.347827 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:45.347827 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:38:45.347827 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:38:45.347827 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:45.347827 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root Nov 24 14:38:45.347827 np0005533992.novalocal cinder-backup[122429]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Nov 24 14:38:45.347827 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:45.347827 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute Nov 24 14:38:45.347827 np0005533992.novalocal cinder-backup[122429]: return putils.execute(on_execute=on_execute, Nov 24 14:38:45.347827 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:45.347827 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute Nov 24 14:38:45.347827 np0005533992.novalocal cinder-backup[122429]: obj = subprocess.Popen(cmd, Nov 24 14:38:45.347827 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:38:45.347827 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__ Nov 24 14:38:45.347827 np0005533992.novalocal cinder-backup[122429]: self._execute_child(args, executable, preexec_fn, close_fds, Nov 24 14:38:45.347827 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child Nov 24 14:38:45.347827 np0005533992.novalocal cinder-backup[122429]: raise child_exception_type(errno_num, err_msg, err_filename) Nov 24 14:38:45.347827 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: 'multipathd' Nov 24 14:38:45.351589 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[298dc066-c2ba-463b-9247-6a69081372ee]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root\n return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute\n return putils.execute(on_execute=on_execute,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute\n obj = subprocess.Popen(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__\n self._execute_child(args, executable, preexec_fn, close_fds,\n File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child\n raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: \'multipathd\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:45.352081 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Acquiring lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:38:45.352826 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:38:45.353190 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Trying to connect to iSCSI portal 38.102.83.12:3260 Nov 24 14:38:45.354363 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa -p 38.102.83.12:3260 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:45.365610 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa -p 38.102.83.12:3260" returned: 21 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:45.365866 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[1cecfaec-50f6-4b27-abea-db37fbae804d]: (4, ('', 'iscsiadm: No records found\n')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:45.366272 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] iscsiadm (): stdout= stderr=iscsiadm: No records found Nov 24 14:38:45.366272 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:45.366883 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa -p 38.102.83.12:3260 --interface default --op new {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:45.382292 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa -p 38.102.83.12:3260 --interface default --op new" returned: 0 in 0.014s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:45.382894 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[678b2d24-a5d5-4127-92dc-e13830932cdf]: (4, ('New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 38.102.83.12,3260,-1 iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa] added\n', 'iscsiadm: Config file line 381 does not have value\niscsiadm: Config file line 381 does not have value\niscsiadm: Config file line 381 does not have value\n')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:45.383391 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--interface', 'default', '--op', 'new'): stdout=New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 38.102.83.12,3260,-1 iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa] added Nov 24 14:38:45.383391 np0005533992.novalocal cinder-backup[122429]: stderr=iscsiadm: Config file line 381 does not have value Nov 24 14:38:45.383391 np0005533992.novalocal cinder-backup[122429]: iscsiadm: Config file line 381 does not have value Nov 24 14:38:45.383391 np0005533992.novalocal cinder-backup[122429]: iscsiadm: Config file line 381 does not have value Nov 24 14:38:45.383391 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:45.383722 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Retrying to connect to iSCSI portal 38.102.83.12:3260 {{(pid=122429) _connect_to_iscsi_portal_unsafe /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1133}} Nov 24 14:38:45.384408 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Finished call to 'os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe' after 0.031(s), this was the 1st time calling it. {{(pid=122429) log_it /usr/local/lib/python3.12/site-packages/tenacity/after.py:44}} Nov 24 14:38:45.384785 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Retrying os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe in 1.0 seconds as it raised BrickException: An unknown exception occurred.. {{(pid=122429) log_it /usr/local/lib/python3.12/site-packages/tenacity/before_sleep.py:65}} Nov 24 14:38:46.386096 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Trying to connect to iSCSI portal 38.102.83.12:3260 Nov 24 14:38:46.387396 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa -p 38.102.83.12:3260 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:46.398548 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa -p 38.102.83.12:3260" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:46.399148 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[edc6f8ca-7da9-45b9-9614-24b700a3f98c]: (4, ('# BEGIN RECORD 6.2.1.11\nnode.name = iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa\nnode.tpgt = -1\nnode.startup = automatic\nnode.leading_login = No\niface.iscsi_ifacename = default\niface.net_ifacename = \niface.ipaddress = \niface.prefix_len = 0\niface.hwaddress = \niface.transport_name = tcp\niface.initiatorname = \niface.state = \niface.vlan_id = 0\niface.vlan_priority = 0\niface.vlan_state = \niface.iface_num = 0\niface.mtu = 0\niface.port = 0\niface.bootproto = \niface.subnet_mask = \niface.gateway = \niface.dhcp_alt_client_id_state = \niface.dhcp_alt_client_id = \niface.dhcp_dns = \niface.dhcp_learn_iqn = \niface.dhcp_req_vendor_id_state = \niface.dhcp_vendor_id_state = \niface.dhcp_vendor_id = \niface.dhcp_slp_da = \niface.fragmentation = \niface.gratuitous_arp = \niface.incoming_forwarding = \niface.tos_state = \niface.tos = 0\niface.ttl = 0\niface.delayed_ack = \niface.tcp_nagle = \niface.tcp_wsf_state = \niface.tcp_wsf = 0\niface.tcp_timer_scale = 0\niface.tcp_timestamp = \niface.redirect = \niface.def_task_mgmt_timeout = 0\niface.header_digest = \niface.data_digest = \niface.immediate_data = \niface.initial_r2t = \niface.data_seq_inorder = \niface.data_pdu_inorder = \niface.erl = 0\niface.max_receive_data_len = 0\niface.first_burst_len = 0\niface.max_outstanding_r2t = 0\niface.max_burst_len = 0\niface.chap_auth = \niface.bidi_chap = \niface.strict_login_compliance = \niface.discovery_auth = \niface.discovery_logout = \nnode.discovery_address = \nnode.discovery_port = 0\nnode.discovery_type = fw\nnode.session.initial_cmdsn = 0\nnode.session.initial_login_retry_max = 8\nnode.session.xmit_thread_priority = 0\nnode.session.cmds_max = 128\nnode.session.queue_depth = 32\nnode.session.nr_sessions = 1\nnode.session.auth.authmethod = None\nnode.session.auth.username = \nnode.session.auth.password = \nnode.session.auth.username_in = \nnode.session.auth.password_in = \nnode.session.auth.chap_algs = SHA3-256,SHA256\nnode.session.timeo.replacement_timeout = 120\nnode.session.err_timeo.abort_timeout = 15\nnode.session.err_timeo.lu_reset_timeout = 30\nnode.session.err_timeo.tgt_reset_timeout = 30\nnode.session.err_timeo.host_reset_timeout = 60\nnode.session.iscsi.FastAbort = Yes\nnode.session.iscsi.InitialR2T = No\nnode.session.iscsi.ImmediateData = Yes\nnode.session.iscsi.FirstBurstLength = 262144\nnode.session.iscsi.MaxBurstLength = 16776192\nnode.session.iscsi.DefaultTime2Retain = 0\nnode.session.iscsi.DefaultTime2Wait = 2\nnode.session.iscsi.MaxConnections = 1\nnode.session.iscsi.MaxOutstandingR2T = 1\nnode.session.iscsi.ERL = 0\nnode.session.scan = auto\nnode.session.reopen_max = 0\nnode.session.conn_reopen_log_freq = 1\nnode.conn[0].address = 38.102.83.12\nnode.conn[0].port = 3260\nnode.conn[0].startup = manual\nnode.conn[0].tcp.window_size = 524288\nnode.conn[0].tcp.type_of_service = 0\nnode.conn[0].timeo.logout_timeout = 15\nnode.conn[0].timeo.login_timeout = 15\nnode.conn[0].timeo.auth_timeout = 45\nnode.conn[0].timeo.noop_out_interval = 5\nnode.conn[0].timeo.noop_out_timeout = 5\nnode.conn[0].iscsi.MaxXmitDataSegmentLength = 0\nnode.conn[0].iscsi.MaxRecvDataSegmentLength = 262144\nnode.conn[0].iscsi.HeaderDigest = None\nnode.conn[0].iscsi.DataDigest = None\nnode.conn[0].iscsi.IFMarker = No\nnode.conn[0].iscsi.OFMarker = No\n# END RECORD\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] iscsiadm (): stdout=# BEGIN RECORD 6.2.1.11 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.name = iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.tpgt = -1 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.startup = automatic Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.leading_login = No Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.iscsi_ifacename = default Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.net_ifacename = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.ipaddress = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.prefix_len = 0 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.hwaddress = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.transport_name = tcp Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.initiatorname = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.state = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.vlan_id = 0 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.vlan_priority = 0 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.vlan_state = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.iface_num = 0 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.mtu = 0 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.port = 0 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.bootproto = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.subnet_mask = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.gateway = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_alt_client_id_state = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_alt_client_id = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_dns = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_learn_iqn = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_req_vendor_id_state = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_vendor_id_state = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_vendor_id = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_slp_da = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.fragmentation = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.gratuitous_arp = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.incoming_forwarding = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.tos_state = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.tos = 0 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.ttl = 0 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.delayed_ack = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.tcp_nagle = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.tcp_wsf_state = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.tcp_wsf = 0 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.tcp_timer_scale = 0 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.tcp_timestamp = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.redirect = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.def_task_mgmt_timeout = 0 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.header_digest = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.data_digest = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.immediate_data = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.initial_r2t = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.data_seq_inorder = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.data_pdu_inorder = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.erl = 0 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.max_receive_data_len = 0 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.first_burst_len = 0 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.max_outstanding_r2t = 0 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.max_burst_len = 0 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.chap_auth = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.bidi_chap = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.strict_login_compliance = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.discovery_auth = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: iface.discovery_logout = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.discovery_address = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.discovery_port = 0 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.discovery_type = fw Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.initial_cmdsn = 0 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.initial_login_retry_max = 8 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.xmit_thread_priority = 0 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.cmds_max = 128 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.queue_depth = 32 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.nr_sessions = 1 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.auth.authmethod = None Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.auth.username = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.auth.password = *** Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.auth.username_in = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.auth.password_in = Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.auth.chap_algs = SHA3-256,SHA256 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.timeo.replacement_timeout = 120 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.abort_timeout = 15 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.lu_reset_timeout = 30 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.tgt_reset_timeout = 30 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.host_reset_timeout = 60 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.FastAbort = Yes Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.InitialR2T = No Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.ImmediateData = Yes Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.FirstBurstLength = 262144 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxBurstLength = 16776192 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.DefaultTime2Retain = 0 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.DefaultTime2Wait = 2 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxConnections = 1 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxOutstandingR2T = 1 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.ERL = 0 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.scan = auto Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.reopen_max = 0 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.session.conn_reopen_log_freq = 1 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.conn[0].address = 38.102.83.12 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.conn[0].port = 3260 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.conn[0].startup = manual Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.conn[0].tcp.window_size = 524288 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.conn[0].tcp.type_of_service = 0 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.logout_timeout = 15 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.login_timeout = 15 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.auth_timeout = 45 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.noop_out_interval = 5 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.noop_out_timeout = 5 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.MaxXmitDataSegmentLength = 0 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144 Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.HeaderDigest = None Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.DataDigest = None Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.IFMarker = No Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.OFMarker = No Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: # END RECORD Nov 24 14:38:46.400886 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:46.403604 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa -p 38.102.83.12:3260 --op update -n node.session.scan -v manual {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:46.413208 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa -p 38.102.83.12:3260 --op update -n node.session.scan -v manual" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:46.413527 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[ec63ea3d-1786-4efe-aabe-03c8be0ac686]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:46.414028 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.scan', '-v', 'manual'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:46.415262 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa -p 38.102.83.12:3260 --op update -n node.session.auth.authmethod -v CHAP {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:46.429335 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa -p 38.102.83.12:3260 --op update -n node.session.auth.authmethod -v CHAP" returned: 0 in 0.014s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:46.429716 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[a65741bd-ae81-47a6-aed1-8781e4d3eff0]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:46.430155 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.authmethod', '-v', 'CHAP'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:46.431243 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa -p 38.102.83.12:3260 --op update -n node.session.auth.username -v 9Ckqgeps5ikyyHL3sYQC {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:46.433320 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:38:46.433717 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:38:46.449023 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa -p 38.102.83.12:3260 --op update -n node.session.auth.username -v 9Ckqgeps5ikyyHL3sYQC" returned: 0 in 0.018s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:46.449373 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[6a057071-3baf-4e25-9b84-aca3088a35d2]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:46.449964 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.username', '-v', '9Ckqgeps5ikyyHL3sYQC'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:46.451203 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa -p 38.102.83.12:3260 --op update -n node.session.auth.password -v *** {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:46.463200 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa -p 38.102.83.12:3260 --op update -n node.session.auth.password -v ***" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:46.463538 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[65f68e88-a619-4cfd-83c4-f06aeb2fd423]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:46.464719 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.password', '-v', '***'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:46.465482 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:46.475275 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 21 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:46.475535 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[0ac9a68c-bcf4-4e69-8125-9f93a9aa9d43]: (4, ('', 'iscsiadm: No active sessions.\n')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:46.475986 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('-m', 'session'): stdout= stderr=iscsiadm: No active sessions. Nov 24 14:38:46.475986 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:38:46.476245 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] iscsi session list stdout= stderr=iscsiadm: No active sessions. Nov 24 14:38:46.476245 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:38:46.477673 np0005533992.novalocal cinder-backup[122429]: WARNING os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions. Nov 24 14:38:46.477673 np0005533992.novalocal cinder-backup[122429]: Nov 24 14:38:46.478701 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa -p 38.102.83.12:3260 --login {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:46.497202 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa -p 38.102.83.12:3260 --login" returned: 0 in 0.017s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:46.497599 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[0af9b41a-edb0-4663-984c-7e75df05b09f]: (4, ('Login to [iface: default, target: iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa, portal: 38.102.83.12,3260] successful.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:46.498438 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--login',): stdout=Login to [iface: default, target: iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa, portal: 38.102.83.12,3260] successful. Nov 24 14:38:46.498438 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:46.498888 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa -p 38.102.83.12:3260 --op update -n node.startup -v automatic {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:46.513413 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa -p 38.102.83.12:3260 --op update -n node.startup -v automatic" returned: 0 in 0.014s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:46.513749 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[791cf558-59c4-4d00-886c-5f8d26138c2b]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:46.514205 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:38:46.515132 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:46.525490 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:46.526163 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[7bd98f40-0e97-4289-bbdd-1215634c800c]: (4, ('tcp: [22] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:46.526659 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('-m', 'session'): stdout=tcp: [22] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa (non-flash) Nov 24 14:38:46.526659 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:38:46.526923 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] iscsi session list stdout=tcp: [22] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa (non-flash) Nov 24 14:38:46.526923 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:38:46.527406 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: held 1.175s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:38:46.527677 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Connected to 38.102.83.12:3260 {{(pid=122429) _connect_vol /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:659}} Nov 24 14:38:46.528337 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] HCTL ('2', '-', '-', 0) found on session 22 with lun 0 {{(pid=122429) get_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:788}} Nov 24 14:38:46.528615 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Scanning host 2 c: -, t: -, l: 0) {{(pid=122429) scan_iscsi /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:814}} Nov 24 14:38:46.530969 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/class/scsi_host/host2/scan {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:46.558506 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/class/scsi_host/host2/scan" returned: 0 in 0.027s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:46.558506 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[0f3b45b6-06bb-47fe-a226-752c81f97f93]: (4, ('- - 0', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:38:46.558506 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Searching for a device in session 22 and hctl ['2', '*', '*', 0] yield: None {{(pid=122429) device_name_by_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:808}} Nov 24 14:38:47.559887 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Searching for a device in session 22 and hctl ['2', '*', '*', 0] yield: sda {{(pid=122429) device_name_by_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:808}} Nov 24 14:38:47.561239 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Connected to sda using {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa', 'target_lun': 0, 'volume_id': 'e6f9e8be-7196-4cc0-bea9-056b2e289daa', 'auth_method': 'CHAP', 'auth_username': '9Ckqgeps5ikyyHL3sYQC', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False} {{(pid=122429) _connect_vol /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:687}} Nov 24 14:38:47.561651 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" :: held 2.225s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:38:47.562021 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] <== connect_volume: return (2226ms) {'type': 'block', 'scsi_wwn': '3600140506b1ba6d7fee4272ab9198593', 'path': '/dev/sda'} {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:38:47.562633 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sda {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:38:47.586037 np0005533992.novalocal sudo[160789]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sda Nov 24 14:38:47.880020 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sda" returned: 0 in 0.317s {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:38:47.884439 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_db.sqlalchemy.engines [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=122429) _check_effective_sql_mode /usr/local/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:397}} Nov 24 14:38:47.888243 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] _create_container started, container: tempest-volumesbackupstest-backup-container-1248068680,backup: f5cd5608-e8db-496c-be30-dfa768060eda. {{(pid=122429) _create_container /opt/stack/cinder/cinder/backup/chunkeddriver.py:252}} Nov 24 14:38:47.962979 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] generate_object_name_prefix: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda {{(pid=122429) _generate_object_name_prefix /opt/stack/cinder/cinder/backup/drivers/swift.py:433}} Nov 24 14:38:47.969488 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] starting backup of volume: e6f9e8be-7196-4cc0-bea9-056b2e289daa, volume size: 1073741824, object names prefix volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda, availability zone: nova {{(pid=122429) _prepare_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:362}} Nov 24 14:38:48.422614 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:48.945341 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 19857195 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:48.945686 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:49.236129 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00001: 82571f7121cb9475563edde615e4e4df {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:49.236355 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:49.672739 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:49.774946 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:49.775150 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:49.896883 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00002: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:49.897177 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:50.280663 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:50.445729 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 1362431 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:50.446007 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:50.608972 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00003: a35891500449df17e893f84bd88472fb {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:50.609100 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:50.729768 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:50.830338 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:50.830630 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:50.948875 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00004: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:50.948875 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:51.152364 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:51.278109 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:51.278363 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:51.467956 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00005: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:51.468356 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:51.875346 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:52.013779 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:52.013909 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:52.174146 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00006: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:52.174428 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:52.617925 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:52.758485 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:52.758485 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:52.884158 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00007: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:52.884488 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:53.473635 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:53.623426 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:53.623786 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:53.733783 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00008: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:53.733971 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:53.881517 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:54.014548 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:54.014548 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:54.119624 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00009: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:54.119624 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:54.577145 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:54.678502 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:54.678728 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:54.796787 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00010: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:54.796991 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:54.901551 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:55.015821 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:55.015821 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:55.180541 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00011: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:55.180634 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:55.323347 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:55.459545 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:55.459766 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:55.638969 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00012: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:55.639034 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:55.786509 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:55.887004 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:55.887591 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:56.003952 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00013: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:56.004238 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:56.383586 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:56.492974 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:56.492974 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:56.602963 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00014: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:56.603228 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:56.608415 np0005533992.novalocal cinder-backup[122429]: DEBUG dbcounter [-] [122429] Writing DB stats cinder:SELECT=97,cinder:UPDATE=25,cinder:INSERT=4 {{(pid=122429) stat_writer /usr/local/lib/python3.12/site-packages/dbcounter.py:115}} Nov 24 14:38:56.997013 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:57.125473 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:57.125586 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:57.272339 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00015: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:57.272633 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:57.463942 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:57.594228 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:57.594322 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:57.749345 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00016: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:57.749643 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:58.195329 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:58.305134 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:58.305279 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:58.454816 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00017: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:58.455090 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:58.590784 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:58.749449 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:58.749449 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:58.949825 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00018: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:58.950107 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:59.335105 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:59.432732 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:59.432995 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:38:59.547472 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00019: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:38:59.547791 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:38:59.728988 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:38:59.861183 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:38:59.861455 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:39:00.007483 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00020: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:39:00.007857 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:39:00.102623 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:39:00.182331 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Compressed 25165824 bytes of data to 24470 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:39:00.182665 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:39:00.292079 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00021: 77377273b0a4b61febdbf7bbf52b9db9 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:39:00.292254 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:39:00.299652 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Getting metadata type 'volume-base-metadata' {{(pid=122429) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Nov 24 14:39:00.306691 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=122429) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Nov 24 14:39:00.306994 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Getting metadata type 'volume-metadata' {{(pid=122429) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Nov 24 14:39:00.312025 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] No metadata type 'volume-metadata' available {{(pid=122429) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Nov 24 14:39:00.312300 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Getting metadata type 'volume-glance-metadata' {{(pid=122429) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Nov 24 14:39:00.316808 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=122429) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}} Nov 24 14:39:00.317275 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] _write_sha256file started, container name: tempest-volumesbackupstest-backup-container-1248068680, sha256file filename: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda_sha256file. {{(pid=122429) _write_sha256file /opt/stack/cinder/cinder/backup/chunkeddriver.py:302}} Nov 24 14:39:00.402685 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] _write_sha256file finished. {{(pid=122429) _write_sha256file /opt/stack/cinder/cinder/backup/chunkeddriver.py:318}} Nov 24 14:39:00.403117 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] _write_metadata started, container name: tempest-volumesbackupstest-backup-container-1248068680, metadata filename: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda_metadata. {{(pid=122429) _write_metadata /opt/stack/cinder/cinder/backup/chunkeddriver.py:279}} Nov 24 14:39:00.437739 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] _write_metadata finished. Metadata: b'{\n "backup_description": null,\n "backup_id": "f5cd5608-e8db-496c-be30-dfa768060eda",\n "backup_name": "tempest-VolumesBackupsTest-Backup-1951522908",\n "created_at": "2025-11-24 19:38:41+00:00",\n "objects": [\n {\n "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00001": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "82571f7121cb9475563edde615e4e4df",\n "offset": 0\n }\n },\n {\n "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00002": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 52428800\n }\n },\n {\n "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00003": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "a35891500449df17e893f84bd88472fb",\n "offset": 104857600\n }\n },\n {\n "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00004": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 157286400\n }\n },\n {\n "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00005": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 209715200\n }\n },\n {\n "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00006": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 262144000\n }\n },\n {\n "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00007": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 314572800\n }\n },\n {\n "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00008": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 367001600\n }\n },\n {\n "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00009": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 419430400\n }\n },\n {\n "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00010": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 471859200\n }\n },\n {\n "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00011": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 524288000\n }\n },\n {\n "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00012": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 576716800\n }\n },\n {\n "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00013": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 629145600\n }\n },\n {\n "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00014": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 681574400\n }\n },\n {\n "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00015": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 734003200\n }\n },\n {\n "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00016": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 786432000\n }\n },\n {\n "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00017": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 838860800\n }\n },\n {\n "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00018": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 891289600\n }\n },\n {\n "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00019": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 943718400\n }\n },\n {\n "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00020": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 996147200\n }\n },\n {\n "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00021": {\n "compression": "zlib",\n "length": 25165824,\n "md5": "77377273b0a4b61febdbf7bbf52b9db9",\n "offset": 1048576000\n }\n }\n ],\n "parent_id": null,\n "version": "1.0.0",\n "volume_id": "e6f9e8be-7196-4cc0-bea9-056b2e289daa",\n "volume_meta": "{\\"version\\": 2, \\"volume-base-metadata\\": {\\"id\\": \\"e6f9e8be-7196-4cc0-bea9-056b2e289daa\\", \\"_name_id\\": null, \\"use_quota\\": true, \\"ec2_id\\": null, \\"user_id\\": \\"0e48df3b35f84ee88136fd47ee531c1f\\", \\"project_id\\": \\"c642367dafaa45dd89634e7ec9822ef2\\", \\"snapshot_id\\": null, \\"cluster_name\\": null, \\"host\\": \\"np0005533992.novalocal@lvmdriver-1#lvmdriver-1\\", \\"size\\": 1, \\"availability_zone\\": \\"nova\\", \\"status\\": \\"backing-up\\", \\"attach_status\\": \\"detached\\", \\"migration_status\\": null, \\"scheduled_at\\": \\"2025-11-24T19:38:39.000000\\", \\"launched_at\\": \\"2025-11-24T19:38:41.000000\\", \\"terminated_at\\": null, \\"display_name\\": \\"tempest-VolumesBackupsTest-Volume-1627712353\\", \\"display_description\\": null, \\"provider_location\\": \\"38.102.83.12:3260,iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa 0\\", \\"provider_auth\\": \\"CHAP 9Ckqgeps5ikyyHL3sYQC 3o8otLEdFarmqEwD\\", \\"provider_geometry\\": null, \\"provider_id\\": null, \\"volume_type_id\\": \\"1f8776b6-c527-4643-876a-506285adfc66\\", \\"source_volid\\": null, \\"encryption_key_id\\": null, \\"consistencygroup_id\\": null, \\"group_id\\": null, \\"bootable\\": true, \\"multiattach\\": false, \\"replication_status\\": null, \\"replication_extended_status\\": null, \\"replication_driver_data\\": null, \\"previous_status\\": \\"available\\", \\"service_uuid\\": \\"6d0941fd-19ec-48ae-82b1-f25526175227\\", \\"shared_targets\\": false, \\"deleted_at\\": null, \\"deleted\\": false, \\"created_at\\": \\"2025-11-24T19:38:39.000000\\", \\"updated_at\\": \\"2025-11-24T19:38:43.000000\\"}, \\"volume-glance-metadata\\": {\\"hw_rng_model\\": \\"virtio\\", \\"owner_specified.openstack.md5\\": \\"\\", \\"owner_specified.openstack.sha256\\": \\"\\", \\"owner_specified.openstack.object\\": \\"images/cirros-0.6.3-x86_64-disk\\", \\"os_glance_importing_to_stores\\": \\"\\", \\"os_glance_failed_import\\": \\"\\", \\"image_id\\": \\"bfe30d4b-f478-4b3d-b058-80ea56840f0b\\", \\"image_name\\": \\"cirros-0.6.3-x86_64-disk\\", \\"checksum\\": \\"87617e24a5e30cb3b87fda8c0764838f\\", \\"container_format\\": \\"bare\\", \\"disk_format\\": \\"qcow2\\", \\"min_disk\\": \\"0\\", \\"min_ram\\": \\"0\\", \\"size\\": \\"21692416\\"}}"\n}'. {{(pid=122429) _write_metadata /opt/stack/cinder/cinder/backup/chunkeddriver.py:298}} Nov 24 14:39:00.442687 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] backup f5cd5608-e8db-496c-be30-dfa768060eda finished. {{(pid=122429) _finalize_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:465}} Nov 24 14:39:00.537823 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sda {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:00.565324 np0005533992.novalocal sudo[160979]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sda Nov 24 14:39:00.931237 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sda" returned: 0 in 0.395s {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:00.934892 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] ==> disconnect_volume: call "{'args': (, {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa', 'target_lun': 0, 'volume_id': 'e6f9e8be-7196-4cc0-bea9-056b2e289daa', 'auth_method': 'CHAP', 'auth_username': '9Ckqgeps5ikyyHL3sYQC', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False, 'enforce_multipath': False}, {'type': 'block', 'scsi_wwn': '3600140506b1ba6d7fee4272ab9198593', 'path': '/dev/sda'}), 'kwargs': {'force': True}}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:39:00.935271 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Acquiring lock "connect_volume" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:39:00.935465 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: waited 0.000s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:39:00.936346 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m discoverydb -o show -P 1 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:00.944243 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m discoverydb -o show -P 1" returned: 0 in 0.008s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:00.944372 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[e43168a3-a2a2-4e4d-a536-dd2b9dbfbb25]: (4, ('SENDTARGETS:\nNo targets found.\niSNS:\nNo targets found.\nSTATIC:\nTarget: iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa\n\tPortal: 38.102.83.12:3260,-1\n\t\tIface Name: default\nFIRMWARE:\nNo targets found.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:00.944710 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ['-m', 'discoverydb', '-o', 'show', '-P', 1]: stdout=SENDTARGETS: Nov 24 14:39:00.944710 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:39:00.944710 np0005533992.novalocal cinder-backup[122429]: iSNS: Nov 24 14:39:00.944710 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:39:00.944710 np0005533992.novalocal cinder-backup[122429]: STATIC: Nov 24 14:39:00.944710 np0005533992.novalocal cinder-backup[122429]: Target: iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa Nov 24 14:39:00.944710 np0005533992.novalocal cinder-backup[122429]: Portal: 38.102.83.12:3260,-1 Nov 24 14:39:00.944710 np0005533992.novalocal cinder-backup[122429]: Iface Name: default Nov 24 14:39:00.944710 np0005533992.novalocal cinder-backup[122429]: FIRMWARE: Nov 24 14:39:00.944710 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:39:00.944710 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:39:00.944906 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Regex to get portals from discoverydb: ^SENDTARGETS: Nov 24 14:39:00.944906 np0005533992.novalocal cinder-backup[122429]: .*?^DiscoveryAddress: 38.102.83.12,3260.*? Nov 24 14:39:00.944906 np0005533992.novalocal cinder-backup[122429]: (.*?)^(?:DiscoveryAddress|iSNS):.* {{(pid=122429) _get_discoverydb_portals /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:392}} Nov 24 14:39:00.945184 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Getting connected devices for (ips,iqns,luns)=[('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa', 0)] {{(pid=122429) _get_connection_devices /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:855}} Nov 24 14:39:00.945721 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:00.953340 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node" returned: 0 in 0.008s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:00.953469 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[d896db15-9ae5-49c2-bb68-47880fad085a]: (4, ('38.102.83.12:3260,4294967295 iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:00.954334 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:00.961323 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.007s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:00.961416 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[c4f2a06f-25ac-4b02-92cd-aa9d2afab93a]: (4, ('tcp: [22] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:00.961636 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('-m', 'session'): stdout=tcp: [22] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa (non-flash) Nov 24 14:39:00.961636 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:39:00.961851 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] iscsi session list stdout=tcp: [22] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa (non-flash) Nov 24 14:39:00.961851 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:39:00.962270 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Resulting device map defaultdict(. at 0x7f5b72910680>, {('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa'): ({'sda'}, set())}) {{(pid=122429) _get_connection_devices /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:888}} Nov 24 14:39:00.962474 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Removing single pathed devices sda {{(pid=122429) remove_connection /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:381}} Nov 24 14:39:00.963239 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd show status {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:00.967682 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] 'multipathd show status' failed. Not Retrying. {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:39:00.969082 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[75c50bab-e063-4440-9ddd-8385d67f20ec]: [Errno 2] No such file or directory: 'multipathd' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:39:00.969082 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:39:00.969082 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:39:00.969082 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:39:00.969082 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:00.969082 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:39:00.969082 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:39:00.969082 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:00.969082 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root Nov 24 14:39:00.969082 np0005533992.novalocal cinder-backup[122429]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Nov 24 14:39:00.969082 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:00.969082 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute Nov 24 14:39:00.969082 np0005533992.novalocal cinder-backup[122429]: return putils.execute(on_execute=on_execute, Nov 24 14:39:00.969082 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:00.969082 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute Nov 24 14:39:00.969082 np0005533992.novalocal cinder-backup[122429]: obj = subprocess.Popen(cmd, Nov 24 14:39:00.969082 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:00.969082 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__ Nov 24 14:39:00.969082 np0005533992.novalocal cinder-backup[122429]: self._execute_child(args, executable, preexec_fn, close_fds, Nov 24 14:39:00.969082 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child Nov 24 14:39:00.969082 np0005533992.novalocal cinder-backup[122429]: raise child_exception_type(errno_num, err_msg, err_filename) Nov 24 14:39:00.969082 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: 'multipathd' Nov 24 14:39:00.971008 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[75c50bab-e063-4440-9ddd-8385d67f20ec]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root\n return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute\n return putils.execute(on_execute=on_execute,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute\n obj = subprocess.Popen(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__\n self._execute_child(args, executable, preexec_fn, close_fds,\n File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child\n raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: \'multipathd\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:00.971296 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Flushing IO for device /dev/sda {{(pid=122429) flush_device_io /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:441}} Nov 24 14:39:00.973583 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blockdev --flushbufs /dev/sda {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:00.979255 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "blockdev --flushbufs /dev/sda" returned: 0 in 0.007s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:00.979630 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[8dbc392e-f66b-4ff8-9920-27bcb940445a]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:00.979814 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Remove SCSI device /dev/sda with /sys/block/sda/device/delete {{(pid=122429) remove_scsi_device /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:151}} Nov 24 14:39:00.980635 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/block/sda/device/delete {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:01.026315 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/block/sda/device/delete" returned: 0 in 0.046s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:01.026782 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[c2f14b9e-86a4-4316-afac-6793cc4d184d]: (4, ('1', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:01.027155 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Checking to see if SCSI volumes sda have been removed. {{(pid=122429) wait_for_volumes_removal /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:159}} Nov 24 14:39:01.027439 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] SCSI volumes sda have been removed. {{(pid=122429) wait_for_volumes_removal /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:169}} Nov 24 14:39:01.027797 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Disconnecting from: [('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa')] {{(pid=122429) _disconnect_connection /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1201}} Nov 24 14:39:01.028644 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa -p 38.102.83.12:3260 --op update -n node.startup -v manual {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:01.039363 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa -p 38.102.83.12:3260 --op update -n node.startup -v manual" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:01.039651 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[6887c346-e99e-4713-a199-b67ae5a56b1c]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:01.040062 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:01.040992 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa -p 38.102.83.12:3260 --logout {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:01.071118 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa -p 38.102.83.12:3260 --logout" returned: 0 in 0.030s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:01.071382 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[0241d99d-e20c-4315-bd29-c50778056fd2]: (4, ('Logging out of session [sid: 22, target: iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa, portal: 38.102.83.12,3260]\nLogout of [sid: 22, target: iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa, portal: 38.102.83.12,3260] successful.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:01.071877 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--logout',): stdout=Logging out of session [sid: 22, target: iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa, portal: 38.102.83.12,3260] Nov 24 14:39:01.071877 np0005533992.novalocal cinder-backup[122429]: Logout of [sid: 22, target: iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa, portal: 38.102.83.12,3260] successful. Nov 24 14:39:01.071877 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:01.072800 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa -p 38.102.83.12:3260 --op delete {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:01.082118 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa -p 38.102.83.12:3260 --op delete" returned: 0 in 0.009s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:01.082656 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[59453ea7-bcd7-4d5c-9643-c27b7c48ec66]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:01.083299 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'delete'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:01.083895 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: held 0.148s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:39:01.084353 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] <== disconnect_volume: return (149ms) None {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:39:04.617548 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-ffe75c4a-7fda-4e74-8d1a-41c5326c9603 tempest-VolumesBackupsTest-1816644175 None] Create backup finished. backup: f5cd5608-e8db-496c-be30-dfa768060eda. Nov 24 14:39:06.693290 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Restore backup started, backup: f5cd5608-e8db-496c-be30-dfa768060eda volume: 7a270e29-bb4d-46d2-8e13-8481707a413a. Nov 24 14:39:06.704368 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Using swift URL https://38.102.83.12:8080/v1/AUTH_c642367dafaa45dd89634e7ec9822ef2 {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:286}} Nov 24 14:39:06.704613 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Connect to https://38.102.83.12:8080/v1/AUTH_ in "per_user" mode {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:287}} Nov 24 14:39:06.705125 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '38.102.83.12', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:39:06.708204 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:06.716018 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:06.717038 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[d5f8d124-2497-44c9-ac49-902450fd3f1c]: (4, ('InitiatorName=iqn.1994-05.com.redhat:5b9c7bcf3f78\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:06.720647 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[1ba9f075-0d3f-48ff-ae24-4313c5f3175d]: [Errno 2] No such file or directory: '/dev/scini' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:39:06.720647 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:39:06.720647 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:39:06.720647 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:39:06.720647 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:06.720647 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:39:06.720647 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:39:06.720647 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:06.720647 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 57, in get_guid Nov 24 14:39:06.720647 np0005533992.novalocal cinder-backup[122429]: with open_scini_device() as fd: Nov 24 14:39:06.720647 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:06.720647 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/contextlib.py", line 137, in __enter__ Nov 24 14:39:06.720647 np0005533992.novalocal cinder-backup[122429]: return next(self.gen) Nov 24 14:39:06.720647 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^ Nov 24 14:39:06.720647 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 40, in open_scini_device Nov 24 14:39:06.720647 np0005533992.novalocal cinder-backup[122429]: fd = os.open(SCINI_DEVICE_PATH, os.O_RDWR) Nov 24 14:39:06.720647 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:06.720647 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: '/dev/scini' Nov 24 14:39:06.721157 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[1ba9f075-0d3f-48ff-ae24-4313c5f3175d]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 57, in get_guid\n with open_scini_device() as fd:\n ^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/contextlib.py", line 137, in __enter__\n return next(self.gen)\n ^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 40, in open_scini_device\n fd = os.open(SCINI_DEVICE_PATH, os.O_RDWR)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory: \'/dev/scini\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:06.721360 np0005533992.novalocal cinder-backup[122429]: ERROR os_brick.initiator.connectors.scaleio [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Error querying sdc guid: [Errno 2] No such file or directory: FileNotFoundError: [Errno 2] No such file or directory Nov 24 14:39:06.721908 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.scaleio [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Unable to find SDC guid: Error querying sdc guid: [Errno 2] No such file or directory Nov 24 14:39:06.724746 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt -v / -n -o SOURCE {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:06.737949 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt -v / -n -o SOURCE" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:06.740239 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[7c79ecc8-1d5b-4986-afa9-2ea5166b4949]: (4, ('/dev/vda2\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:06.741680 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/vda2 -s UUID -o value {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:06.751829 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/vda2 -s UUID -o value" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:06.752089 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[81cc27a4-c53e-4b07-8caa-4ae74911a01e]: (4, ('3fc68afb-d7e4-475c-9164-ea34f338d965\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:06.753403 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[5df127f3-c36e-46ef-9cd9-7ff92b6346c2]: (4, 'e2c36df9-c117-4156-b18b-d355565d8e8e') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:06.753991 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Running cmd (subprocess): nvme version {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:06.766529 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] 'nvme version' failed. Not Retrying. {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:39:06.766965 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.nvmeof [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] nvme not present on system {{(pid=122429) nvme_present /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/nvmeof.py:784}} Nov 24 14:39:06.769015 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=122429) find_dsc /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:132}} Nov 24 14:39:06.770117 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.lightos [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Current host hostNQN nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e and IP(s) are ['38.102.83.12', 'fe80::f816:3eff:fe17:2fa8', '192.168.122.1', '172.24.5.1', '2001:db8::2', 'fe80::64fc:4aff:fe7b:de4b', 'fe80::fc16:3eff:fec1:6190', 'fe80::e0ae:4fff:fe1e:508c'] Nov 24 14:39:06.770456 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=122429) get_connector_properties /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:109}} Nov 24 14:39:06.770760 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e dsc: {{(pid=122429) get_connector_properties /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:112}} Nov 24 14:39:06.771287 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] <== get_connector_properties: return (65ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '38.102.83.12', 'host': 'np0005533992.novalocal', 'multipath': False, 'enforce_multipath': False, 'initiator': 'iqn.1994-05.com.redhat:5b9c7bcf3f78', 'do_local_attach': False, 'uuid': '3fc68afb-d7e4-475c-9164-ea34f338d965', 'system uuid': 'e2c36df9-c117-4156-b18b-d355565d8e8e', 'nvme_native_multipath': False, 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e', 'found_dsc': '', 'host_ips': ['38.102.83.12', 'fe80::f816:3eff:fe17:2fa8', '192.168.122.1', '172.24.5.1', '2001:db8::2', 'fe80::64fc:4aff:fe7b:de4b', 'fe80::fc16:3eff:fec1:6190', 'fe80::e0ae:4fff:fe1e:508c']} {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:39:10.784591 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connector [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Factory for iscsi on None {{(pid=122429) factory /usr/local/lib/python3.12/site-packages/os_brick/initiator/connector.py:281}} Nov 24 14:39:10.785838 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] ==> connect_volume: call "{'self': , 'connection_properties': {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a', 'target_lun': 0, 'volume_id': '7a270e29-bb4d-46d2-8e13-8481707a413a', 'auth_method': 'CHAP', 'auth_username': 'BFhrxZSPBa488fEQrE8A', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False}}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:39:10.786142 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Acquiring lock "connect_volume" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:39:10.786813 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:39:10.787091 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Connection properties {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a', 'target_lun': 0, 'volume_id': '7a270e29-bb4d-46d2-8e13-8481707a413a', 'auth_method': 'CHAP', 'auth_username': 'BFhrxZSPBa488fEQrE8A', 'auth_password': 'nHjTKMJhAk5CeZZ3', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False, 'enforce_multipath': False} {{(pid=122429) check_multipath /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:133}} Nov 24 14:39:10.788792 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd show status {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:10.796759 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] 'multipathd show status' failed. Not Retrying. {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:39:10.799978 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[b9f869d3-8206-4b6a-89a6-c778b0e25b6b]: [Errno 2] No such file or directory: 'multipathd' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:39:10.799978 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:39:10.799978 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:39:10.799978 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:39:10.799978 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:10.799978 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:39:10.799978 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:39:10.799978 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:10.799978 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root Nov 24 14:39:10.799978 np0005533992.novalocal cinder-backup[122429]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Nov 24 14:39:10.799978 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:10.799978 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute Nov 24 14:39:10.799978 np0005533992.novalocal cinder-backup[122429]: return putils.execute(on_execute=on_execute, Nov 24 14:39:10.799978 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:10.799978 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute Nov 24 14:39:10.799978 np0005533992.novalocal cinder-backup[122429]: obj = subprocess.Popen(cmd, Nov 24 14:39:10.799978 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:10.799978 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__ Nov 24 14:39:10.799978 np0005533992.novalocal cinder-backup[122429]: self._execute_child(args, executable, preexec_fn, close_fds, Nov 24 14:39:10.799978 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child Nov 24 14:39:10.799978 np0005533992.novalocal cinder-backup[122429]: raise child_exception_type(errno_num, err_msg, err_filename) Nov 24 14:39:10.799978 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: 'multipathd' Nov 24 14:39:10.801271 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[b9f869d3-8206-4b6a-89a6-c778b0e25b6b]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root\n return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute\n return putils.execute(on_execute=on_execute,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute\n obj = subprocess.Popen(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__\n self._execute_child(args, executable, preexec_fn, close_fds,\n File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child\n raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: \'multipathd\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:10.803325 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Acquiring lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:39:10.803624 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:39:10.804027 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Trying to connect to iSCSI portal 38.102.83.12:3260 Nov 24 14:39:10.804956 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a -p 38.102.83.12:3260 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:10.816509 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a -p 38.102.83.12:3260" returned: 21 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:10.817681 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[ea5580c5-87d1-46e2-8f6d-f0bb945a7397]: (4, ('', 'iscsiadm: No records found\n')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:10.818110 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] iscsiadm (): stdout= stderr=iscsiadm: No records found Nov 24 14:39:10.818110 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:10.820609 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a -p 38.102.83.12:3260 --interface default --op new {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:10.831056 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a -p 38.102.83.12:3260 --interface default --op new" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:10.831335 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[3dc2a85d-363b-4953-9372-e04b197f7a43]: (4, ('New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 38.102.83.12,3260,-1 iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a] added\n', 'iscsiadm: Config file line 381 does not have value\niscsiadm: Config file line 381 does not have value\niscsiadm: Config file line 381 does not have value\n')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:10.832074 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--interface', 'default', '--op', 'new'): stdout=New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 38.102.83.12,3260,-1 iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a] added Nov 24 14:39:10.832074 np0005533992.novalocal cinder-backup[122429]: stderr=iscsiadm: Config file line 381 does not have value Nov 24 14:39:10.832074 np0005533992.novalocal cinder-backup[122429]: iscsiadm: Config file line 381 does not have value Nov 24 14:39:10.832074 np0005533992.novalocal cinder-backup[122429]: iscsiadm: Config file line 381 does not have value Nov 24 14:39:10.832074 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:10.832074 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Retrying to connect to iSCSI portal 38.102.83.12:3260 {{(pid=122429) _connect_to_iscsi_portal_unsafe /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1133}} Nov 24 14:39:10.832755 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Finished call to 'os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe' after 0.029(s), this was the 1st time calling it. {{(pid=122429) log_it /usr/local/lib/python3.12/site-packages/tenacity/after.py:44}} Nov 24 14:39:10.833085 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Retrying os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe in 1.0 seconds as it raised BrickException: An unknown exception occurred.. {{(pid=122429) log_it /usr/local/lib/python3.12/site-packages/tenacity/before_sleep.py:65}} Nov 24 14:39:11.834235 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Trying to connect to iSCSI portal 38.102.83.12:3260 Nov 24 14:39:11.837848 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a -p 38.102.83.12:3260 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:11.847502 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a -p 38.102.83.12:3260" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[1b238eb6-7d3b-42ab-aaef-3f1b063a3467]: (4, ('# BEGIN RECORD 6.2.1.11\nnode.name = iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a\nnode.tpgt = -1\nnode.startup = automatic\nnode.leading_login = No\niface.iscsi_ifacename = default\niface.net_ifacename = \niface.ipaddress = \niface.prefix_len = 0\niface.hwaddress = \niface.transport_name = tcp\niface.initiatorname = \niface.state = \niface.vlan_id = 0\niface.vlan_priority = 0\niface.vlan_state = \niface.iface_num = 0\niface.mtu = 0\niface.port = 0\niface.bootproto = \niface.subnet_mask = \niface.gateway = \niface.dhcp_alt_client_id_state = \niface.dhcp_alt_client_id = \niface.dhcp_dns = \niface.dhcp_learn_iqn = \niface.dhcp_req_vendor_id_state = \niface.dhcp_vendor_id_state = \niface.dhcp_vendor_id = \niface.dhcp_slp_da = \niface.fragmentation = \niface.gratuitous_arp = \niface.incoming_forwarding = \niface.tos_state = \niface.tos = 0\niface.ttl = 0\niface.delayed_ack = \niface.tcp_nagle = \niface.tcp_wsf_state = \niface.tcp_wsf = 0\niface.tcp_timer_scale = 0\niface.tcp_timestamp = \niface.redirect = \niface.def_task_mgmt_timeout = 0\niface.header_digest = \niface.data_digest = \niface.immediate_data = \niface.initial_r2t = \niface.data_seq_inorder = \niface.data_pdu_inorder = \niface.erl = 0\niface.max_receive_data_len = 0\niface.first_burst_len = 0\niface.max_outstanding_r2t = 0\niface.max_burst_len = 0\niface.chap_auth = \niface.bidi_chap = \niface.strict_login_compliance = \niface.discovery_auth = \niface.discovery_logout = \nnode.discovery_address = \nnode.discovery_port = 0\nnode.discovery_type = fw\nnode.session.initial_cmdsn = 0\nnode.session.initial_login_retry_max = 8\nnode.session.xmit_thread_priority = 0\nnode.session.cmds_max = 128\nnode.session.queue_depth = 32\nnode.session.nr_sessions = 1\nnode.session.auth.authmethod = None\nnode.session.auth.username = \nnode.session.auth.password = \nnode.session.auth.username_in = \nnode.session.auth.password_in = \nnode.session.auth.chap_algs = SHA3-256,SHA256\nnode.session.timeo.replacement_timeout = 120\nnode.session.err_timeo.abort_timeout = 15\nnode.session.err_timeo.lu_reset_timeout = 30\nnode.session.err_timeo.tgt_reset_timeout = 30\nnode.session.err_timeo.host_reset_timeout = 60\nnode.session.iscsi.FastAbort = Yes\nnode.session.iscsi.InitialR2T = No\nnode.session.iscsi.ImmediateData = Yes\nnode.session.iscsi.FirstBurstLength = 262144\nnode.session.iscsi.MaxBurstLength = 16776192\nnode.session.iscsi.DefaultTime2Retain = 0\nnode.session.iscsi.DefaultTime2Wait = 2\nnode.session.iscsi.MaxConnections = 1\nnode.session.iscsi.MaxOutstandingR2T = 1\nnode.session.iscsi.ERL = 0\nnode.session.scan = auto\nnode.session.reopen_max = 0\nnode.session.conn_reopen_log_freq = 1\nnode.conn[0].address = 38.102.83.12\nnode.conn[0].port = 3260\nnode.conn[0].startup = manual\nnode.conn[0].tcp.window_size = 524288\nnode.conn[0].tcp.type_of_service = 0\nnode.conn[0].timeo.logout_timeout = 15\nnode.conn[0].timeo.login_timeout = 15\nnode.conn[0].timeo.auth_timeout = 45\nnode.conn[0].timeo.noop_out_interval = 5\nnode.conn[0].timeo.noop_out_timeout = 5\nnode.conn[0].iscsi.MaxXmitDataSegmentLength = 0\nnode.conn[0].iscsi.MaxRecvDataSegmentLength = 262144\nnode.conn[0].iscsi.HeaderDigest = None\nnode.conn[0].iscsi.DataDigest = None\nnode.conn[0].iscsi.IFMarker = No\nnode.conn[0].iscsi.OFMarker = No\n# END RECORD\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] iscsiadm (): stdout=# BEGIN RECORD 6.2.1.11 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.name = iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.tpgt = -1 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.startup = automatic Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.leading_login = No Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.iscsi_ifacename = default Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.net_ifacename = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.ipaddress = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.prefix_len = 0 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.hwaddress = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.transport_name = tcp Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.initiatorname = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.state = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.vlan_id = 0 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.vlan_priority = 0 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.vlan_state = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.iface_num = 0 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.mtu = 0 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.port = 0 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.bootproto = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.subnet_mask = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.gateway = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_alt_client_id_state = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_alt_client_id = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_dns = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_learn_iqn = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_req_vendor_id_state = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_vendor_id_state = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_vendor_id = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_slp_da = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.fragmentation = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.gratuitous_arp = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.incoming_forwarding = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.tos_state = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.tos = 0 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.ttl = 0 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.delayed_ack = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.tcp_nagle = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.tcp_wsf_state = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.tcp_wsf = 0 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.tcp_timer_scale = 0 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.tcp_timestamp = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.redirect = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.def_task_mgmt_timeout = 0 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.header_digest = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.data_digest = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.immediate_data = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.initial_r2t = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.data_seq_inorder = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.data_pdu_inorder = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.erl = 0 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.max_receive_data_len = 0 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.first_burst_len = 0 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.max_outstanding_r2t = 0 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.max_burst_len = 0 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.chap_auth = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.bidi_chap = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.strict_login_compliance = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.discovery_auth = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: iface.discovery_logout = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.discovery_address = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.discovery_port = 0 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.discovery_type = fw Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.initial_cmdsn = 0 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.initial_login_retry_max = 8 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.xmit_thread_priority = 0 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.cmds_max = 128 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.queue_depth = 32 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.nr_sessions = 1 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.auth.authmethod = None Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.auth.username = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.auth.password = *** Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.auth.username_in = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.auth.password_in = Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.auth.chap_algs = SHA3-256,SHA256 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.timeo.replacement_timeout = 120 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.abort_timeout = 15 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.lu_reset_timeout = 30 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.tgt_reset_timeout = 30 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.host_reset_timeout = 60 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.FastAbort = Yes Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.InitialR2T = No Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.ImmediateData = Yes Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.FirstBurstLength = 262144 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxBurstLength = 16776192 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.DefaultTime2Retain = 0 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.DefaultTime2Wait = 2 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxConnections = 1 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxOutstandingR2T = 1 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.ERL = 0 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.scan = auto Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.reopen_max = 0 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.session.conn_reopen_log_freq = 1 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.conn[0].address = 38.102.83.12 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.conn[0].port = 3260 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.conn[0].startup = manual Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.conn[0].tcp.window_size = 524288 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.conn[0].tcp.type_of_service = 0 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.logout_timeout = 15 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.login_timeout = 15 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.auth_timeout = 45 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.noop_out_interval = 5 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.noop_out_timeout = 5 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.MaxXmitDataSegmentLength = 0 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144 Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.HeaderDigest = None Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.DataDigest = None Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.IFMarker = No Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.OFMarker = No Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: # END RECORD Nov 24 14:39:11.850797 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:11.855037 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a -p 38.102.83.12:3260 --op update -n node.session.scan -v manual {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:11.863046 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a -p 38.102.83.12:3260 --op update -n node.session.scan -v manual" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:11.864122 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[99c72739-0f9e-4456-9b3d-39904f462b01]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:11.864122 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.scan', '-v', 'manual'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:11.868413 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a -p 38.102.83.12:3260 --op update -n node.session.auth.authmethod -v CHAP {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:11.888586 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a -p 38.102.83.12:3260 --op update -n node.session.auth.authmethod -v CHAP" returned: 0 in 0.023s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:11.890023 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[842dfe21-a6d0-4e27-b1f0-5556132f5c14]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:11.890740 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.authmethod', '-v', 'CHAP'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:11.892199 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a -p 38.102.83.12:3260 --op update -n node.session.auth.username -v BFhrxZSPBa488fEQrE8A {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:11.905299 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a -p 38.102.83.12:3260 --op update -n node.session.auth.username -v BFhrxZSPBa488fEQrE8A" returned: 0 in 0.013s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:11.905671 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[b2e0279c-d036-4e75-adbe-e3e0a6860046]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:11.906205 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.username', '-v', 'BFhrxZSPBa488fEQrE8A'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:11.908587 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a -p 38.102.83.12:3260 --op update -n node.session.auth.password -v *** {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:11.918538 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a -p 38.102.83.12:3260 --op update -n node.session.auth.password -v ***" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:11.920487 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[0f05e199-9744-4e70-9feb-8dcf0e5e5061]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:11.920487 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.password', '-v', '***'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:11.921416 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:11.930774 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 21 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:11.931137 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[bb33ec4b-05d7-46d6-a691-9a5ae7283652]: (4, ('', 'iscsiadm: No active sessions.\n')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:11.931731 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('-m', 'session'): stdout= stderr=iscsiadm: No active sessions. Nov 24 14:39:11.931731 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:39:11.932149 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] iscsi session list stdout= stderr=iscsiadm: No active sessions. Nov 24 14:39:11.932149 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:39:11.932529 np0005533992.novalocal cinder-backup[122429]: WARNING os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions. Nov 24 14:39:11.932529 np0005533992.novalocal cinder-backup[122429]: Nov 24 14:39:11.934216 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a -p 38.102.83.12:3260 --login {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:11.953393 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a -p 38.102.83.12:3260 --login" returned: 0 in 0.020s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:11.954404 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[fd2415cd-3423-4484-921e-b8870ae36aa3]: (4, ('Login to [iface: default, target: iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a, portal: 38.102.83.12,3260] successful.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:11.955444 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--login',): stdout=Login to [iface: default, target: iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a, portal: 38.102.83.12,3260] successful. Nov 24 14:39:11.955444 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:11.956581 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a -p 38.102.83.12:3260 --op update -n node.startup -v automatic {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:11.970455 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a -p 38.102.83.12:3260 --op update -n node.startup -v automatic" returned: 0 in 0.013s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:11.971019 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[e47078e2-d6c8-4321-8dc2-c168ccea7fcb]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:11.971693 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:11.972930 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:11.986600 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.014s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:11.987507 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[02c790a0-4c51-4b8e-8a90-6846a8e49fc8]: (4, ('tcp: [23] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:11.988088 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('-m', 'session'): stdout=tcp: [23] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a (non-flash) Nov 24 14:39:11.988088 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:39:11.988489 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] iscsi session list stdout=tcp: [23] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a (non-flash) Nov 24 14:39:11.988489 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:39:11.989136 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: held 1.186s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:39:11.989632 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Connected to 38.102.83.12:3260 {{(pid=122429) _connect_vol /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:659}} Nov 24 14:39:11.990399 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] HCTL ('2', '-', '-', 0) found on session 23 with lun 0 {{(pid=122429) get_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:788}} Nov 24 14:39:11.990817 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Scanning host 2 c: -, t: -, l: 0) {{(pid=122429) scan_iscsi /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:814}} Nov 24 14:39:11.992002 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/class/scsi_host/host2/scan {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:12.018440 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/class/scsi_host/host2/scan" returned: 0 in 0.026s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:12.020204 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[69af76e1-619f-4a39-928e-dbf3ce4230f5]: (4, ('- - 0', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:12.020204 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Searching for a device in session 23 and hctl ['2', '*', '*', 0] yield: None {{(pid=122429) device_name_by_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:808}} Nov 24 14:39:13.022427 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Searching for a device in session 23 and hctl ['2', '*', '*', 0] yield: sda {{(pid=122429) device_name_by_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:808}} Nov 24 14:39:13.023312 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Connected to sda using {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a', 'target_lun': 0, 'volume_id': '7a270e29-bb4d-46d2-8e13-8481707a413a', 'auth_method': 'CHAP', 'auth_username': 'BFhrxZSPBa488fEQrE8A', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False} {{(pid=122429) _connect_vol /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:687}} Nov 24 14:39:13.024286 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" :: held 2.237s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:39:13.024892 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] <== connect_volume: return (2238ms) {'type': 'block', 'scsi_wwn': '360014057d68763049c6412a86edce65f', 'path': '/dev/sda'} {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:39:13.025509 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sda {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:13.043815 np0005533992.novalocal sudo[161390]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sda Nov 24 14:39:13.361626 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sda" returned: 0 in 0.335s {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:13.362634 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] starting restore of backup volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda container: tempest-volumesbackupstest-backup-container-1248068680, to new volume 7a270e29-bb4d-46d2-8e13-8481707a413a, backup: f5cd5608-e8db-496c-be30-dfa768060eda. {{(pid=122429) restore /opt/stack/cinder/cinder/backup/chunkeddriver.py:783}} Nov 24 14:39:13.363097 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] _read_metadata started, container name: tempest-volumesbackupstest-backup-container-1248068680, metadata filename: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda_metadata. {{(pid=122429) _read_metadata /opt/stack/cinder/cinder/backup/chunkeddriver.py:323}} Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] _read_metadata finished. Metadata: { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "backup_description": null, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "backup_id": "f5cd5608-e8db-496c-be30-dfa768060eda", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "backup_name": "tempest-VolumesBackupsTest-Backup-1951522908", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "created_at": "2025-11-24 19:38:41+00:00", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "objects": [ Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00001": { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "md5": "82571f7121cb9475563edde615e4e4df", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "offset": 0 Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00002": { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "offset": 52428800 Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00003": { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "md5": "a35891500449df17e893f84bd88472fb", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "offset": 104857600 Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00004": { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "offset": 157286400 Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00005": { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "offset": 209715200 Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00006": { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "offset": 262144000 Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00007": { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "offset": 314572800 Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00008": { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "offset": 367001600 Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00009": { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "offset": 419430400 Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00010": { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "offset": 471859200 Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00011": { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "offset": 524288000 Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00012": { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "offset": 576716800 Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00013": { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "offset": 629145600 Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00014": { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "offset": 681574400 Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00015": { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "offset": 734003200 Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00016": { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "offset": 786432000 Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00017": { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "offset": 838860800 Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00018": { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "offset": 891289600 Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00019": { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "offset": 943718400 Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00020": { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "offset": 996147200 Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00021": { Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "length": 25165824, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "md5": "77377273b0a4b61febdbf7bbf52b9db9", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "offset": 1048576000 Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: ], Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "parent_id": null, Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "version": "1.0.0", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "volume_id": "e6f9e8be-7196-4cc0-bea9-056b2e289daa", Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: "volume_meta": "{\"version\": 2, \"volume-base-metadata\": {\"id\": \"e6f9e8be-7196-4cc0-bea9-056b2e289daa\", \"_name_id\": null, \"use_quota\": true, \"ec2_id\": null, \"user_id\": \"0e48df3b35f84ee88136fd47ee531c1f\", \"project_id\": \"c642367dafaa45dd89634e7ec9822ef2\", \"snapshot_id\": null, \"cluster_name\": null, \"host\": \"np0005533992.novalocal@lvmdriver-1#lvmdriver-1\", \"size\": 1, \"availability_zone\": \"nova\", \"status\": \"backing-up\", \"attach_status\": \"detached\", \"migration_status\": null, \"scheduled_at\": \"2025-11-24T19:38:39.000000\", \"launched_at\": \"2025-11-24T19:38:41.000000\", \"terminated_at\": null, \"display_name\": \"tempest-VolumesBackupsTest-Volume-1627712353\", \"display_description\": null, \"provider_location\": \"38.102.83.12:3260,iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa 0\", \"provider_auth\": \"CHAP 9Ckqgeps5ikyyHL3sYQC 3o8otLEdFarmqEwD\", \"provider_geometry\": null, \"provider_id\": null, \"volume_type_id\": \"1f8776b6-c527-4643-876a-506285adfc66\", \"source_volid\": null, \"encryption_key_id\": null, \"consistencygroup_id\": null, \"group_id\": null, \"bootable\": true, \"multiattach\": false, \"replication_status\": null, \"replication_extended_status\": null, \"replication_driver_data\": null, \"previous_status\": \"available\", \"service_uuid\": \"6d0941fd-19ec-48ae-82b1-f25526175227\", \"shared_targets\": false, \"deleted_at\": null, \"deleted\": false, \"created_at\": \"2025-11-24T19:38:39.000000\", \"updated_at\": \"2025-11-24T19:38:43.000000\"}, \"volume-glance-metadata\": {\"hw_rng_model\": \"virtio\", \"owner_specified.openstack.md5\": \"\", \"owner_specified.openstack.sha256\": \"\", \"owner_specified.openstack.object\": \"images/cirros-0.6.3-x86_64-disk\", \"os_glance_importing_to_stores\": \"\", \"os_glance_failed_import\": \"\", \"image_id\": \"bfe30d4b-f478-4b3d-b058-80ea56840f0b\", \"image_name\": \"cirros-0.6.3-x86_64-disk\", \"checksum\": \"87617e24a5e30cb3b87fda8c0764838f\", \"container_format\": \"bare\", \"disk_format\": \"qcow2\", \"min_disk\": \"0\", \"min_ram\": \"0\", \"size\": \"21692416\"}}" Nov 24 14:39:13.401737 np0005533992.novalocal cinder-backup[122429]: }. {{(pid=122429) _read_metadata /opt/stack/cinder/cinder/backup/chunkeddriver.py:330}} Nov 24 14:39:13.406368 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Restoring backup version 1.0.0 {{(pid=122429) restore /opt/stack/cinder/cinder/backup/chunkeddriver.py:796}} Nov 24 14:39:13.406368 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] _read_metadata started, container name: tempest-volumesbackupstest-backup-container-1248068680, metadata filename: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda_metadata. {{(pid=122429) _read_metadata /opt/stack/cinder/cinder/backup/chunkeddriver.py:323}} Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] _read_metadata finished. Metadata: { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "backup_description": null, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "backup_id": "f5cd5608-e8db-496c-be30-dfa768060eda", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "backup_name": "tempest-VolumesBackupsTest-Backup-1951522908", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "created_at": "2025-11-24 19:38:41+00:00", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "objects": [ Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00001": { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "md5": "82571f7121cb9475563edde615e4e4df", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "offset": 0 Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00002": { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "offset": 52428800 Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00003": { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "md5": "a35891500449df17e893f84bd88472fb", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "offset": 104857600 Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00004": { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "offset": 157286400 Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00005": { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "offset": 209715200 Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00006": { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "offset": 262144000 Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00007": { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "offset": 314572800 Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00008": { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "offset": 367001600 Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00009": { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "offset": 419430400 Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00010": { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "offset": 471859200 Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00011": { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "offset": 524288000 Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00012": { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "offset": 576716800 Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00013": { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "offset": 629145600 Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00014": { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "offset": 681574400 Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00015": { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "offset": 734003200 Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00016": { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "offset": 786432000 Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00017": { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "offset": 838860800 Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00018": { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "offset": 891289600 Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00019": { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "offset": 943718400 Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00020": { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "offset": 996147200 Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00021": { Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "length": 25165824, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "md5": "77377273b0a4b61febdbf7bbf52b9db9", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "offset": 1048576000 Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: ], Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "parent_id": null, Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "version": "1.0.0", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "volume_id": "e6f9e8be-7196-4cc0-bea9-056b2e289daa", Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: "volume_meta": "{\"version\": 2, \"volume-base-metadata\": {\"id\": \"e6f9e8be-7196-4cc0-bea9-056b2e289daa\", \"_name_id\": null, \"use_quota\": true, \"ec2_id\": null, \"user_id\": \"0e48df3b35f84ee88136fd47ee531c1f\", \"project_id\": \"c642367dafaa45dd89634e7ec9822ef2\", \"snapshot_id\": null, \"cluster_name\": null, \"host\": \"np0005533992.novalocal@lvmdriver-1#lvmdriver-1\", \"size\": 1, \"availability_zone\": \"nova\", \"status\": \"backing-up\", \"attach_status\": \"detached\", \"migration_status\": null, \"scheduled_at\": \"2025-11-24T19:38:39.000000\", \"launched_at\": \"2025-11-24T19:38:41.000000\", \"terminated_at\": null, \"display_name\": \"tempest-VolumesBackupsTest-Volume-1627712353\", \"display_description\": null, \"provider_location\": \"38.102.83.12:3260,iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa iqn.2010-10.org.openstack:volume-e6f9e8be-7196-4cc0-bea9-056b2e289daa 0\", \"provider_auth\": \"CHAP 9Ckqgeps5ikyyHL3sYQC 3o8otLEdFarmqEwD\", \"provider_geometry\": null, \"provider_id\": null, \"volume_type_id\": \"1f8776b6-c527-4643-876a-506285adfc66\", \"source_volid\": null, \"encryption_key_id\": null, \"consistencygroup_id\": null, \"group_id\": null, \"bootable\": true, \"multiattach\": false, \"replication_status\": null, \"replication_extended_status\": null, \"replication_driver_data\": null, \"previous_status\": \"available\", \"service_uuid\": \"6d0941fd-19ec-48ae-82b1-f25526175227\", \"shared_targets\": false, \"deleted_at\": null, \"deleted\": false, \"created_at\": \"2025-11-24T19:38:39.000000\", \"updated_at\": \"2025-11-24T19:38:43.000000\"}, \"volume-glance-metadata\": {\"hw_rng_model\": \"virtio\", \"owner_specified.openstack.md5\": \"\", \"owner_specified.openstack.sha256\": \"\", \"owner_specified.openstack.object\": \"images/cirros-0.6.3-x86_64-disk\", \"os_glance_importing_to_stores\": \"\", \"os_glance_failed_import\": \"\", \"image_id\": \"bfe30d4b-f478-4b3d-b058-80ea56840f0b\", \"image_name\": \"cirros-0.6.3-x86_64-disk\", \"checksum\": \"87617e24a5e30cb3b87fda8c0764838f\", \"container_format\": \"bare\", \"disk_format\": \"qcow2\", \"min_disk\": \"0\", \"min_ram\": \"0\", \"size\": \"21692416\"}}" Nov 24 14:39:13.423816 np0005533992.novalocal cinder-backup[122429]: }. {{(pid=122429) _read_metadata /opt/stack/cinder/cinder/backup/chunkeddriver.py:330}} Nov 24 14:39:13.428829 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] v1 volume backup restore of f5cd5608-e8db-496c-be30-dfa768060eda started. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:700}} Nov 24 14:39:13.428829 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] metadata_object_names = ['volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00001', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00002', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00003', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00004', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00005', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00006', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00007', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00008', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00009', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00010', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00011', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00012', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00013', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00014', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00015', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00016', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00017', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00018', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00019', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00020', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00021']. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:707}} Nov 24 14:39:13.476788 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] generated object list: ['volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00001', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00002', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00003', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00004', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00005', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00006', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00007', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00008', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00009', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00010', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00011', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00012', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00013', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00014', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00015', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00016', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00017', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00018', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00019', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00020', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00021', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda_metadata', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda_sha256file']. {{(pid=122429) _generate_object_names /opt/stack/cinder/cinder/backup/chunkeddriver.py:263}} Nov 24 14:39:13.480790 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_db.sqlalchemy.engines [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=122429) _check_effective_sql_mode /usr/local/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:397}} Nov 24 14:39:13.486093 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: f5cd5608-e8db-496c-be30-dfa768060eda, container: tempest-volumesbackupstest-backup-container-1248068680, object name: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00001, volume: 7a270e29-bb4d-46d2-8e13-8481707a413a. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:39:13.592532 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:39:14.047860 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: f5cd5608-e8db-496c-be30-dfa768060eda, container: tempest-volumesbackupstest-backup-container-1248068680, object name: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00002, volume: 7a270e29-bb4d-46d2-8e13-8481707a413a. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:39:14.068647 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:39:14.327169 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: f5cd5608-e8db-496c-be30-dfa768060eda, container: tempest-volumesbackupstest-backup-container-1248068680, object name: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00003, volume: 7a270e29-bb4d-46d2-8e13-8481707a413a. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:39:14.346198 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:39:14.596644 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: f5cd5608-e8db-496c-be30-dfa768060eda, container: tempest-volumesbackupstest-backup-container-1248068680, object name: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00004, volume: 7a270e29-bb4d-46d2-8e13-8481707a413a. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:39:14.611309 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:39:14.820350 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: f5cd5608-e8db-496c-be30-dfa768060eda, container: tempest-volumesbackupstest-backup-container-1248068680, object name: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00005, volume: 7a270e29-bb4d-46d2-8e13-8481707a413a. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:39:14.839991 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:39:15.065414 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: f5cd5608-e8db-496c-be30-dfa768060eda, container: tempest-volumesbackupstest-backup-container-1248068680, object name: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00006, volume: 7a270e29-bb4d-46d2-8e13-8481707a413a. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:39:15.084259 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:39:15.304907 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: f5cd5608-e8db-496c-be30-dfa768060eda, container: tempest-volumesbackupstest-backup-container-1248068680, object name: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00007, volume: 7a270e29-bb4d-46d2-8e13-8481707a413a. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:39:15.321690 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:39:15.540763 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: f5cd5608-e8db-496c-be30-dfa768060eda, container: tempest-volumesbackupstest-backup-container-1248068680, object name: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00008, volume: 7a270e29-bb4d-46d2-8e13-8481707a413a. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:39:15.554295 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:39:15.753574 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: f5cd5608-e8db-496c-be30-dfa768060eda, container: tempest-volumesbackupstest-backup-container-1248068680, object name: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00009, volume: 7a270e29-bb4d-46d2-8e13-8481707a413a. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:39:15.770089 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:39:15.974288 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: f5cd5608-e8db-496c-be30-dfa768060eda, container: tempest-volumesbackupstest-backup-container-1248068680, object name: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00010, volume: 7a270e29-bb4d-46d2-8e13-8481707a413a. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:39:15.992799 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:39:16.171879 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: f5cd5608-e8db-496c-be30-dfa768060eda, container: tempest-volumesbackupstest-backup-container-1248068680, object name: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00011, volume: 7a270e29-bb4d-46d2-8e13-8481707a413a. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:39:16.190633 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:39:16.392050 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: f5cd5608-e8db-496c-be30-dfa768060eda, container: tempest-volumesbackupstest-backup-container-1248068680, object name: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00012, volume: 7a270e29-bb4d-46d2-8e13-8481707a413a. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:39:16.415146 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:39:16.618952 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: f5cd5608-e8db-496c-be30-dfa768060eda, container: tempest-volumesbackupstest-backup-container-1248068680, object name: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00013, volume: 7a270e29-bb4d-46d2-8e13-8481707a413a. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:39:16.641023 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:39:16.825638 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: f5cd5608-e8db-496c-be30-dfa768060eda, container: tempest-volumesbackupstest-backup-container-1248068680, object name: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00014, volume: 7a270e29-bb4d-46d2-8e13-8481707a413a. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:39:16.841916 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:39:17.040446 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: f5cd5608-e8db-496c-be30-dfa768060eda, container: tempest-volumesbackupstest-backup-container-1248068680, object name: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00015, volume: 7a270e29-bb4d-46d2-8e13-8481707a413a. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:39:17.057550 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:39:17.259438 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: f5cd5608-e8db-496c-be30-dfa768060eda, container: tempest-volumesbackupstest-backup-container-1248068680, object name: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00016, volume: 7a270e29-bb4d-46d2-8e13-8481707a413a. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:39:17.280819 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:39:17.548040 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: f5cd5608-e8db-496c-be30-dfa768060eda, container: tempest-volumesbackupstest-backup-container-1248068680, object name: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00017, volume: 7a270e29-bb4d-46d2-8e13-8481707a413a. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:39:17.569063 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:39:17.784370 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: f5cd5608-e8db-496c-be30-dfa768060eda, container: tempest-volumesbackupstest-backup-container-1248068680, object name: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00018, volume: 7a270e29-bb4d-46d2-8e13-8481707a413a. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:39:17.802121 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:39:18.021112 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: f5cd5608-e8db-496c-be30-dfa768060eda, container: tempest-volumesbackupstest-backup-container-1248068680, object name: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00019, volume: 7a270e29-bb4d-46d2-8e13-8481707a413a. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:39:18.037578 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:39:18.226987 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: f5cd5608-e8db-496c-be30-dfa768060eda, container: tempest-volumesbackupstest-backup-container-1248068680, object name: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00020, volume: 7a270e29-bb4d-46d2-8e13-8481707a413a. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:39:18.242313 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:39:18.427655 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: f5cd5608-e8db-496c-be30-dfa768060eda, container: tempest-volumesbackupstest-backup-container-1248068680, object name: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00021, volume: 7a270e29-bb4d-46d2-8e13-8481707a413a. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:39:18.445031 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:39:18.529740 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] v1 volume backup restore of f5cd5608-e8db-496c-be30-dfa768060eda finished. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:772}} Nov 24 14:39:18.529930 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Restoring volume base metadata {{(pid=122429) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} Nov 24 14:39:18.542844 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] No metadata of type 'volume-metadata' to restore {{(pid=122429) put /opt/stack/cinder/cinder/backup/driver.py:345}} Nov 24 14:39:18.544981 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Restoring volume glance metadata {{(pid=122429) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}} Nov 24 14:39:18.663479 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] restore f5cd5608-e8db-496c-be30-dfa768060eda to 7a270e29-bb4d-46d2-8e13-8481707a413a finished. {{(pid=122429) restore /opt/stack/cinder/cinder/backup/chunkeddriver.py:837}} Nov 24 14:39:18.672517 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sda {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:18.700050 np0005533992.novalocal sudo[161493]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sda Nov 24 14:39:19.049077 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sda" returned: 0 in 0.376s {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:19.050582 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] ==> disconnect_volume: call "{'args': (, {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a', 'target_lun': 0, 'volume_id': '7a270e29-bb4d-46d2-8e13-8481707a413a', 'auth_method': 'CHAP', 'auth_username': 'BFhrxZSPBa488fEQrE8A', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False, 'enforce_multipath': False}, {'type': 'block', 'scsi_wwn': '360014057d68763049c6412a86edce65f', 'path': '/dev/sda'}), 'kwargs': {'force': False}}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:39:19.050908 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Acquiring lock "connect_volume" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:39:19.051591 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:39:19.053600 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m discoverydb -o show -P 1 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:19.066079 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m discoverydb -o show -P 1" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:19.066339 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[e3149c29-2366-4148-bcf4-194a00cd8f64]: (4, ('SENDTARGETS:\nNo targets found.\niSNS:\nNo targets found.\nSTATIC:\nTarget: iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a\n\tPortal: 38.102.83.12:3260,-1\n\t\tIface Name: default\nFIRMWARE:\nNo targets found.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:19.066855 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ['-m', 'discoverydb', '-o', 'show', '-P', 1]: stdout=SENDTARGETS: Nov 24 14:39:19.066855 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:39:19.066855 np0005533992.novalocal cinder-backup[122429]: iSNS: Nov 24 14:39:19.066855 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:39:19.066855 np0005533992.novalocal cinder-backup[122429]: STATIC: Nov 24 14:39:19.066855 np0005533992.novalocal cinder-backup[122429]: Target: iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a Nov 24 14:39:19.066855 np0005533992.novalocal cinder-backup[122429]: Portal: 38.102.83.12:3260,-1 Nov 24 14:39:19.066855 np0005533992.novalocal cinder-backup[122429]: Iface Name: default Nov 24 14:39:19.066855 np0005533992.novalocal cinder-backup[122429]: FIRMWARE: Nov 24 14:39:19.066855 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:39:19.066855 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:39:19.067162 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Regex to get portals from discoverydb: ^SENDTARGETS: Nov 24 14:39:19.067162 np0005533992.novalocal cinder-backup[122429]: .*?^DiscoveryAddress: 38.102.83.12,3260.*? Nov 24 14:39:19.067162 np0005533992.novalocal cinder-backup[122429]: (.*?)^(?:DiscoveryAddress|iSNS):.* {{(pid=122429) _get_discoverydb_portals /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:392}} Nov 24 14:39:19.067620 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Getting connected devices for (ips,iqns,luns)=[('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a', 0)] {{(pid=122429) _get_connection_devices /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:855}} Nov 24 14:39:19.068502 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:19.078119 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node" returned: 0 in 0.009s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:19.078362 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[2188411f-4479-4c0b-9015-d9154f06aa71]: (4, ('38.102.83.12:3260,4294967295 iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:19.079278 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:19.088880 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.009s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:19.089119 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[c99f2cd3-2f0e-4772-b57a-f86d2aa92e62]: (4, ('tcp: [23] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:19.089517 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('-m', 'session'): stdout=tcp: [23] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a (non-flash) Nov 24 14:39:19.089517 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:39:19.089811 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] iscsi session list stdout=tcp: [23] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a (non-flash) Nov 24 14:39:19.089811 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:39:19.090818 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Resulting device map defaultdict(. at 0x7f5b72945120>, {('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a'): ({'sda'}, set())}) {{(pid=122429) _get_connection_devices /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:888}} Nov 24 14:39:19.091222 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Removing single pathed devices sda {{(pid=122429) remove_connection /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:381}} Nov 24 14:39:19.092136 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd show status {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:19.098524 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] 'multipathd show status' failed. Not Retrying. {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:39:19.100651 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[22fa9784-1fc4-4cf2-8b95-a317d88cf015]: [Errno 2] No such file or directory: 'multipathd' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:39:19.100651 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:39:19.100651 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:39:19.100651 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:39:19.100651 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:19.100651 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:39:19.100651 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:39:19.100651 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:19.100651 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root Nov 24 14:39:19.100651 np0005533992.novalocal cinder-backup[122429]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Nov 24 14:39:19.100651 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:19.100651 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute Nov 24 14:39:19.100651 np0005533992.novalocal cinder-backup[122429]: return putils.execute(on_execute=on_execute, Nov 24 14:39:19.100651 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:19.100651 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute Nov 24 14:39:19.100651 np0005533992.novalocal cinder-backup[122429]: obj = subprocess.Popen(cmd, Nov 24 14:39:19.100651 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:19.100651 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__ Nov 24 14:39:19.100651 np0005533992.novalocal cinder-backup[122429]: self._execute_child(args, executable, preexec_fn, close_fds, Nov 24 14:39:19.100651 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child Nov 24 14:39:19.100651 np0005533992.novalocal cinder-backup[122429]: raise child_exception_type(errno_num, err_msg, err_filename) Nov 24 14:39:19.100651 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: 'multipathd' Nov 24 14:39:19.102277 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[22fa9784-1fc4-4cf2-8b95-a317d88cf015]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root\n return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute\n return putils.execute(on_execute=on_execute,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute\n obj = subprocess.Popen(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__\n self._execute_child(args, executable, preexec_fn, close_fds,\n File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child\n raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: \'multipathd\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:19.102868 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Flushing IO for device /dev/sda {{(pid=122429) flush_device_io /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:441}} Nov 24 14:39:19.103688 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blockdev --flushbufs /dev/sda {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:19.110855 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "blockdev --flushbufs /dev/sda" returned: 0 in 0.007s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:19.111150 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[050fb87e-031c-4dac-8a5a-04391e4fbefa]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:19.111514 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Remove SCSI device /dev/sda with /sys/block/sda/device/delete {{(pid=122429) remove_scsi_device /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:151}} Nov 24 14:39:19.112368 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/block/sda/device/delete {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:19.154342 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/block/sda/device/delete" returned: 0 in 0.042s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:19.154653 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[b987477b-0095-4c7a-b814-df721b33dacc]: (4, ('1', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:19.155155 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Checking to see if SCSI volumes sda have been removed. {{(pid=122429) wait_for_volumes_removal /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:159}} Nov 24 14:39:19.155447 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] SCSI volumes sda have been removed. {{(pid=122429) wait_for_volumes_removal /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:169}} Nov 24 14:39:19.155906 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Disconnecting from: [('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a')] {{(pid=122429) _disconnect_connection /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1201}} Nov 24 14:39:19.156928 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a -p 38.102.83.12:3260 --op update -n node.startup -v manual {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:19.167610 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a -p 38.102.83.12:3260 --op update -n node.startup -v manual" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:19.167876 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[cb2913ed-b71c-4c48-b762-a6c3c2838f2d]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:19.168325 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:19.169176 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a -p 38.102.83.12:3260 --logout {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:19.197607 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a -p 38.102.83.12:3260 --logout" returned: 0 in 0.027s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:19.197607 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[b7b7d512-6e28-4762-8560-b32002710809]: (4, ('Logging out of session [sid: 23, target: iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a, portal: 38.102.83.12,3260]\nLogout of [sid: 23, target: iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a, portal: 38.102.83.12,3260] successful.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:19.197789 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--logout',): stdout=Logging out of session [sid: 23, target: iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a, portal: 38.102.83.12,3260] Nov 24 14:39:19.197789 np0005533992.novalocal cinder-backup[122429]: Logout of [sid: 23, target: iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a, portal: 38.102.83.12,3260] successful. Nov 24 14:39:19.197789 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:19.198607 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a -p 38.102.83.12:3260 --op delete {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:19.209716 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7a270e29-bb4d-46d2-8e13-8481707a413a -p 38.102.83.12:3260 --op delete" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:19.209962 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[567cb7fe-482b-4bc6-99ec-563285dfee72]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:19.210362 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'delete'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:19.210979 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: held 0.159s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:39:19.211172 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] <== disconnect_volume: return (160ms) None {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:39:22.736038 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.manager [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Encryption key ID for volume 7a270e29-bb4d-46d2-8e13-8481707a413a already matches encryption key ID in backup f5cd5608-e8db-496c-be30-dfa768060eda. {{(pid=122429) _run_restore /opt/stack/cinder/cinder/backup/manager.py:874}} Nov 24 14:39:22.762287 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-25e42b65-bbaf-40c8-b3a4-3ddc835554a3 tempest-VolumesBackupsTest-1816644175 None] Finished restoring backup f5cd5608-e8db-496c-be30-dfa768060eda to volume 7a270e29-bb4d-46d2-8e13-8481707a413a. Nov 24 14:39:25.544533 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] Delete backup started, backup: f5cd5608-e8db-496c-be30-dfa768060eda. Nov 24 14:39:25.546189 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] Using swift URL https://38.102.83.12:8080/v1/AUTH_c642367dafaa45dd89634e7ec9822ef2 {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:286}} Nov 24 14:39:25.546450 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] Connect to https://38.102.83.12:8080/v1/AUTH_ in "per_user" mode {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:287}} Nov 24 14:39:25.546850 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] delete started, backup: f5cd5608-e8db-496c-be30-dfa768060eda, container: tempest-volumesbackupstest-backup-container-1248068680, prefix: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:844}} Nov 24 14:39:25.603528 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] generated object list: ['volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00001', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00002', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00003', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00004', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00005', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00006', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00007', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00008', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00009', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00010', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00011', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00012', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00013', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00014', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00015', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00016', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00017', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00018', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00019', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00020', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00021', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda_metadata', 'volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda_sha256file']. {{(pid=122429) _generate_object_names /opt/stack/cinder/cinder/backup/chunkeddriver.py:263}} Nov 24 14:39:25.634160 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00001 in container: tempest-volumesbackupstest-backup-container-1248068680. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:39:25.672604 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00002 in container: tempest-volumesbackupstest-backup-container-1248068680. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:39:25.712246 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00003 in container: tempest-volumesbackupstest-backup-container-1248068680. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:39:25.747876 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00004 in container: tempest-volumesbackupstest-backup-container-1248068680. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:39:25.789598 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00005 in container: tempest-volumesbackupstest-backup-container-1248068680. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:39:25.821634 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00006 in container: tempest-volumesbackupstest-backup-container-1248068680. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:39:25.841183 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00007 in container: tempest-volumesbackupstest-backup-container-1248068680. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:39:25.866203 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00008 in container: tempest-volumesbackupstest-backup-container-1248068680. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:39:25.890470 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00009 in container: tempest-volumesbackupstest-backup-container-1248068680. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:39:25.907434 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00010 in container: tempest-volumesbackupstest-backup-container-1248068680. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:39:25.928843 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00011 in container: tempest-volumesbackupstest-backup-container-1248068680. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:39:25.951602 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00012 in container: tempest-volumesbackupstest-backup-container-1248068680. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:39:25.983232 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00013 in container: tempest-volumesbackupstest-backup-container-1248068680. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:39:26.016617 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00014 in container: tempest-volumesbackupstest-backup-container-1248068680. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:39:26.041224 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00015 in container: tempest-volumesbackupstest-backup-container-1248068680. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:39:26.066614 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00016 in container: tempest-volumesbackupstest-backup-container-1248068680. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:39:26.089600 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00017 in container: tempest-volumesbackupstest-backup-container-1248068680. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:39:26.115327 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00018 in container: tempest-volumesbackupstest-backup-container-1248068680. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:39:26.141593 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00019 in container: tempest-volumesbackupstest-backup-container-1248068680. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:39:26.172356 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00020 in container: tempest-volumesbackupstest-backup-container-1248068680. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:39:26.209822 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda-00021 in container: tempest-volumesbackupstest-backup-container-1248068680. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:39:26.233161 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda_metadata in container: tempest-volumesbackupstest-backup-container-1248068680. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:39:26.257824 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_e6f9e8be-7196-4cc0-bea9-056b2e289daa/20251124193847/az_nova_backup_f5cd5608-e8db-496c-be30-dfa768060eda_sha256file in container: tempest-volumesbackupstest-backup-container-1248068680. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:39:26.257824 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] delete f5cd5608-e8db-496c-be30-dfa768060eda finished. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:870}} Nov 24 14:39:26.259754 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_db.sqlalchemy.engines [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=122429) _check_effective_sql_mode /usr/local/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:397}} Nov 24 14:39:26.273462 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.quota [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] Created reservations ['8bdb6b6c-5218-48dc-bcfa-ec79e154c07f', 'c256dff9-53ec-46ee-ab46-d19abaddf1da'] {{(pid=122429) reserve /opt/stack/cinder/cinder/quota.py:752}} Nov 24 14:39:26.286822 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-17e65f68-7f4d-48e9-b029-4955dbbedcff tempest-VolumesBackupsTest-1816644175 None] Delete backup finished, backup f5cd5608-e8db-496c-be30-dfa768060eda deleted. Nov 24 14:39:31.330177 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Create backup started, backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451 volume: 9c72607f-3da7-4df8-8808-5a7d5aebdece. Nov 24 14:39:31.334587 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Call Volume Manager to get_backup_device for Backup(availability_zone='nova',container='tempest-volumesbackupstest-backup-container-1555790666',created_at=2025-11-24T19:39:31Z,data_timestamp=2025-11-24T19:39:31Z,deleted=False,deleted_at=None,display_description='tempest-backup-description-2115847672',display_name='tempest-VolumesBackupsTest-Backup-2008692352',encryption_key_id=None,fail_reason=None,host='np0005533992.novalocal',id=5b2b93ef-5bc1-40f8-a171-c0aa7b733451,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='c642367dafaa45dd89634e7ec9822ef2',restore_volume_id=None,service='cinder.backup.drivers.swift.SwiftBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='0e48df3b35f84ee88136fd47ee531c1f',volume_id=9c72607f-3da7-4df8-8808-5a7d5aebdece) Nov 24 14:39:31.371236 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Using swift URL https://38.102.83.12:8080/v1/AUTH_c642367dafaa45dd89634e7ec9822ef2 {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:286}} Nov 24 14:39:31.371490 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Connect to https://38.102.83.12:8080/v1/AUTH_ in "per_user" mode {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:287}} Nov 24 14:39:31.372005 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '38.102.83.12', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:39:31.372951 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:31.380904 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.008s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:31.381144 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[a758fd79-7110-4db5-90f0-9860dd7c1333]: (4, ('InitiatorName=iqn.1994-05.com.redhat:5b9c7bcf3f78\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:31.383270 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[f0772d0d-2644-4902-9a7b-f94e617d1ea4]: [Errno 2] No such file or directory: '/dev/scini' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:39:31.383270 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:39:31.383270 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:39:31.383270 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:39:31.383270 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:31.383270 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:39:31.383270 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:39:31.383270 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:31.383270 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 57, in get_guid Nov 24 14:39:31.383270 np0005533992.novalocal cinder-backup[122429]: with open_scini_device() as fd: Nov 24 14:39:31.383270 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:31.383270 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/contextlib.py", line 137, in __enter__ Nov 24 14:39:31.383270 np0005533992.novalocal cinder-backup[122429]: return next(self.gen) Nov 24 14:39:31.383270 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^ Nov 24 14:39:31.383270 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 40, in open_scini_device Nov 24 14:39:31.383270 np0005533992.novalocal cinder-backup[122429]: fd = os.open(SCINI_DEVICE_PATH, os.O_RDWR) Nov 24 14:39:31.383270 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:31.383270 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: '/dev/scini' Nov 24 14:39:31.383895 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[f0772d0d-2644-4902-9a7b-f94e617d1ea4]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 57, in get_guid\n with open_scini_device() as fd:\n ^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/contextlib.py", line 137, in __enter__\n return next(self.gen)\n ^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 40, in open_scini_device\n fd = os.open(SCINI_DEVICE_PATH, os.O_RDWR)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory: \'/dev/scini\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:31.384478 np0005533992.novalocal cinder-backup[122429]: ERROR os_brick.initiator.connectors.scaleio [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Error querying sdc guid: [Errno 2] No such file or directory: FileNotFoundError: [Errno 2] No such file or directory Nov 24 14:39:31.384951 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.scaleio [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Unable to find SDC guid: Error querying sdc guid: [Errno 2] No such file or directory Nov 24 14:39:31.385763 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt -v / -n -o SOURCE {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:31.395657 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt -v / -n -o SOURCE" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:31.395892 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[ab1e23d7-f2b4-4f5b-a7f7-0e2b86c43fa7]: (4, ('/dev/vda2\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:31.396784 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/vda2 -s UUID -o value {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:31.409246 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/vda2 -s UUID -o value" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:31.409488 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[6a920bb8-2fe8-4dd8-a155-387af3098352]: (4, ('3fc68afb-d7e4-475c-9164-ea34f338d965\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:31.410538 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[ec4444ad-7dd9-4cfe-bb87-49001fe6e708]: (4, 'e2c36df9-c117-4156-b18b-d355565d8e8e') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:31.411041 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Running cmd (subprocess): nvme version {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:31.426222 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] 'nvme version' failed. Not Retrying. {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:39:31.426632 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.nvmeof [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] nvme not present on system {{(pid=122429) nvme_present /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/nvmeof.py:784}} Nov 24 14:39:31.428536 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=122429) find_dsc /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:132}} Nov 24 14:39:31.429684 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.lightos [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Current host hostNQN nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e and IP(s) are ['38.102.83.12', 'fe80::f816:3eff:fe17:2fa8', '192.168.122.1', '172.24.5.1', '2001:db8::2', 'fe80::64fc:4aff:fe7b:de4b', 'fe80::fc16:3eff:fec1:6190', 'fe80::e0ae:4fff:fe1e:508c'] Nov 24 14:39:31.430016 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=122429) get_connector_properties /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:109}} Nov 24 14:39:31.430247 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e dsc: {{(pid=122429) get_connector_properties /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:112}} Nov 24 14:39:31.430795 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] <== get_connector_properties: return (57ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '38.102.83.12', 'host': 'np0005533992.novalocal', 'multipath': False, 'enforce_multipath': False, 'initiator': 'iqn.1994-05.com.redhat:5b9c7bcf3f78', 'do_local_attach': False, 'uuid': '3fc68afb-d7e4-475c-9164-ea34f338d965', 'system uuid': 'e2c36df9-c117-4156-b18b-d355565d8e8e', 'nvme_native_multipath': False, 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e', 'found_dsc': '', 'host_ips': ['38.102.83.12', 'fe80::f816:3eff:fe17:2fa8', '192.168.122.1', '172.24.5.1', '2001:db8::2', 'fe80::64fc:4aff:fe7b:de4b', 'fe80::fc16:3eff:fec1:6190', 'fe80::e0ae:4fff:fe1e:508c']} {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:39:34.941253 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connector [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Factory for iscsi on None {{(pid=122429) factory /usr/local/lib/python3.12/site-packages/os_brick/initiator/connector.py:281}} Nov 24 14:39:34.942441 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] ==> connect_volume: call "{'self': , 'connection_properties': {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece', 'target_lun': 0, 'volume_id': '9c72607f-3da7-4df8-8808-5a7d5aebdece', 'auth_method': 'CHAP', 'auth_username': 'AKeA5emVLUh2ijS5wMpQ', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False}}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:39:34.942852 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Acquiring lock "connect_volume" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:39:34.943521 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:39:34.943847 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Connection properties {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece', 'target_lun': 0, 'volume_id': '9c72607f-3da7-4df8-8808-5a7d5aebdece', 'auth_method': 'CHAP', 'auth_username': 'AKeA5emVLUh2ijS5wMpQ', 'auth_password': '76R7STAYKJf9trMe', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False, 'enforce_multipath': False} {{(pid=122429) check_multipath /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:133}} Nov 24 14:39:34.945034 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd show status {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:34.952224 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] 'multipathd show status' failed. Not Retrying. {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:39:34.954345 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[26dc4b5a-0a8b-44b2-9f34-32f2339150db]: [Errno 2] No such file or directory: 'multipathd' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:39:34.954345 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:39:34.954345 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:39:34.954345 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:39:34.954345 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:34.954345 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:39:34.954345 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:39:34.954345 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:34.954345 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root Nov 24 14:39:34.954345 np0005533992.novalocal cinder-backup[122429]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Nov 24 14:39:34.954345 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:34.954345 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute Nov 24 14:39:34.954345 np0005533992.novalocal cinder-backup[122429]: return putils.execute(on_execute=on_execute, Nov 24 14:39:34.954345 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:34.954345 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute Nov 24 14:39:34.954345 np0005533992.novalocal cinder-backup[122429]: obj = subprocess.Popen(cmd, Nov 24 14:39:34.954345 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:34.954345 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__ Nov 24 14:39:34.954345 np0005533992.novalocal cinder-backup[122429]: self._execute_child(args, executable, preexec_fn, close_fds, Nov 24 14:39:34.954345 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child Nov 24 14:39:34.954345 np0005533992.novalocal cinder-backup[122429]: raise child_exception_type(errno_num, err_msg, err_filename) Nov 24 14:39:34.954345 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: 'multipathd' Nov 24 14:39:34.956057 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[26dc4b5a-0a8b-44b2-9f34-32f2339150db]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root\n return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute\n return putils.execute(on_execute=on_execute,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute\n obj = subprocess.Popen(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__\n self._execute_child(args, executable, preexec_fn, close_fds,\n File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child\n raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: \'multipathd\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:34.956951 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Acquiring lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:39:34.957606 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:39:34.958065 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Trying to connect to iSCSI portal 38.102.83.12:3260 Nov 24 14:39:34.959025 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece -p 38.102.83.12:3260 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:34.972738 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece -p 38.102.83.12:3260" returned: 21 in 0.013s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:34.972996 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[530556ec-7f8b-4a5d-9b09-fc6971d10056]: (4, ('', 'iscsiadm: No records found\n')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:34.973473 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] iscsiadm (): stdout= stderr=iscsiadm: No records found Nov 24 14:39:34.973473 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:34.974445 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece -p 38.102.83.12:3260 --interface default --op new {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:34.986520 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece -p 38.102.83.12:3260 --interface default --op new" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:34.986868 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[c15946ae-117d-42f6-b2a9-7cb43e532e10]: (4, ('New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 38.102.83.12,3260,-1 iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece] added\n', 'iscsiadm: Config file line 381 does not have value\niscsiadm: Config file line 381 does not have value\niscsiadm: Config file line 381 does not have value\n')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:34.987357 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--interface', 'default', '--op', 'new'): stdout=New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 38.102.83.12,3260,-1 iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece] added Nov 24 14:39:34.987357 np0005533992.novalocal cinder-backup[122429]: stderr=iscsiadm: Config file line 381 does not have value Nov 24 14:39:34.987357 np0005533992.novalocal cinder-backup[122429]: iscsiadm: Config file line 381 does not have value Nov 24 14:39:34.987357 np0005533992.novalocal cinder-backup[122429]: iscsiadm: Config file line 381 does not have value Nov 24 14:39:34.987357 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:34.987685 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Retrying to connect to iSCSI portal 38.102.83.12:3260 {{(pid=122429) _connect_to_iscsi_portal_unsafe /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1133}} Nov 24 14:39:34.988342 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Finished call to 'os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe' after 0.030(s), this was the 1st time calling it. {{(pid=122429) log_it /usr/local/lib/python3.12/site-packages/tenacity/after.py:44}} Nov 24 14:39:34.988744 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Retrying os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe in 1.0 seconds as it raised BrickException: An unknown exception occurred.. {{(pid=122429) log_it /usr/local/lib/python3.12/site-packages/tenacity/before_sleep.py:65}} Nov 24 14:39:35.991176 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Trying to connect to iSCSI portal 38.102.83.12:3260 Nov 24 14:39:35.995351 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece -p 38.102.83.12:3260 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:36.002420 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece -p 38.102.83.12:3260" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:36.002815 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[d86b309f-bd42-45e0-9d25-26368c1e9088]: (4, ('# BEGIN RECORD 6.2.1.11\nnode.name = iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece\nnode.tpgt = -1\nnode.startup = automatic\nnode.leading_login = No\niface.iscsi_ifacename = default\niface.net_ifacename = \niface.ipaddress = \niface.prefix_len = 0\niface.hwaddress = \niface.transport_name = tcp\niface.initiatorname = \niface.state = \niface.vlan_id = 0\niface.vlan_priority = 0\niface.vlan_state = \niface.iface_num = 0\niface.mtu = 0\niface.port = 0\niface.bootproto = \niface.subnet_mask = \niface.gateway = \niface.dhcp_alt_client_id_state = \niface.dhcp_alt_client_id = \niface.dhcp_dns = \niface.dhcp_learn_iqn = \niface.dhcp_req_vendor_id_state = \niface.dhcp_vendor_id_state = \niface.dhcp_vendor_id = \niface.dhcp_slp_da = \niface.fragmentation = \niface.gratuitous_arp = \niface.incoming_forwarding = \niface.tos_state = \niface.tos = 0\niface.ttl = 0\niface.delayed_ack = \niface.tcp_nagle = \niface.tcp_wsf_state = \niface.tcp_wsf = 0\niface.tcp_timer_scale = 0\niface.tcp_timestamp = \niface.redirect = \niface.def_task_mgmt_timeout = 0\niface.header_digest = \niface.data_digest = \niface.immediate_data = \niface.initial_r2t = \niface.data_seq_inorder = \niface.data_pdu_inorder = \niface.erl = 0\niface.max_receive_data_len = 0\niface.first_burst_len = 0\niface.max_outstanding_r2t = 0\niface.max_burst_len = 0\niface.chap_auth = \niface.bidi_chap = \niface.strict_login_compliance = \niface.discovery_auth = \niface.discovery_logout = \nnode.discovery_address = \nnode.discovery_port = 0\nnode.discovery_type = fw\nnode.session.initial_cmdsn = 0\nnode.session.initial_login_retry_max = 8\nnode.session.xmit_thread_priority = 0\nnode.session.cmds_max = 128\nnode.session.queue_depth = 32\nnode.session.nr_sessions = 1\nnode.session.auth.authmethod = None\nnode.session.auth.username = \nnode.session.auth.password = \nnode.session.auth.username_in = \nnode.session.auth.password_in = \nnode.session.auth.chap_algs = SHA3-256,SHA256\nnode.session.timeo.replacement_timeout = 120\nnode.session.err_timeo.abort_timeout = 15\nnode.session.err_timeo.lu_reset_timeout = 30\nnode.session.err_timeo.tgt_reset_timeout = 30\nnode.session.err_timeo.host_reset_timeout = 60\nnode.session.iscsi.FastAbort = Yes\nnode.session.iscsi.InitialR2T = No\nnode.session.iscsi.ImmediateData = Yes\nnode.session.iscsi.FirstBurstLength = 262144\nnode.session.iscsi.MaxBurstLength = 16776192\nnode.session.iscsi.DefaultTime2Retain = 0\nnode.session.iscsi.DefaultTime2Wait = 2\nnode.session.iscsi.MaxConnections = 1\nnode.session.iscsi.MaxOutstandingR2T = 1\nnode.session.iscsi.ERL = 0\nnode.session.scan = auto\nnode.session.reopen_max = 0\nnode.session.conn_reopen_log_freq = 1\nnode.conn[0].address = 38.102.83.12\nnode.conn[0].port = 3260\nnode.conn[0].startup = manual\nnode.conn[0].tcp.window_size = 524288\nnode.conn[0].tcp.type_of_service = 0\nnode.conn[0].timeo.logout_timeout = 15\nnode.conn[0].timeo.login_timeout = 15\nnode.conn[0].timeo.auth_timeout = 45\nnode.conn[0].timeo.noop_out_interval = 5\nnode.conn[0].timeo.noop_out_timeout = 5\nnode.conn[0].iscsi.MaxXmitDataSegmentLength = 0\nnode.conn[0].iscsi.MaxRecvDataSegmentLength = 262144\nnode.conn[0].iscsi.HeaderDigest = None\nnode.conn[0].iscsi.DataDigest = None\nnode.conn[0].iscsi.IFMarker = No\nnode.conn[0].iscsi.OFMarker = No\n# END RECORD\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] iscsiadm (): stdout=# BEGIN RECORD 6.2.1.11 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.name = iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.tpgt = -1 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.startup = automatic Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.leading_login = No Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.iscsi_ifacename = default Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.net_ifacename = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.ipaddress = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.prefix_len = 0 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.hwaddress = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.transport_name = tcp Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.initiatorname = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.state = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.vlan_id = 0 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.vlan_priority = 0 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.vlan_state = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.iface_num = 0 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.mtu = 0 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.port = 0 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.bootproto = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.subnet_mask = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.gateway = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_alt_client_id_state = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_alt_client_id = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_dns = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_learn_iqn = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_req_vendor_id_state = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_vendor_id_state = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_vendor_id = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_slp_da = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.fragmentation = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.gratuitous_arp = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.incoming_forwarding = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.tos_state = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.tos = 0 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.ttl = 0 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.delayed_ack = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.tcp_nagle = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.tcp_wsf_state = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.tcp_wsf = 0 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.tcp_timer_scale = 0 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.tcp_timestamp = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.redirect = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.def_task_mgmt_timeout = 0 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.header_digest = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.data_digest = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.immediate_data = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.initial_r2t = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.data_seq_inorder = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.data_pdu_inorder = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.erl = 0 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.max_receive_data_len = 0 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.first_burst_len = 0 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.max_outstanding_r2t = 0 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.max_burst_len = 0 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.chap_auth = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.bidi_chap = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.strict_login_compliance = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.discovery_auth = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: iface.discovery_logout = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.discovery_address = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.discovery_port = 0 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.discovery_type = fw Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.initial_cmdsn = 0 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.initial_login_retry_max = 8 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.xmit_thread_priority = 0 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.cmds_max = 128 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.queue_depth = 32 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.nr_sessions = 1 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.auth.authmethod = None Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.auth.username = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.auth.password = *** Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.auth.username_in = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.auth.password_in = Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.auth.chap_algs = SHA3-256,SHA256 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.timeo.replacement_timeout = 120 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.abort_timeout = 15 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.lu_reset_timeout = 30 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.tgt_reset_timeout = 30 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.host_reset_timeout = 60 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.FastAbort = Yes Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.InitialR2T = No Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.ImmediateData = Yes Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.FirstBurstLength = 262144 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxBurstLength = 16776192 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.DefaultTime2Retain = 0 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.DefaultTime2Wait = 2 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxConnections = 1 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxOutstandingR2T = 1 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.ERL = 0 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.scan = auto Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.reopen_max = 0 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.session.conn_reopen_log_freq = 1 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.conn[0].address = 38.102.83.12 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.conn[0].port = 3260 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.conn[0].startup = manual Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.conn[0].tcp.window_size = 524288 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.conn[0].tcp.type_of_service = 0 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.logout_timeout = 15 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.login_timeout = 15 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.auth_timeout = 45 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.noop_out_interval = 5 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.noop_out_timeout = 5 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.MaxXmitDataSegmentLength = 0 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144 Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.HeaderDigest = None Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.DataDigest = None Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.IFMarker = No Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.OFMarker = No Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: # END RECORD Nov 24 14:39:36.004469 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:36.007773 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece -p 38.102.83.12:3260 --op update -n node.session.scan -v manual {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:36.015879 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece -p 38.102.83.12:3260 --op update -n node.session.scan -v manual" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:36.016201 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[f9809057-f4c6-4b4e-884a-26ec54bfc8eb]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:36.016638 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.scan', '-v', 'manual'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:36.017499 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece -p 38.102.83.12:3260 --op update -n node.session.auth.authmethod -v CHAP {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:36.027148 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece -p 38.102.83.12:3260 --op update -n node.session.auth.authmethod -v CHAP" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:36.027454 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[d8ff8d1b-6157-47e3-b11d-4feee2c7cca0]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:36.027874 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.authmethod', '-v', 'CHAP'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:36.028737 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece -p 38.102.83.12:3260 --op update -n node.session.auth.username -v AKeA5emVLUh2ijS5wMpQ {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:36.039146 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece -p 38.102.83.12:3260 --op update -n node.session.auth.username -v AKeA5emVLUh2ijS5wMpQ" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:36.039564 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[45165f51-baa3-43f6-914e-f09a7c7c6479]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:36.039989 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.username', '-v', 'AKeA5emVLUh2ijS5wMpQ'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:36.040976 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece -p 38.102.83.12:3260 --op update -n node.session.auth.password -v *** {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:36.051686 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece -p 38.102.83.12:3260 --op update -n node.session.auth.password -v ***" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:36.051934 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[7afda859-48c1-4ee5-ab1f-4c8c95460da7]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:36.052418 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.password', '-v', '***'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:36.053238 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:36.064265 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:36.064609 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[da4d6acf-bb9a-4bf4-b990-79f0a1928187]: (4, ('tcp: [24] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-7d458d0d-eb5c-42b1-bc81-712b2309a02f (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:36.065022 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('-m', 'session'): stdout=tcp: [24] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-7d458d0d-eb5c-42b1-bc81-712b2309a02f (non-flash) Nov 24 14:39:36.065022 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:39:36.065263 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] iscsi session list stdout=tcp: [24] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-7d458d0d-eb5c-42b1-bc81-712b2309a02f (non-flash) Nov 24 14:39:36.065263 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:39:36.066108 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece -p 38.102.83.12:3260 --login {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:36.086357 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece -p 38.102.83.12:3260 --login" returned: 0 in 0.019s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:36.086654 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[e2608a28-e168-4663-b11f-5e812573dd65]: (4, ('Login to [iface: default, target: iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece, portal: 38.102.83.12,3260] successful.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:36.087113 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--login',): stdout=Login to [iface: default, target: iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece, portal: 38.102.83.12,3260] successful. Nov 24 14:39:36.087113 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:36.088005 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece -p 38.102.83.12:3260 --op update -n node.startup -v automatic {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:36.098920 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece -p 38.102.83.12:3260 --op update -n node.startup -v automatic" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:36.099153 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[fa18b6de-0aab-41cf-9f19-4614f74813b3]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:36.099541 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:36.100362 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:36.115583 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.015s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:36.115951 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[756ffdfa-87cc-49e9-aaf5-39a7d3bd79e2]: (4, ('tcp: [24] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-7d458d0d-eb5c-42b1-bc81-712b2309a02f (non-flash)\ntcp: [25] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:36.116442 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('-m', 'session'): stdout=tcp: [24] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-7d458d0d-eb5c-42b1-bc81-712b2309a02f (non-flash) Nov 24 14:39:36.116442 np0005533992.novalocal cinder-backup[122429]: tcp: [25] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece (non-flash) Nov 24 14:39:36.116442 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:39:36.116783 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] iscsi session list stdout=tcp: [24] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-7d458d0d-eb5c-42b1-bc81-712b2309a02f (non-flash) Nov 24 14:39:36.116783 np0005533992.novalocal cinder-backup[122429]: tcp: [25] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece (non-flash) Nov 24 14:39:36.116783 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:39:36.117254 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: held 1.160s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:39:36.117473 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Connected to 38.102.83.12:3260 {{(pid=122429) _connect_vol /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:659}} Nov 24 14:39:36.118268 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] HCTL ('3', '-', '-', 0) found on session 25 with lun 0 {{(pid=122429) get_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:788}} Nov 24 14:39:36.118521 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Scanning host 3 c: -, t: -, l: 0) {{(pid=122429) scan_iscsi /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:814}} Nov 24 14:39:36.119406 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/class/scsi_host/host3/scan {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:36.143849 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/class/scsi_host/host3/scan" returned: 0 in 0.024s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:36.144975 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[ffa64f93-7e57-4dd0-9412-eab9e97f34f0]: (4, ('- - 0', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:36.145733 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Searching for a device in session 25 and hctl ['3', '*', '*', 0] yield: None {{(pid=122429) device_name_by_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:808}} Nov 24 14:39:36.424440 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_db.sqlalchemy.engines [None req-02f864e7-9278-4f64-9b96-1347b85b7e07 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=122429) _check_effective_sql_mode /usr/local/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:397}} Nov 24 14:39:37.148404 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Searching for a device in session 25 and hctl ['3', '*', '*', 0] yield: sdb {{(pid=122429) device_name_by_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:808}} Nov 24 14:39:37.148404 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Connected to sdb using {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece', 'target_lun': 0, 'volume_id': '9c72607f-3da7-4df8-8808-5a7d5aebdece', 'auth_method': 'CHAP', 'auth_username': 'AKeA5emVLUh2ijS5wMpQ', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False} {{(pid=122429) _connect_vol /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:687}} Nov 24 14:39:37.149355 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" :: held 2.206s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:39:37.149524 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] <== connect_volume: return (2206ms) {'type': 'block', 'scsi_wwn': '360014053030e6af26764b349ad052a1b', 'path': '/dev/sdb'} {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:39:37.150142 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sdb {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:37.180467 np0005533992.novalocal sudo[162017]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sdb Nov 24 14:39:37.475288 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sdb" returned: 0 in 0.325s {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:37.487874 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] _create_container started, container: tempest-volumesbackupstest-backup-container-1555790666,backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451. {{(pid=122429) _create_container /opt/stack/cinder/cinder/backup/chunkeddriver.py:252}} Nov 24 14:39:37.564697 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] generate_object_name_prefix: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451 {{(pid=122429) _generate_object_name_prefix /opt/stack/cinder/cinder/backup/drivers/swift.py:433}} Nov 24 14:39:37.570103 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] starting backup of volume: 9c72607f-3da7-4df8-8808-5a7d5aebdece, volume size: 1073741824, object names prefix volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451, availability zone: nova {{(pid=122429) _prepare_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:362}} Nov 24 14:39:37.960431 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:39:38.095764 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:39:38.095939 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:39:38.283723 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00001: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:39:38.284188 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:39:38.712351 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:39:38.843782 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:39:38.843782 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:39:39.020658 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00002: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:39:39.020658 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:39:39.190879 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:39:39.323670 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:39:39.324409 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:39:39.483663 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00003: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:39:39.483663 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:39:39.861198 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:39:39.968278 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:39:39.968278 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:39:40.100681 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00004: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:39:40.100681 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:39:40.388157 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:39:40.491306 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:39:40.491306 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:39:40.640457 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00005: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:39:40.640633 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:39:41.232859 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:39:41.366590 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:39:41.366843 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:39:41.501045 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00006: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:39:41.501898 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:39:41.668485 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:39:41.796602 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:39:41.797019 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:39:41.941665 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00007: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:39:41.941665 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:39:42.332630 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:39:42.449293 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:39:42.449542 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:39:42.595476 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00008: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:39:42.595909 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:39:42.995059 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:39:43.122643 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:39:43.122643 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:39:43.267999 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00009: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:39:43.268655 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:39:43.483136 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:39:43.614777 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:39:43.615200 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:39:43.794763 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00010: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:39:43.795080 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:39:43.998624 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:39:44.108735 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:39:44.108735 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:39:44.216756 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00011: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:39:44.216897 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:39:44.651535 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:39:44.781179 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:39:44.781363 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:39:44.891234 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00012: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:39:44.891406 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:39:45.071741 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:39:45.201366 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:39:45.201669 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:39:45.341595 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00013: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:39:45.341901 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:39:45.762465 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:39:45.895473 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:39:45.895700 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:39:46.045473 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00014: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:39:46.045679 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:39:46.428724 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:39:46.428930 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:39:46.463457 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:39:46.596228 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:39:46.596427 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:39:46.746623 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00015: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:39:46.746623 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:39:46.969805 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:39:47.106692 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:39:47.106850 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:39:47.259706 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00016: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:39:47.259855 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:39:48.256035 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:39:48.387762 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:39:48.387762 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:39:48.512547 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00017: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:39:48.512851 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:39:48.958445 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:39:49.075904 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:39:49.076050 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:39:49.261266 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00018: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:39:49.261780 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:39:49.655584 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:39:49.789521 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:39:49.789820 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:39:49.973245 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00019: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:39:49.973404 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:39:50.234276 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:39:50.330427 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:39:50.331077 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:39:50.476538 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00020: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:39:50.476728 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:39:50.579331 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:39:50.654434 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Compressed 25165824 bytes of data to 24470 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:39:50.654740 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:39:50.770182 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00021: 77377273b0a4b61febdbf7bbf52b9db9 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:39:50.770484 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:39:50.775453 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Getting metadata type 'volume-base-metadata' {{(pid=122429) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Nov 24 14:39:50.788252 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=122429) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Nov 24 14:39:50.788683 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Getting metadata type 'volume-metadata' {{(pid=122429) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Nov 24 14:39:50.794778 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Completed fetching metadata type 'volume-metadata' {{(pid=122429) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:121}} Nov 24 14:39:50.795092 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Getting metadata type 'volume-glance-metadata' {{(pid=122429) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Nov 24 14:39:50.800331 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] No metadata type 'volume-glance-metadata' available {{(pid=122429) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Nov 24 14:39:50.800809 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] _write_sha256file started, container name: tempest-volumesbackupstest-backup-container-1555790666, sha256file filename: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451_sha256file. {{(pid=122429) _write_sha256file /opt/stack/cinder/cinder/backup/chunkeddriver.py:302}} Nov 24 14:39:50.888166 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] _write_sha256file finished. {{(pid=122429) _write_sha256file /opt/stack/cinder/cinder/backup/chunkeddriver.py:318}} Nov 24 14:39:50.889468 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] _write_metadata started, container name: tempest-volumesbackupstest-backup-container-1555790666, metadata filename: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451_metadata. {{(pid=122429) _write_metadata /opt/stack/cinder/cinder/backup/chunkeddriver.py:279}} Nov 24 14:39:50.924429 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] _write_metadata finished. Metadata: b'{\n "backup_description": "tempest-backup-description-2115847672",\n "backup_id": "5b2b93ef-5bc1-40f8-a171-c0aa7b733451",\n "backup_name": "tempest-VolumesBackupsTest-Backup-2008692352",\n "created_at": "2025-11-24 19:39:31+00:00",\n "objects": [\n {\n "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00001": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 0\n }\n },\n {\n "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00002": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 52428800\n }\n },\n {\n "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00003": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 104857600\n }\n },\n {\n "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00004": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 157286400\n }\n },\n {\n "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00005": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 209715200\n }\n },\n {\n "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00006": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 262144000\n }\n },\n {\n "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00007": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 314572800\n }\n },\n {\n "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00008": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 367001600\n }\n },\n {\n "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00009": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 419430400\n }\n },\n {\n "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00010": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 471859200\n }\n },\n {\n "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00011": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 524288000\n }\n },\n {\n "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00012": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 576716800\n }\n },\n {\n "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00013": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 629145600\n }\n },\n {\n "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00014": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 681574400\n }\n },\n {\n "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00015": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 734003200\n }\n },\n {\n "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00016": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 786432000\n }\n },\n {\n "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00017": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 838860800\n }\n },\n {\n "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00018": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 891289600\n }\n },\n {\n "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00019": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 943718400\n }\n },\n {\n "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00020": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 996147200\n }\n },\n {\n "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00021": {\n "compression": "zlib",\n "length": 25165824,\n "md5": "77377273b0a4b61febdbf7bbf52b9db9",\n "offset": 1048576000\n }\n }\n ],\n "parent_id": null,\n "version": "1.0.0",\n "volume_id": "9c72607f-3da7-4df8-8808-5a7d5aebdece",\n "volume_meta": "{\\"version\\": 2, \\"volume-base-metadata\\": {\\"id\\": \\"9c72607f-3da7-4df8-8808-5a7d5aebdece\\", \\"_name_id\\": null, \\"use_quota\\": true, \\"ec2_id\\": null, \\"user_id\\": \\"0e48df3b35f84ee88136fd47ee531c1f\\", \\"project_id\\": \\"c642367dafaa45dd89634e7ec9822ef2\\", \\"snapshot_id\\": null, \\"cluster_name\\": null, \\"host\\": \\"np0005533992.novalocal@lvmdriver-1#lvmdriver-1\\", \\"size\\": 1, \\"availability_zone\\": \\"nova\\", \\"status\\": \\"backing-up\\", \\"attach_status\\": \\"detached\\", \\"migration_status\\": null, \\"scheduled_at\\": \\"2025-11-24T19:39:30.000000\\", \\"launched_at\\": \\"2025-11-24T19:39:30.000000\\", \\"terminated_at\\": null, \\"display_name\\": \\"tempest-VolumesBackupsTest-Volume-470359589\\", \\"display_description\\": null, \\"provider_location\\": \\"38.102.83.12:3260,iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece 0\\", \\"provider_auth\\": \\"CHAP AKeA5emVLUh2ijS5wMpQ 76R7STAYKJf9trMe\\", \\"provider_geometry\\": null, \\"provider_id\\": null, \\"volume_type_id\\": \\"1f8776b6-c527-4643-876a-506285adfc66\\", \\"source_volid\\": null, \\"encryption_key_id\\": null, \\"consistencygroup_id\\": null, \\"group_id\\": null, \\"bootable\\": false, \\"multiattach\\": false, \\"replication_status\\": null, \\"replication_extended_status\\": null, \\"replication_driver_data\\": null, \\"previous_status\\": \\"available\\", \\"service_uuid\\": \\"6d0941fd-19ec-48ae-82b1-f25526175227\\", \\"shared_targets\\": false, \\"deleted_at\\": null, \\"deleted\\": false, \\"created_at\\": \\"2025-11-24T19:39:30.000000\\", \\"updated_at\\": \\"2025-11-24T19:39:33.000000\\"}, \\"volume-metadata\\": {\\"vol-meta1\\": \\"value1\\", \\"vol-meta2\\": \\"value2\\", \\"vol-meta3\\": \\"value3\\"}}"\n}'. {{(pid=122429) _write_metadata /opt/stack/cinder/cinder/backup/chunkeddriver.py:298}} Nov 24 14:39:50.928504 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] backup 5b2b93ef-5bc1-40f8-a171-c0aa7b733451 finished. {{(pid=122429) _finalize_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:465}} Nov 24 14:39:51.022291 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sdb {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:51.046190 np0005533992.novalocal sudo[162174]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sdb Nov 24 14:39:51.362182 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sdb" returned: 0 in 0.340s {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:51.363865 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] ==> disconnect_volume: call "{'args': (, {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece', 'target_lun': 0, 'volume_id': '9c72607f-3da7-4df8-8808-5a7d5aebdece', 'auth_method': 'CHAP', 'auth_username': 'AKeA5emVLUh2ijS5wMpQ', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False, 'enforce_multipath': False}, {'type': 'block', 'scsi_wwn': '360014053030e6af26764b349ad052a1b', 'path': '/dev/sdb'}), 'kwargs': {'force': True}}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:39:51.364145 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Acquiring lock "connect_volume" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:39:51.364927 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:39:51.366536 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m discoverydb -o show -P 1 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:51.374711 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m discoverydb -o show -P 1" returned: 0 in 0.008s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:51.374996 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[06488d0f-a731-4058-902b-e038a7523bd9]: (4, ('SENDTARGETS:\nNo targets found.\niSNS:\nNo targets found.\nSTATIC:\nTarget: iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece\n\tPortal: 38.102.83.12:3260,-1\n\t\tIface Name: default\nFIRMWARE:\nNo targets found.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:51.375491 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] iscsiadm ['-m', 'discoverydb', '-o', 'show', '-P', 1]: stdout=SENDTARGETS: Nov 24 14:39:51.375491 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:39:51.375491 np0005533992.novalocal cinder-backup[122429]: iSNS: Nov 24 14:39:51.375491 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:39:51.375491 np0005533992.novalocal cinder-backup[122429]: STATIC: Nov 24 14:39:51.375491 np0005533992.novalocal cinder-backup[122429]: Target: iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece Nov 24 14:39:51.375491 np0005533992.novalocal cinder-backup[122429]: Portal: 38.102.83.12:3260,-1 Nov 24 14:39:51.375491 np0005533992.novalocal cinder-backup[122429]: Iface Name: default Nov 24 14:39:51.375491 np0005533992.novalocal cinder-backup[122429]: FIRMWARE: Nov 24 14:39:51.375491 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:39:51.375491 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:39:51.375796 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Regex to get portals from discoverydb: ^SENDTARGETS: Nov 24 14:39:51.375796 np0005533992.novalocal cinder-backup[122429]: .*?^DiscoveryAddress: 38.102.83.12,3260.*? Nov 24 14:39:51.375796 np0005533992.novalocal cinder-backup[122429]: (.*?)^(?:DiscoveryAddress|iSNS):.* {{(pid=122429) _get_discoverydb_portals /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:392}} Nov 24 14:39:51.376257 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Getting connected devices for (ips,iqns,luns)=[('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece', 0)] {{(pid=122429) _get_connection_devices /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:855}} Nov 24 14:39:51.377229 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:51.387959 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:51.388305 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[33a5c08b-77d9-4c85-8287-8dc1e4af04a0]: (4, ('38.102.83.12:3260,4294967295 iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:51.389370 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:51.397825 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.008s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:51.398261 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[a70ebe99-a787-4156-9d51-884dad45ed22]: (4, ('tcp: [25] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:51.398726 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('-m', 'session'): stdout=tcp: [25] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece (non-flash) Nov 24 14:39:51.398726 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:39:51.399009 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] iscsi session list stdout=tcp: [25] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece (non-flash) Nov 24 14:39:51.399009 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:39:51.399986 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Resulting device map defaultdict(. at 0x7f5b90ef42c0>, {('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece'): ({'sdb'}, set())}) {{(pid=122429) _get_connection_devices /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:888}} Nov 24 14:39:51.400387 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Removing single pathed devices sdb {{(pid=122429) remove_connection /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:381}} Nov 24 14:39:51.401435 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd show status {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:51.406901 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] 'multipathd show status' failed. Not Retrying. {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:39:51.409185 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[05a0357e-5009-4470-805f-5d488bb6a5ee]: [Errno 2] No such file or directory: 'multipathd' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:39:51.409185 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:39:51.409185 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:39:51.409185 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:39:51.409185 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:51.409185 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:39:51.409185 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:39:51.409185 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:51.409185 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root Nov 24 14:39:51.409185 np0005533992.novalocal cinder-backup[122429]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Nov 24 14:39:51.409185 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:51.409185 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute Nov 24 14:39:51.409185 np0005533992.novalocal cinder-backup[122429]: return putils.execute(on_execute=on_execute, Nov 24 14:39:51.409185 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:51.409185 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute Nov 24 14:39:51.409185 np0005533992.novalocal cinder-backup[122429]: obj = subprocess.Popen(cmd, Nov 24 14:39:51.409185 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:51.409185 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__ Nov 24 14:39:51.409185 np0005533992.novalocal cinder-backup[122429]: self._execute_child(args, executable, preexec_fn, close_fds, Nov 24 14:39:51.409185 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child Nov 24 14:39:51.409185 np0005533992.novalocal cinder-backup[122429]: raise child_exception_type(errno_num, err_msg, err_filename) Nov 24 14:39:51.409185 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: 'multipathd' Nov 24 14:39:51.410926 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[05a0357e-5009-4470-805f-5d488bb6a5ee]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root\n return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute\n return putils.execute(on_execute=on_execute,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute\n obj = subprocess.Popen(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__\n self._execute_child(args, executable, preexec_fn, close_fds,\n File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child\n raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: \'multipathd\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:51.411723 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Flushing IO for device /dev/sdb {{(pid=122429) flush_device_io /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:441}} Nov 24 14:39:51.412733 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blockdev --flushbufs /dev/sdb {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:51.419897 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "blockdev --flushbufs /dev/sdb" returned: 0 in 0.007s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:51.420166 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[fbe7d9d2-fdb7-4077-a49d-20fc9d476b6c]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:51.420601 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Remove SCSI device /dev/sdb with /sys/block/sdb/device/delete {{(pid=122429) remove_scsi_device /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:151}} Nov 24 14:39:51.421512 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/block/sdb/device/delete {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:51.483156 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/block/sdb/device/delete" returned: 0 in 0.061s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:51.483515 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[bc97408a-19b9-46c4-96c1-1a365f288ef9]: (4, ('1', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:51.484188 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Checking to see if SCSI volumes sdb have been removed. {{(pid=122429) wait_for_volumes_removal /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:159}} Nov 24 14:39:51.484541 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] SCSI volumes sdb have been removed. {{(pid=122429) wait_for_volumes_removal /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:169}} Nov 24 14:39:51.485096 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Disconnecting from: [('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece')] {{(pid=122429) _disconnect_connection /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1201}} Nov 24 14:39:51.486410 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece -p 38.102.83.12:3260 --op update -n node.startup -v manual {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:51.498055 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece -p 38.102.83.12:3260 --op update -n node.startup -v manual" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:51.498292 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[66abe04d-6cc4-40d4-b315-cefa009374ce]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:51.498797 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:51.499915 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece -p 38.102.83.12:3260 --logout {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:51.530274 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece -p 38.102.83.12:3260 --logout" returned: 0 in 0.030s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:51.531033 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[53b59f8a-b9a7-4e85-ada2-aa1fcc14c543]: (4, ('Logging out of session [sid: 25, target: iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece, portal: 38.102.83.12,3260]\nLogout of [sid: 25, target: iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece, portal: 38.102.83.12,3260] successful.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:51.531477 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--logout',): stdout=Logging out of session [sid: 25, target: iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece, portal: 38.102.83.12,3260] Nov 24 14:39:51.531477 np0005533992.novalocal cinder-backup[122429]: Logout of [sid: 25, target: iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece, portal: 38.102.83.12,3260] successful. Nov 24 14:39:51.531477 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:51.532611 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece -p 38.102.83.12:3260 --op delete {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:51.541691 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece -p 38.102.83.12:3260 --op delete" returned: 0 in 0.009s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:51.542160 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[e4f2b3ab-c9f7-4722-863d-ff2e8bdfd1c9]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:51.542588 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'delete'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:39:51.543037 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: held 0.178s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:39:51.543320 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] <== disconnect_volume: return (179ms) None {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:39:55.071278 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-441894ed-d63a-4b10-8f94-bb57ccacf5bd tempest-VolumesBackupsTest-1816644175 None] Create backup finished. backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451. Nov 24 14:39:57.414645 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Restore backup started, backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451 volume: 96b9f01f-3dff-415d-9c78-d1149c899edb. Nov 24 14:39:57.419626 np0005533992.novalocal cinder-backup[122429]: DEBUG dbcounter [-] [122429] Writing DB stats cinder:SELECT=109,cinder:UPDATE=20,cinder:INSERT=17 {{(pid=122429) stat_writer /usr/local/lib/python3.12/site-packages/dbcounter.py:115}} Nov 24 14:39:57.429162 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Using swift URL https://38.102.83.12:8080/v1/AUTH_c642367dafaa45dd89634e7ec9822ef2 {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:286}} Nov 24 14:39:57.429602 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Connect to https://38.102.83.12:8080/v1/AUTH_ in "per_user" mode {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:287}} Nov 24 14:39:57.430238 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '38.102.83.12', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:39:57.434114 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:57.440252 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.008s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:57.440515 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[f18dfeff-53ec-47e3-8d4c-031b54e10ac5]: (4, ('InitiatorName=iqn.1994-05.com.redhat:5b9c7bcf3f78\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:57.443285 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[62d97e9b-a43b-4dc5-95ad-89ab4beefe8b]: [Errno 2] No such file or directory: '/dev/scini' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:39:57.443285 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:39:57.443285 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:39:57.443285 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:39:57.443285 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:57.443285 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:39:57.443285 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:39:57.443285 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:57.443285 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 57, in get_guid Nov 24 14:39:57.443285 np0005533992.novalocal cinder-backup[122429]: with open_scini_device() as fd: Nov 24 14:39:57.443285 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:57.443285 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/contextlib.py", line 137, in __enter__ Nov 24 14:39:57.443285 np0005533992.novalocal cinder-backup[122429]: return next(self.gen) Nov 24 14:39:57.443285 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^ Nov 24 14:39:57.443285 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 40, in open_scini_device Nov 24 14:39:57.443285 np0005533992.novalocal cinder-backup[122429]: fd = os.open(SCINI_DEVICE_PATH, os.O_RDWR) Nov 24 14:39:57.443285 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:39:57.443285 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: '/dev/scini' Nov 24 14:39:57.444944 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[62d97e9b-a43b-4dc5-95ad-89ab4beefe8b]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 57, in get_guid\n with open_scini_device() as fd:\n ^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/contextlib.py", line 137, in __enter__\n return next(self.gen)\n ^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 40, in open_scini_device\n fd = os.open(SCINI_DEVICE_PATH, os.O_RDWR)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory: \'/dev/scini\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:57.445109 np0005533992.novalocal cinder-backup[122429]: ERROR os_brick.initiator.connectors.scaleio [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Error querying sdc guid: [Errno 2] No such file or directory: FileNotFoundError: [Errno 2] No such file or directory Nov 24 14:39:57.445445 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.scaleio [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Unable to find SDC guid: Error querying sdc guid: [Errno 2] No such file or directory Nov 24 14:39:57.446480 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt -v / -n -o SOURCE {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:57.456129 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt -v / -n -o SOURCE" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:57.456374 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[de6f9d3f-6b45-47b2-b2fe-3e49d5b5fda2]: (4, ('/dev/vda2\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:57.457432 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/vda2 -s UUID -o value {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:57.467876 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/vda2 -s UUID -o value" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:39:57.468121 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[da6a34d6-9851-4f74-b5a8-c5aadc3645bd]: (4, ('3fc68afb-d7e4-475c-9164-ea34f338d965\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:57.469262 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[1a0089e6-33e1-426d-bc93-5827d91a51cb]: (4, 'e2c36df9-c117-4156-b18b-d355565d8e8e') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:39:57.470047 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Running cmd (subprocess): nvme version {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:39:57.484643 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] 'nvme version' failed. Not Retrying. {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:39:57.485057 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.nvmeof [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] nvme not present on system {{(pid=122429) nvme_present /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/nvmeof.py:784}} Nov 24 14:39:57.487413 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=122429) find_dsc /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:132}} Nov 24 14:39:57.488371 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.lightos [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Current host hostNQN nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e and IP(s) are ['38.102.83.12', 'fe80::f816:3eff:fe17:2fa8', '192.168.122.1', '172.24.5.1', '2001:db8::2', 'fe80::64fc:4aff:fe7b:de4b'] Nov 24 14:39:57.488678 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=122429) get_connector_properties /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:109}} Nov 24 14:39:57.488972 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e dsc: {{(pid=122429) get_connector_properties /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:112}} Nov 24 14:39:57.489534 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] <== get_connector_properties: return (58ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '38.102.83.12', 'host': 'np0005533992.novalocal', 'multipath': False, 'enforce_multipath': False, 'initiator': 'iqn.1994-05.com.redhat:5b9c7bcf3f78', 'do_local_attach': False, 'uuid': '3fc68afb-d7e4-475c-9164-ea34f338d965', 'system uuid': 'e2c36df9-c117-4156-b18b-d355565d8e8e', 'nvme_native_multipath': False, 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e', 'found_dsc': '', 'host_ips': ['38.102.83.12', 'fe80::f816:3eff:fe17:2fa8', '192.168.122.1', '172.24.5.1', '2001:db8::2', 'fe80::64fc:4aff:fe7b:de4b']} {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:40:01.505628 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connector [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Factory for iscsi on None {{(pid=122429) factory /usr/local/lib/python3.12/site-packages/os_brick/initiator/connector.py:281}} Nov 24 14:40:01.506991 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] ==> connect_volume: call "{'self': , 'connection_properties': {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb', 'target_lun': 0, 'volume_id': '96b9f01f-3dff-415d-9c78-d1149c899edb', 'auth_method': 'CHAP', 'auth_username': 'KD5tYUQFPJFsRd32qJcy', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False}}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:40:01.508602 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Acquiring lock "connect_volume" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:40:01.508744 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:40:01.509236 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Connection properties {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb', 'target_lun': 0, 'volume_id': '96b9f01f-3dff-415d-9c78-d1149c899edb', 'auth_method': 'CHAP', 'auth_username': 'KD5tYUQFPJFsRd32qJcy', 'auth_password': 'Qj4QaiUHqoAAJSKB', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False, 'enforce_multipath': False} {{(pid=122429) check_multipath /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:133}} Nov 24 14:40:01.510958 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd show status {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:40:01.518131 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] 'multipathd show status' failed. Not Retrying. {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:40:01.520085 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[13c36910-edd1-4618-8cb2-1b866faa9b65]: [Errno 2] No such file or directory: 'multipathd' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:40:01.520085 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:40:01.520085 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:40:01.520085 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:40:01.520085 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:40:01.520085 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:40:01.520085 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:40:01.520085 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:40:01.520085 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root Nov 24 14:40:01.520085 np0005533992.novalocal cinder-backup[122429]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Nov 24 14:40:01.520085 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:40:01.520085 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute Nov 24 14:40:01.520085 np0005533992.novalocal cinder-backup[122429]: return putils.execute(on_execute=on_execute, Nov 24 14:40:01.520085 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:40:01.520085 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute Nov 24 14:40:01.520085 np0005533992.novalocal cinder-backup[122429]: obj = subprocess.Popen(cmd, Nov 24 14:40:01.520085 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:40:01.520085 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__ Nov 24 14:40:01.520085 np0005533992.novalocal cinder-backup[122429]: self._execute_child(args, executable, preexec_fn, close_fds, Nov 24 14:40:01.520085 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child Nov 24 14:40:01.520085 np0005533992.novalocal cinder-backup[122429]: raise child_exception_type(errno_num, err_msg, err_filename) Nov 24 14:40:01.520085 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: 'multipathd' Nov 24 14:40:01.523052 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[13c36910-edd1-4618-8cb2-1b866faa9b65]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root\n return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute\n return putils.execute(on_execute=on_execute,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute\n obj = subprocess.Popen(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__\n self._execute_child(args, executable, preexec_fn, close_fds,\n File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child\n raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: \'multipathd\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:40:01.523052 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Acquiring lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:40:01.524546 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: waited 0.002s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:40:01.525331 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Trying to connect to iSCSI portal 38.102.83.12:3260 Nov 24 14:40:01.526532 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb -p 38.102.83.12:3260 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:40:01.539928 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb -p 38.102.83.12:3260" returned: 21 in 0.013s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:40:01.542158 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[55a52186-4c5d-491e-b1e3-217d5087f651]: (4, ('', 'iscsiadm: No records found\n')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:40:01.542158 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] iscsiadm (): stdout= stderr=iscsiadm: No records found Nov 24 14:40:01.542158 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:40:01.542981 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb -p 38.102.83.12:3260 --interface default --op new {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:40:01.559437 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb -p 38.102.83.12:3260 --interface default --op new" returned: 0 in 0.013s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:40:01.559437 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[4d39c963-0865-41a1-8158-9c7315358de2]: (4, ('New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 38.102.83.12,3260,-1 iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb] added\n', 'iscsiadm: Config file line 381 does not have value\niscsiadm: Config file line 381 does not have value\niscsiadm: Config file line 381 does not have value\n')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:40:01.559437 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--interface', 'default', '--op', 'new'): stdout=New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 38.102.83.12,3260,-1 iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb] added Nov 24 14:40:01.559437 np0005533992.novalocal cinder-backup[122429]: stderr=iscsiadm: Config file line 381 does not have value Nov 24 14:40:01.559437 np0005533992.novalocal cinder-backup[122429]: iscsiadm: Config file line 381 does not have value Nov 24 14:40:01.559437 np0005533992.novalocal cinder-backup[122429]: iscsiadm: Config file line 381 does not have value Nov 24 14:40:01.559437 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:40:01.559437 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Retrying to connect to iSCSI portal 38.102.83.12:3260 {{(pid=122429) _connect_to_iscsi_portal_unsafe /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1133}} Nov 24 14:40:01.559437 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Finished call to 'os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe' after 0.033(s), this was the 1st time calling it. {{(pid=122429) log_it /usr/local/lib/python3.12/site-packages/tenacity/after.py:44}} Nov 24 14:40:01.559437 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Retrying os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe in 1.0 seconds as it raised BrickException: An unknown exception occurred.. {{(pid=122429) log_it /usr/local/lib/python3.12/site-packages/tenacity/before_sleep.py:65}} Nov 24 14:40:02.561646 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Trying to connect to iSCSI portal 38.102.83.12:3260 Nov 24 14:40:02.563434 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb -p 38.102.83.12:3260 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:40:02.575781 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb -p 38.102.83.12:3260" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:40:02.576125 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[592aeed7-554a-4a4e-ba06-cfcd7de18f61]: (4, ('# BEGIN RECORD 6.2.1.11\nnode.name = iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb\nnode.tpgt = -1\nnode.startup = automatic\nnode.leading_login = No\niface.iscsi_ifacename = default\niface.net_ifacename = \niface.ipaddress = \niface.prefix_len = 0\niface.hwaddress = \niface.transport_name = tcp\niface.initiatorname = \niface.state = \niface.vlan_id = 0\niface.vlan_priority = 0\niface.vlan_state = \niface.iface_num = 0\niface.mtu = 0\niface.port = 0\niface.bootproto = \niface.subnet_mask = \niface.gateway = \niface.dhcp_alt_client_id_state = \niface.dhcp_alt_client_id = \niface.dhcp_dns = \niface.dhcp_learn_iqn = \niface.dhcp_req_vendor_id_state = \niface.dhcp_vendor_id_state = \niface.dhcp_vendor_id = \niface.dhcp_slp_da = \niface.fragmentation = \niface.gratuitous_arp = \niface.incoming_forwarding = \niface.tos_state = \niface.tos = 0\niface.ttl = 0\niface.delayed_ack = \niface.tcp_nagle = \niface.tcp_wsf_state = \niface.tcp_wsf = 0\niface.tcp_timer_scale = 0\niface.tcp_timestamp = \niface.redirect = \niface.def_task_mgmt_timeout = 0\niface.header_digest = \niface.data_digest = \niface.immediate_data = \niface.initial_r2t = \niface.data_seq_inorder = \niface.data_pdu_inorder = \niface.erl = 0\niface.max_receive_data_len = 0\niface.first_burst_len = 0\niface.max_outstanding_r2t = 0\niface.max_burst_len = 0\niface.chap_auth = \niface.bidi_chap = \niface.strict_login_compliance = \niface.discovery_auth = \niface.discovery_logout = \nnode.discovery_address = \nnode.discovery_port = 0\nnode.discovery_type = fw\nnode.session.initial_cmdsn = 0\nnode.session.initial_login_retry_max = 8\nnode.session.xmit_thread_priority = 0\nnode.session.cmds_max = 128\nnode.session.queue_depth = 32\nnode.session.nr_sessions = 1\nnode.session.auth.authmethod = None\nnode.session.auth.username = \nnode.session.auth.password = \nnode.session.auth.username_in = \nnode.session.auth.password_in = \nnode.session.auth.chap_algs = SHA3-256,SHA256\nnode.session.timeo.replacement_timeout = 120\nnode.session.err_timeo.abort_timeout = 15\nnode.session.err_timeo.lu_reset_timeout = 30\nnode.session.err_timeo.tgt_reset_timeout = 30\nnode.session.err_timeo.host_reset_timeout = 60\nnode.session.iscsi.FastAbort = Yes\nnode.session.iscsi.InitialR2T = No\nnode.session.iscsi.ImmediateData = Yes\nnode.session.iscsi.FirstBurstLength = 262144\nnode.session.iscsi.MaxBurstLength = 16776192\nnode.session.iscsi.DefaultTime2Retain = 0\nnode.session.iscsi.DefaultTime2Wait = 2\nnode.session.iscsi.MaxConnections = 1\nnode.session.iscsi.MaxOutstandingR2T = 1\nnode.session.iscsi.ERL = 0\nnode.session.scan = auto\nnode.session.reopen_max = 0\nnode.session.conn_reopen_log_freq = 1\nnode.conn[0].address = 38.102.83.12\nnode.conn[0].port = 3260\nnode.conn[0].startup = manual\nnode.conn[0].tcp.window_size = 524288\nnode.conn[0].tcp.type_of_service = 0\nnode.conn[0].timeo.logout_timeout = 15\nnode.conn[0].timeo.login_timeout = 15\nnode.conn[0].timeo.auth_timeout = 45\nnode.conn[0].timeo.noop_out_interval = 5\nnode.conn[0].timeo.noop_out_timeout = 5\nnode.conn[0].iscsi.MaxXmitDataSegmentLength = 0\nnode.conn[0].iscsi.MaxRecvDataSegmentLength = 262144\nnode.conn[0].iscsi.HeaderDigest = None\nnode.conn[0].iscsi.DataDigest = None\nnode.conn[0].iscsi.IFMarker = No\nnode.conn[0].iscsi.OFMarker = No\n# END RECORD\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] iscsiadm (): stdout=# BEGIN RECORD 6.2.1.11 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.name = iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.tpgt = -1 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.startup = automatic Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.leading_login = No Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.iscsi_ifacename = default Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.net_ifacename = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.ipaddress = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.prefix_len = 0 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.hwaddress = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.transport_name = tcp Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.initiatorname = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.state = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.vlan_id = 0 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.vlan_priority = 0 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.vlan_state = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.iface_num = 0 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.mtu = 0 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.port = 0 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.bootproto = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.subnet_mask = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.gateway = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_alt_client_id_state = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_alt_client_id = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_dns = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_learn_iqn = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_req_vendor_id_state = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_vendor_id_state = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_vendor_id = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_slp_da = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.fragmentation = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.gratuitous_arp = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.incoming_forwarding = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.tos_state = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.tos = 0 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.ttl = 0 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.delayed_ack = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.tcp_nagle = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.tcp_wsf_state = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.tcp_wsf = 0 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.tcp_timer_scale = 0 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.tcp_timestamp = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.redirect = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.def_task_mgmt_timeout = 0 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.header_digest = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.data_digest = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.immediate_data = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.initial_r2t = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.data_seq_inorder = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.data_pdu_inorder = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.erl = 0 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.max_receive_data_len = 0 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.first_burst_len = 0 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.max_outstanding_r2t = 0 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.max_burst_len = 0 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.chap_auth = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.bidi_chap = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.strict_login_compliance = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.discovery_auth = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: iface.discovery_logout = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.discovery_address = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.discovery_port = 0 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.discovery_type = fw Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.initial_cmdsn = 0 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.initial_login_retry_max = 8 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.xmit_thread_priority = 0 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.cmds_max = 128 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.queue_depth = 32 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.nr_sessions = 1 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.auth.authmethod = None Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.auth.username = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.auth.password = *** Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.auth.username_in = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.auth.password_in = Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.auth.chap_algs = SHA3-256,SHA256 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.timeo.replacement_timeout = 120 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.abort_timeout = 15 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.lu_reset_timeout = 30 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.tgt_reset_timeout = 30 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.host_reset_timeout = 60 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.FastAbort = Yes Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.InitialR2T = No Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.ImmediateData = Yes Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.FirstBurstLength = 262144 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxBurstLength = 16776192 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.DefaultTime2Retain = 0 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.DefaultTime2Wait = 2 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxConnections = 1 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxOutstandingR2T = 1 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.ERL = 0 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.scan = auto Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.reopen_max = 0 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.session.conn_reopen_log_freq = 1 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.conn[0].address = 38.102.83.12 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.conn[0].port = 3260 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.conn[0].startup = manual Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.conn[0].tcp.window_size = 524288 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.conn[0].tcp.type_of_service = 0 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.logout_timeout = 15 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.login_timeout = 15 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.auth_timeout = 45 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.noop_out_interval = 5 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.noop_out_timeout = 5 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.MaxXmitDataSegmentLength = 0 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144 Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.HeaderDigest = None Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.DataDigest = None Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.IFMarker = No Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.OFMarker = No Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: # END RECORD Nov 24 14:40:02.578705 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:40:02.581645 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb -p 38.102.83.12:3260 --op update -n node.session.scan -v manual {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:40:02.594681 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb -p 38.102.83.12:3260 --op update -n node.session.scan -v manual" returned: 0 in 0.014s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:40:02.595266 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[2dff2936-df08-4ca1-bb8e-16b2a79aa9f7]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:40:02.595837 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.scan', '-v', 'manual'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:40:02.597111 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb -p 38.102.83.12:3260 --op update -n node.session.auth.authmethod -v CHAP {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:40:02.611981 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb -p 38.102.83.12:3260 --op update -n node.session.auth.authmethod -v CHAP" returned: 0 in 0.014s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:40:02.611981 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[4bfc9a35-7ca0-4dcb-902e-beeb5d92c76c]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:40:02.612327 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.authmethod', '-v', 'CHAP'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:40:02.613453 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb -p 38.102.83.12:3260 --op update -n node.session.auth.username -v KD5tYUQFPJFsRd32qJcy {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:40:02.625103 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb -p 38.102.83.12:3260 --op update -n node.session.auth.username -v KD5tYUQFPJFsRd32qJcy" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:40:02.625523 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[cdc22e51-3374-4e1d-98e4-2b846f5db5eb]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:40:02.626023 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.username', '-v', 'KD5tYUQFPJFsRd32qJcy'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:40:02.628984 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb -p 38.102.83.12:3260 --op update -n node.session.auth.password -v *** {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:40:02.638505 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb -p 38.102.83.12:3260 --op update -n node.session.auth.password -v ***" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:40:02.638798 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[48a0c1d1-60b0-4103-a42a-d2414bd808de]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:40:02.639475 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.password', '-v', '***'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:40:02.640803 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:40:02.651180 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 21 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:40:02.651510 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[77d5c065-168f-4909-8fa2-73bcec2ac8bd]: (4, ('', 'iscsiadm: No active sessions.\n')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:40:02.652133 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('-m', 'session'): stdout= stderr=iscsiadm: No active sessions. Nov 24 14:40:02.652133 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:40:02.652349 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] iscsi session list stdout= stderr=iscsiadm: No active sessions. Nov 24 14:40:02.652349 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:40:02.652639 np0005533992.novalocal cinder-backup[122429]: WARNING os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions. Nov 24 14:40:02.652639 np0005533992.novalocal cinder-backup[122429]: Nov 24 14:40:02.653752 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb -p 38.102.83.12:3260 --login {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:40:02.671342 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb -p 38.102.83.12:3260 --login" returned: 0 in 0.017s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:40:02.671646 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[12de355c-3816-4490-8304-321dd0e1173f]: (4, ('Login to [iface: default, target: iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb, portal: 38.102.83.12,3260] successful.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:40:02.672203 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--login',): stdout=Login to [iface: default, target: iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb, portal: 38.102.83.12,3260] successful. Nov 24 14:40:02.672203 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:40:02.673398 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb -p 38.102.83.12:3260 --op update -n node.startup -v automatic {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:40:02.685029 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb -p 38.102.83.12:3260 --op update -n node.startup -v automatic" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:40:02.685293 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[a0fa0698-9b4e-4e41-ae89-d9d9e2f5d1af]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:40:02.685872 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:40:02.686914 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:40:02.698457 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:40:02.698752 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[2dce1a53-f30b-4daf-9a06-009f031d3948]: (4, ('tcp: [26] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:40:02.699280 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('-m', 'session'): stdout=tcp: [26] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb (non-flash) Nov 24 14:40:02.699280 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:40:02.699542 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] iscsi session list stdout=tcp: [26] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb (non-flash) Nov 24 14:40:02.699542 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:40:02.700260 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: held 1.176s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:40:02.700575 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Connected to 38.102.83.12:3260 {{(pid=122429) _connect_vol /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:659}} Nov 24 14:40:02.701352 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] HCTL ('2', '-', '-', 0) found on session 26 with lun 0 {{(pid=122429) get_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:788}} Nov 24 14:40:02.701649 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Scanning host 2 c: -, t: -, l: 0) {{(pid=122429) scan_iscsi /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:814}} Nov 24 14:40:02.704458 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/class/scsi_host/host2/scan {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:40:02.727861 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/class/scsi_host/host2/scan" returned: 0 in 0.025s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:40:02.729237 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[01db850f-bd43-4813-9df2-0951d4ff4a58]: (4, ('- - 0', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:40:02.730013 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Searching for a device in session 26 and hctl ['2', '*', '*', 0] yield: None {{(pid=122429) device_name_by_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:808}} Nov 24 14:40:03.732377 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Searching for a device in session 26 and hctl ['2', '*', '*', 0] yield: sda {{(pid=122429) device_name_by_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:808}} Nov 24 14:40:03.732932 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Connected to sda using {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb', 'target_lun': 0, 'volume_id': '96b9f01f-3dff-415d-9c78-d1149c899edb', 'auth_method': 'CHAP', 'auth_username': 'KD5tYUQFPJFsRd32qJcy', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False} {{(pid=122429) _connect_vol /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:687}} Nov 24 14:40:03.733636 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" :: held 2.225s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:40:03.733837 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] <== connect_volume: return (2226ms) {'type': 'block', 'scsi_wwn': '36001405a6c257ac644040be9d009ea4d', 'path': '/dev/sda'} {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:40:03.734177 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sda {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:40:03.759730 np0005533992.novalocal sudo[162417]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sda Nov 24 14:40:04.096255 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sda" returned: 0 in 0.362s {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:40:04.096877 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] starting restore of backup volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451 container: tempest-volumesbackupstest-backup-container-1555790666, to new volume 96b9f01f-3dff-415d-9c78-d1149c899edb, backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451. {{(pid=122429) restore /opt/stack/cinder/cinder/backup/chunkeddriver.py:783}} Nov 24 14:40:04.097213 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] _read_metadata started, container name: tempest-volumesbackupstest-backup-container-1555790666, metadata filename: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451_metadata. {{(pid=122429) _read_metadata /opt/stack/cinder/cinder/backup/chunkeddriver.py:323}} Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] _read_metadata finished. Metadata: { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "backup_description": "tempest-backup-description-2115847672", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "backup_id": "5b2b93ef-5bc1-40f8-a171-c0aa7b733451", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "backup_name": "tempest-VolumesBackupsTest-Backup-2008692352", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "created_at": "2025-11-24 19:39:31+00:00", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "objects": [ Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00001": { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "offset": 0 Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00002": { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "offset": 52428800 Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00003": { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "offset": 104857600 Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00004": { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "offset": 157286400 Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00005": { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "offset": 209715200 Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00006": { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "offset": 262144000 Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00007": { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "offset": 314572800 Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00008": { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "offset": 367001600 Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00009": { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "offset": 419430400 Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00010": { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "offset": 471859200 Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00011": { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "offset": 524288000 Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00012": { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "offset": 576716800 Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00013": { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "offset": 629145600 Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00014": { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "offset": 681574400 Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00015": { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "offset": 734003200 Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00016": { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "offset": 786432000 Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00017": { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "offset": 838860800 Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00018": { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "offset": 891289600 Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00019": { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "offset": 943718400 Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00020": { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "offset": 996147200 Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00021": { Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "length": 25165824, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "md5": "77377273b0a4b61febdbf7bbf52b9db9", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "offset": 1048576000 Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: ], Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "parent_id": null, Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "version": "1.0.0", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "volume_id": "9c72607f-3da7-4df8-8808-5a7d5aebdece", Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: "volume_meta": "{\"version\": 2, \"volume-base-metadata\": {\"id\": \"9c72607f-3da7-4df8-8808-5a7d5aebdece\", \"_name_id\": null, \"use_quota\": true, \"ec2_id\": null, \"user_id\": \"0e48df3b35f84ee88136fd47ee531c1f\", \"project_id\": \"c642367dafaa45dd89634e7ec9822ef2\", \"snapshot_id\": null, \"cluster_name\": null, \"host\": \"np0005533992.novalocal@lvmdriver-1#lvmdriver-1\", \"size\": 1, \"availability_zone\": \"nova\", \"status\": \"backing-up\", \"attach_status\": \"detached\", \"migration_status\": null, \"scheduled_at\": \"2025-11-24T19:39:30.000000\", \"launched_at\": \"2025-11-24T19:39:30.000000\", \"terminated_at\": null, \"display_name\": \"tempest-VolumesBackupsTest-Volume-470359589\", \"display_description\": null, \"provider_location\": \"38.102.83.12:3260,iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece 0\", \"provider_auth\": \"CHAP AKeA5emVLUh2ijS5wMpQ 76R7STAYKJf9trMe\", \"provider_geometry\": null, \"provider_id\": null, \"volume_type_id\": \"1f8776b6-c527-4643-876a-506285adfc66\", \"source_volid\": null, \"encryption_key_id\": null, \"consistencygroup_id\": null, \"group_id\": null, \"bootable\": false, \"multiattach\": false, \"replication_status\": null, \"replication_extended_status\": null, \"replication_driver_data\": null, \"previous_status\": \"available\", \"service_uuid\": \"6d0941fd-19ec-48ae-82b1-f25526175227\", \"shared_targets\": false, \"deleted_at\": null, \"deleted\": false, \"created_at\": \"2025-11-24T19:39:30.000000\", \"updated_at\": \"2025-11-24T19:39:33.000000\"}, \"volume-metadata\": {\"vol-meta1\": \"value1\", \"vol-meta2\": \"value2\", \"vol-meta3\": \"value3\"}}" Nov 24 14:40:04.125874 np0005533992.novalocal cinder-backup[122429]: }. {{(pid=122429) _read_metadata /opt/stack/cinder/cinder/backup/chunkeddriver.py:330}} Nov 24 14:40:04.130273 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Restoring backup version 1.0.0 {{(pid=122429) restore /opt/stack/cinder/cinder/backup/chunkeddriver.py:796}} Nov 24 14:40:04.130273 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] _read_metadata started, container name: tempest-volumesbackupstest-backup-container-1555790666, metadata filename: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451_metadata. {{(pid=122429) _read_metadata /opt/stack/cinder/cinder/backup/chunkeddriver.py:323}} Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] _read_metadata finished. Metadata: { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "backup_description": "tempest-backup-description-2115847672", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "backup_id": "5b2b93ef-5bc1-40f8-a171-c0aa7b733451", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "backup_name": "tempest-VolumesBackupsTest-Backup-2008692352", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "created_at": "2025-11-24 19:39:31+00:00", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "objects": [ Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00001": { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "offset": 0 Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00002": { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "offset": 52428800 Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00003": { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "offset": 104857600 Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00004": { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "offset": 157286400 Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00005": { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "offset": 209715200 Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00006": { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "offset": 262144000 Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00007": { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "offset": 314572800 Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00008": { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "offset": 367001600 Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00009": { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "offset": 419430400 Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00010": { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "offset": 471859200 Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00011": { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "offset": 524288000 Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00012": { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "offset": 576716800 Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00013": { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "offset": 629145600 Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00014": { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "offset": 681574400 Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00015": { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "offset": 734003200 Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00016": { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "offset": 786432000 Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00017": { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "offset": 838860800 Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00018": { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "offset": 891289600 Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00019": { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "offset": 943718400 Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00020": { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "length": 52428800, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "md5": "25e317773f308e446cc84c503a6d1f85", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "offset": 996147200 Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: }, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00021": { Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "compression": "zlib", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "length": 25165824, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "md5": "77377273b0a4b61febdbf7bbf52b9db9", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "offset": 1048576000 Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: } Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: ], Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "parent_id": null, Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "version": "1.0.0", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "volume_id": "9c72607f-3da7-4df8-8808-5a7d5aebdece", Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: "volume_meta": "{\"version\": 2, \"volume-base-metadata\": {\"id\": \"9c72607f-3da7-4df8-8808-5a7d5aebdece\", \"_name_id\": null, \"use_quota\": true, \"ec2_id\": null, \"user_id\": \"0e48df3b35f84ee88136fd47ee531c1f\", \"project_id\": \"c642367dafaa45dd89634e7ec9822ef2\", \"snapshot_id\": null, \"cluster_name\": null, \"host\": \"np0005533992.novalocal@lvmdriver-1#lvmdriver-1\", \"size\": 1, \"availability_zone\": \"nova\", \"status\": \"backing-up\", \"attach_status\": \"detached\", \"migration_status\": null, \"scheduled_at\": \"2025-11-24T19:39:30.000000\", \"launched_at\": \"2025-11-24T19:39:30.000000\", \"terminated_at\": null, \"display_name\": \"tempest-VolumesBackupsTest-Volume-470359589\", \"display_description\": null, \"provider_location\": \"38.102.83.12:3260,iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece iqn.2010-10.org.openstack:volume-9c72607f-3da7-4df8-8808-5a7d5aebdece 0\", \"provider_auth\": \"CHAP AKeA5emVLUh2ijS5wMpQ 76R7STAYKJf9trMe\", \"provider_geometry\": null, \"provider_id\": null, \"volume_type_id\": \"1f8776b6-c527-4643-876a-506285adfc66\", \"source_volid\": null, \"encryption_key_id\": null, \"consistencygroup_id\": null, \"group_id\": null, \"bootable\": false, \"multiattach\": false, \"replication_status\": null, \"replication_extended_status\": null, \"replication_driver_data\": null, \"previous_status\": \"available\", \"service_uuid\": \"6d0941fd-19ec-48ae-82b1-f25526175227\", \"shared_targets\": false, \"deleted_at\": null, \"deleted\": false, \"created_at\": \"2025-11-24T19:39:30.000000\", \"updated_at\": \"2025-11-24T19:39:33.000000\"}, \"volume-metadata\": {\"vol-meta1\": \"value1\", \"vol-meta2\": \"value2\", \"vol-meta3\": \"value3\"}}" Nov 24 14:40:04.148512 np0005533992.novalocal cinder-backup[122429]: }. {{(pid=122429) _read_metadata /opt/stack/cinder/cinder/backup/chunkeddriver.py:330}} Nov 24 14:40:04.152986 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] v1 volume backup restore of 5b2b93ef-5bc1-40f8-a171-c0aa7b733451 started. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:700}} Nov 24 14:40:04.152986 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] metadata_object_names = ['volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00001', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00002', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00003', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00004', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00005', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00006', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00007', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00008', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00009', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00010', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00011', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00012', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00013', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00014', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00015', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00016', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00017', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00018', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00019', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00020', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00021']. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:707}} Nov 24 14:40:04.194929 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] generated object list: ['volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00001', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00002', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00003', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00004', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00005', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00006', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00007', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00008', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00009', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00010', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00011', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00012', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00013', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00014', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00015', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00016', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00017', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00018', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00019', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00020', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00021', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451_metadata', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451_sha256file']. {{(pid=122429) _generate_object_names /opt/stack/cinder/cinder/backup/chunkeddriver.py:263}} Nov 24 14:40:04.199761 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_db.sqlalchemy.engines [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=122429) _check_effective_sql_mode /usr/local/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:397}} Nov 24 14:40:04.204118 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451, container: tempest-volumesbackupstest-backup-container-1555790666, object name: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00001, volume: 96b9f01f-3dff-415d-9c78-d1149c899edb. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:40:04.224415 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:40:04.454252 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451, container: tempest-volumesbackupstest-backup-container-1555790666, object name: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00002, volume: 96b9f01f-3dff-415d-9c78-d1149c899edb. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:40:04.472938 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:40:04.671520 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451, container: tempest-volumesbackupstest-backup-container-1555790666, object name: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00003, volume: 96b9f01f-3dff-415d-9c78-d1149c899edb. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:40:04.691980 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:40:04.935374 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451, container: tempest-volumesbackupstest-backup-container-1555790666, object name: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00004, volume: 96b9f01f-3dff-415d-9c78-d1149c899edb. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:40:04.951038 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:40:05.166336 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451, container: tempest-volumesbackupstest-backup-container-1555790666, object name: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00005, volume: 96b9f01f-3dff-415d-9c78-d1149c899edb. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:40:05.184426 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:40:05.390977 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451, container: tempest-volumesbackupstest-backup-container-1555790666, object name: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00006, volume: 96b9f01f-3dff-415d-9c78-d1149c899edb. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:40:05.410795 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:40:05.604422 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451, container: tempest-volumesbackupstest-backup-container-1555790666, object name: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00007, volume: 96b9f01f-3dff-415d-9c78-d1149c899edb. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:40:05.624997 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:40:05.831690 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451, container: tempest-volumesbackupstest-backup-container-1555790666, object name: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00008, volume: 96b9f01f-3dff-415d-9c78-d1149c899edb. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:40:05.851008 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:40:06.026480 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451, container: tempest-volumesbackupstest-backup-container-1555790666, object name: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00009, volume: 96b9f01f-3dff-415d-9c78-d1149c899edb. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:40:06.046183 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:40:06.233049 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451, container: tempest-volumesbackupstest-backup-container-1555790666, object name: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00010, volume: 96b9f01f-3dff-415d-9c78-d1149c899edb. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:40:06.250726 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:40:06.461794 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451, container: tempest-volumesbackupstest-backup-container-1555790666, object name: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00011, volume: 96b9f01f-3dff-415d-9c78-d1149c899edb. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:40:06.478026 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:40:06.679738 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451, container: tempest-volumesbackupstest-backup-container-1555790666, object name: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00012, volume: 96b9f01f-3dff-415d-9c78-d1149c899edb. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:40:06.700721 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:40:06.913058 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451, container: tempest-volumesbackupstest-backup-container-1555790666, object name: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00013, volume: 96b9f01f-3dff-415d-9c78-d1149c899edb. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:40:06.931970 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:40:07.110954 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451, container: tempest-volumesbackupstest-backup-container-1555790666, object name: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00014, volume: 96b9f01f-3dff-415d-9c78-d1149c899edb. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:40:07.125750 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:40:07.312815 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451, container: tempest-volumesbackupstest-backup-container-1555790666, object name: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00015, volume: 96b9f01f-3dff-415d-9c78-d1149c899edb. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:40:07.330007 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:40:07.504436 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451, container: tempest-volumesbackupstest-backup-container-1555790666, object name: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00016, volume: 96b9f01f-3dff-415d-9c78-d1149c899edb. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:40:07.521047 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:40:07.705324 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451, container: tempest-volumesbackupstest-backup-container-1555790666, object name: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00017, volume: 96b9f01f-3dff-415d-9c78-d1149c899edb. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:40:07.722820 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:40:07.919839 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451, container: tempest-volumesbackupstest-backup-container-1555790666, object name: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00018, volume: 96b9f01f-3dff-415d-9c78-d1149c899edb. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:40:07.937891 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:40:08.110702 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451, container: tempest-volumesbackupstest-backup-container-1555790666, object name: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00019, volume: 96b9f01f-3dff-415d-9c78-d1149c899edb. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:40:08.130949 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:40:08.319546 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451, container: tempest-volumesbackupstest-backup-container-1555790666, object name: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00020, volume: 96b9f01f-3dff-415d-9c78-d1149c899edb. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:40:08.336356 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:40:08.541578 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] restoring object. backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451, container: tempest-volumesbackupstest-backup-container-1555790666, object name: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00021, volume: 96b9f01f-3dff-415d-9c78-d1149c899edb. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:727}} Nov 24 14:40:08.557762 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] decompressing data using zlib algorithm {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:744}} Nov 24 14:40:08.644923 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] v1 volume backup restore of 5b2b93ef-5bc1-40f8-a171-c0aa7b733451 finished. {{(pid=122429) _restore_v1 /opt/stack/cinder/cinder/backup/chunkeddriver.py:772}} Nov 24 14:40:08.645116 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Restoring volume base metadata {{(pid=122429) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} Nov 24 14:40:08.658287 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Restoring volume metadata {{(pid=122429) _restore_vol_meta /opt/stack/cinder/cinder/backup/driver.py:249}} Nov 24 14:40:08.670644 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] No metadata of type 'volume-glance-metadata' to restore {{(pid=122429) put /opt/stack/cinder/cinder/backup/driver.py:345}} Nov 24 14:40:08.670976 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] restore 5b2b93ef-5bc1-40f8-a171-c0aa7b733451 to 96b9f01f-3dff-415d-9c78-d1149c899edb finished. {{(pid=122429) restore /opt/stack/cinder/cinder/backup/chunkeddriver.py:837}} Nov 24 14:40:08.671639 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sda {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:40:08.694438 np0005533992.novalocal sudo[162500]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sda Nov 24 14:40:09.039280 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sda" returned: 0 in 0.367s {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:40:09.040853 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] ==> disconnect_volume: call "{'args': (, {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb', 'target_lun': 0, 'volume_id': '96b9f01f-3dff-415d-9c78-d1149c899edb', 'auth_method': 'CHAP', 'auth_username': 'KD5tYUQFPJFsRd32qJcy', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False, 'enforce_multipath': False}, {'type': 'block', 'scsi_wwn': '36001405a6c257ac644040be9d009ea4d', 'path': '/dev/sda'}), 'kwargs': {'force': False}}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:40:09.041197 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Acquiring lock "connect_volume" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:40:09.041901 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:40:09.043522 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m discoverydb -o show -P 1 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:40:09.057408 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m discoverydb -o show -P 1" returned: 0 in 0.014s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:40:09.057715 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[d1f9b7f4-78a6-4a31-ba87-84c1aec1610f]: (4, ('SENDTARGETS:\nNo targets found.\niSNS:\nNo targets found.\nSTATIC:\nTarget: iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb\n\tPortal: 38.102.83.12:3260,-1\n\t\tIface Name: default\nFIRMWARE:\nNo targets found.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:40:09.058144 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ['-m', 'discoverydb', '-o', 'show', '-P', 1]: stdout=SENDTARGETS: Nov 24 14:40:09.058144 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:40:09.058144 np0005533992.novalocal cinder-backup[122429]: iSNS: Nov 24 14:40:09.058144 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:40:09.058144 np0005533992.novalocal cinder-backup[122429]: STATIC: Nov 24 14:40:09.058144 np0005533992.novalocal cinder-backup[122429]: Target: iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb Nov 24 14:40:09.058144 np0005533992.novalocal cinder-backup[122429]: Portal: 38.102.83.12:3260,-1 Nov 24 14:40:09.058144 np0005533992.novalocal cinder-backup[122429]: Iface Name: default Nov 24 14:40:09.058144 np0005533992.novalocal cinder-backup[122429]: FIRMWARE: Nov 24 14:40:09.058144 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:40:09.058144 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:40:09.058441 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Regex to get portals from discoverydb: ^SENDTARGETS: Nov 24 14:40:09.058441 np0005533992.novalocal cinder-backup[122429]: .*?^DiscoveryAddress: 38.102.83.12,3260.*? Nov 24 14:40:09.058441 np0005533992.novalocal cinder-backup[122429]: (.*?)^(?:DiscoveryAddress|iSNS):.* {{(pid=122429) _get_discoverydb_portals /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:392}} Nov 24 14:40:09.058963 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Getting connected devices for (ips,iqns,luns)=[('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb', 0)] {{(pid=122429) _get_connection_devices /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:855}} Nov 24 14:40:09.060107 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:40:09.072533 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:40:09.072787 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[1a01f877-3edc-41e2-b79a-fa41e0d86a2f]: (4, ('38.102.83.12:3260,4294967295 iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:40:09.073789 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:40:09.083843 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:40:09.084038 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[44527b81-bf81-4631-88b6-88795f404409]: (4, ('tcp: [26] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:40:09.084403 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('-m', 'session'): stdout=tcp: [26] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb (non-flash) Nov 24 14:40:09.084403 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:40:09.084631 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] iscsi session list stdout=tcp: [26] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb (non-flash) Nov 24 14:40:09.084631 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:40:09.085310 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Resulting device map defaultdict(. at 0x7f5b7298e2a0>, {('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb'): ({'sda'}, set())}) {{(pid=122429) _get_connection_devices /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:888}} Nov 24 14:40:09.085609 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Removing single pathed devices sda {{(pid=122429) remove_connection /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:381}} Nov 24 14:40:09.086442 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd show status {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:40:09.094346 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] 'multipathd show status' failed. Not Retrying. {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:40:09.096356 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[9d31dc2a-4ada-4d83-8f5d-7ffe1fe0b711]: [Errno 2] No such file or directory: 'multipathd' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:40:09.096356 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:40:09.096356 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:40:09.096356 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:40:09.096356 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:40:09.096356 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:40:09.096356 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:40:09.096356 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:40:09.096356 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root Nov 24 14:40:09.096356 np0005533992.novalocal cinder-backup[122429]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Nov 24 14:40:09.096356 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:40:09.096356 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute Nov 24 14:40:09.096356 np0005533992.novalocal cinder-backup[122429]: return putils.execute(on_execute=on_execute, Nov 24 14:40:09.096356 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:40:09.096356 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute Nov 24 14:40:09.096356 np0005533992.novalocal cinder-backup[122429]: obj = subprocess.Popen(cmd, Nov 24 14:40:09.096356 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:40:09.096356 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__ Nov 24 14:40:09.096356 np0005533992.novalocal cinder-backup[122429]: self._execute_child(args, executable, preexec_fn, close_fds, Nov 24 14:40:09.096356 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child Nov 24 14:40:09.096356 np0005533992.novalocal cinder-backup[122429]: raise child_exception_type(errno_num, err_msg, err_filename) Nov 24 14:40:09.096356 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: 'multipathd' Nov 24 14:40:09.098163 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[9d31dc2a-4ada-4d83-8f5d-7ffe1fe0b711]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root\n return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute\n return putils.execute(on_execute=on_execute,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute\n obj = subprocess.Popen(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__\n self._execute_child(args, executable, preexec_fn, close_fds,\n File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child\n raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: \'multipathd\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:40:09.098713 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Flushing IO for device /dev/sda {{(pid=122429) flush_device_io /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:441}} Nov 24 14:40:09.099642 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blockdev --flushbufs /dev/sda {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:40:09.108453 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "blockdev --flushbufs /dev/sda" returned: 0 in 0.008s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:40:09.108737 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[222fc266-359b-428c-b972-598d2589f783]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:40:09.109358 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Remove SCSI device /dev/sda with /sys/block/sda/device/delete {{(pid=122429) remove_scsi_device /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:151}} Nov 24 14:40:09.110455 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/block/sda/device/delete {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:40:09.163251 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/block/sda/device/delete" returned: 0 in 0.053s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:40:09.163849 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[29caac48-6abf-4afa-b5b6-ec3fe89f1e63]: (4, ('1', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:40:09.164076 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Checking to see if SCSI volumes sda have been removed. {{(pid=122429) wait_for_volumes_removal /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:159}} Nov 24 14:40:09.164353 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] SCSI volumes sda have been removed. {{(pid=122429) wait_for_volumes_removal /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:169}} Nov 24 14:40:09.164795 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Disconnecting from: [('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb')] {{(pid=122429) _disconnect_connection /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1201}} Nov 24 14:40:09.167205 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb -p 38.102.83.12:3260 --op update -n node.startup -v manual {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:40:09.178849 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb -p 38.102.83.12:3260 --op update -n node.startup -v manual" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:40:09.179140 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[0e622f9b-f20a-4b8d-878a-f85443de9cdf]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:40:09.179711 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:40:09.180926 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb -p 38.102.83.12:3260 --logout {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:40:09.207622 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb -p 38.102.83.12:3260 --logout" returned: 0 in 0.026s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:40:09.207935 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[ec509fff-6b82-477e-900a-3d0eb77e7548]: (4, ('Logging out of session [sid: 26, target: iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb, portal: 38.102.83.12,3260]\nLogout of [sid: 26, target: iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb, portal: 38.102.83.12,3260] successful.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:40:09.208847 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--logout',): stdout=Logging out of session [sid: 26, target: iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb, portal: 38.102.83.12,3260] Nov 24 14:40:09.208847 np0005533992.novalocal cinder-backup[122429]: Logout of [sid: 26, target: iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb, portal: 38.102.83.12,3260] successful. Nov 24 14:40:09.208847 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:40:09.209611 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb -p 38.102.83.12:3260 --op delete {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:40:09.220512 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-96b9f01f-3dff-415d-9c78-d1149c899edb -p 38.102.83.12:3260 --op delete" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:40:09.220867 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[ea78b086-70ff-4962-94fd-2c6eb6c15e1e]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:40:09.221397 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'delete'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:40:09.222647 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: held 0.180s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:40:09.222960 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] <== disconnect_volume: return (182ms) None {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:40:12.745200 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.manager [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Encryption key ID for volume 96b9f01f-3dff-415d-9c78-d1149c899edb already matches encryption key ID in backup 5b2b93ef-5bc1-40f8-a171-c0aa7b733451. {{(pid=122429) _run_restore /opt/stack/cinder/cinder/backup/manager.py:874}} Nov 24 14:40:12.764603 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-f4007b96-a75e-465c-97eb-803ac9fbc9f4 tempest-VolumesBackupsTest-1816644175 None] Finished restoring backup 5b2b93ef-5bc1-40f8-a171-c0aa7b733451 to volume 96b9f01f-3dff-415d-9c78-d1149c899edb. Nov 24 14:40:15.204737 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] Delete backup started, backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451. Nov 24 14:40:15.206237 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] Using swift URL https://38.102.83.12:8080/v1/AUTH_c642367dafaa45dd89634e7ec9822ef2 {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:286}} Nov 24 14:40:15.206502 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] Connect to https://38.102.83.12:8080/v1/AUTH_ in "per_user" mode {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:287}} Nov 24 14:40:15.206851 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] delete started, backup: 5b2b93ef-5bc1-40f8-a171-c0aa7b733451, container: tempest-volumesbackupstest-backup-container-1555790666, prefix: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:844}} Nov 24 14:40:15.283957 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] generated object list: ['volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00001', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00002', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00003', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00004', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00005', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00006', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00007', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00008', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00009', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00010', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00011', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00012', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00013', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00014', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00015', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00016', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00017', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00018', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00019', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00020', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00021', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451_metadata', 'volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451_sha256file']. {{(pid=122429) _generate_object_names /opt/stack/cinder/cinder/backup/chunkeddriver.py:263}} Nov 24 14:40:15.314427 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00001 in container: tempest-volumesbackupstest-backup-container-1555790666. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:40:15.343727 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00002 in container: tempest-volumesbackupstest-backup-container-1555790666. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:40:15.370987 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00003 in container: tempest-volumesbackupstest-backup-container-1555790666. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:40:15.398846 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00004 in container: tempest-volumesbackupstest-backup-container-1555790666. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:40:15.423791 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00005 in container: tempest-volumesbackupstest-backup-container-1555790666. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:40:15.450490 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00006 in container: tempest-volumesbackupstest-backup-container-1555790666. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:40:15.475927 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00007 in container: tempest-volumesbackupstest-backup-container-1555790666. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:40:15.503977 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00008 in container: tempest-volumesbackupstest-backup-container-1555790666. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:40:15.530498 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00009 in container: tempest-volumesbackupstest-backup-container-1555790666. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:40:15.558192 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00010 in container: tempest-volumesbackupstest-backup-container-1555790666. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:40:15.595203 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00011 in container: tempest-volumesbackupstest-backup-container-1555790666. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:40:15.628119 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00012 in container: tempest-volumesbackupstest-backup-container-1555790666. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:40:15.669768 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00013 in container: tempest-volumesbackupstest-backup-container-1555790666. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:40:15.693331 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00014 in container: tempest-volumesbackupstest-backup-container-1555790666. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:40:15.722979 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00015 in container: tempest-volumesbackupstest-backup-container-1555790666. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:40:15.766582 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00016 in container: tempest-volumesbackupstest-backup-container-1555790666. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:40:15.788924 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00017 in container: tempest-volumesbackupstest-backup-container-1555790666. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:40:15.810943 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00018 in container: tempest-volumesbackupstest-backup-container-1555790666. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:40:15.830697 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00019 in container: tempest-volumesbackupstest-backup-container-1555790666. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:40:15.855069 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00020 in container: tempest-volumesbackupstest-backup-container-1555790666. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:40:15.877110 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451-00021 in container: tempest-volumesbackupstest-backup-container-1555790666. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:40:15.899139 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451_metadata in container: tempest-volumesbackupstest-backup-container-1555790666. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:40:15.924111 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_9c72607f-3da7-4df8-8808-5a7d5aebdece/20251124193937/az_nova_backup_5b2b93ef-5bc1-40f8-a171-c0aa7b733451_sha256file in container: tempest-volumesbackupstest-backup-container-1555790666. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:40:15.924111 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] delete 5b2b93ef-5bc1-40f8-a171-c0aa7b733451 finished. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:870}} Nov 24 14:40:15.925681 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_db.sqlalchemy.engines [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=122429) _check_effective_sql_mode /usr/local/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:397}} Nov 24 14:40:15.938285 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.quota [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] Created reservations ['c3c0914a-1350-4caf-9e6e-0261e4cbb658', '975dadeb-aac3-4de0-a9ee-812032e4f8bb'] {{(pid=122429) reserve /opt/stack/cinder/cinder/quota.py:752}} Nov 24 14:40:15.949062 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-89183a50-5fe0-47c6-b459-07ab4c9db553 tempest-VolumesBackupsTest-1816644175 None] Delete backup finished, backup 5b2b93ef-5bc1-40f8-a171-c0aa7b733451 deleted. Nov 24 14:40:25.948864 np0005533992.novalocal cinder-backup[122429]: DEBUG dbcounter [-] [122429] Writing DB stats cinder:SELECT=36,cinder:UPDATE=10,cinder:INSERT=4 {{(pid=122429) stat_writer /usr/local/lib/python3.12/site-packages/dbcounter.py:115}} Nov 24 14:40:46.425532 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:40:46.425532 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:41:13.851601 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Create backup started, backup: 519039aa-9b69-46f1-b149-a260745c08c7 volume: 662c5cd4-cc35-4e90-a911-679b13dfa7a5. Nov 24 14:41:13.856674 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Call Volume Manager to get_backup_device for Backup(availability_zone='nova',container='tempest-volumesbackupstest-backup-container-601227527',created_at=2025-11-24T19:41:13Z,data_timestamp=2025-11-24T19:41:13Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-322041849',encryption_key_id=None,fail_reason=None,host='np0005533992.novalocal',id=519039aa-9b69-46f1-b149-a260745c08c7,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='c642367dafaa45dd89634e7ec9822ef2',restore_volume_id=None,service='cinder.backup.drivers.swift.SwiftBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='0e48df3b35f84ee88136fd47ee531c1f',volume_id=662c5cd4-cc35-4e90-a911-679b13dfa7a5) Nov 24 14:41:15.305601 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Using swift URL https://38.102.83.12:8080/v1/AUTH_c642367dafaa45dd89634e7ec9822ef2 {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:286}} Nov 24 14:41:15.306666 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Connect to https://38.102.83.12:8080/v1/AUTH_ in "per_user" mode {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:287}} Nov 24 14:41:15.307027 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '38.102.83.12', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:41:15.308293 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:15.314501 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.007s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:15.314773 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[ee00e06e-1740-4d65-ba6f-ddf8e5409eeb]: (4, ('InitiatorName=iqn.1994-05.com.redhat:5b9c7bcf3f78\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:15.317073 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[225ce1b4-d856-4e74-a544-35491ff31750]: [Errno 2] No such file or directory: '/dev/scini' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:41:15.317073 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:41:15.317073 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:41:15.317073 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:41:15.317073 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:15.317073 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:41:15.317073 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:41:15.317073 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:15.317073 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 57, in get_guid Nov 24 14:41:15.317073 np0005533992.novalocal cinder-backup[122429]: with open_scini_device() as fd: Nov 24 14:41:15.317073 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:15.317073 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/contextlib.py", line 137, in __enter__ Nov 24 14:41:15.317073 np0005533992.novalocal cinder-backup[122429]: return next(self.gen) Nov 24 14:41:15.317073 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^ Nov 24 14:41:15.317073 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 40, in open_scini_device Nov 24 14:41:15.317073 np0005533992.novalocal cinder-backup[122429]: fd = os.open(SCINI_DEVICE_PATH, os.O_RDWR) Nov 24 14:41:15.317073 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:15.317073 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: '/dev/scini' Nov 24 14:41:15.317577 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[225ce1b4-d856-4e74-a544-35491ff31750]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 57, in get_guid\n with open_scini_device() as fd:\n ^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/contextlib.py", line 137, in __enter__\n return next(self.gen)\n ^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 40, in open_scini_device\n fd = os.open(SCINI_DEVICE_PATH, os.O_RDWR)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory: \'/dev/scini\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:15.317965 np0005533992.novalocal cinder-backup[122429]: ERROR os_brick.initiator.connectors.scaleio [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Error querying sdc guid: [Errno 2] No such file or directory: FileNotFoundError: [Errno 2] No such file or directory Nov 24 14:41:15.318423 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.scaleio [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Unable to find SDC guid: Error querying sdc guid: [Errno 2] No such file or directory Nov 24 14:41:15.319114 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt -v / -n -o SOURCE {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:15.325988 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt -v / -n -o SOURCE" returned: 0 in 0.007s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:15.326227 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[f07b5b88-81a6-4cdb-97e8-86c9f57062dd]: (4, ('/dev/vda2\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:15.327131 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/vda2 -s UUID -o value {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:15.334111 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/vda2 -s UUID -o value" returned: 0 in 0.007s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:15.334427 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[19c40ce0-f53c-4750-9b6f-0e28f399ca58]: (4, ('3fc68afb-d7e4-475c-9164-ea34f338d965\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:15.335404 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[88138a0b-f852-42df-bcb3-f0b68bf6638c]: (4, 'e2c36df9-c117-4156-b18b-d355565d8e8e') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:15.335967 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Running cmd (subprocess): nvme version {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:15.349511 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] 'nvme version' failed. Not Retrying. {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:41:15.349958 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.nvmeof [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] nvme not present on system {{(pid=122429) nvme_present /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/nvmeof.py:784}} Nov 24 14:41:15.351829 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=122429) find_dsc /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:132}} Nov 24 14:41:15.353051 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.lightos [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Current host hostNQN nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e and IP(s) are ['38.102.83.12', 'fe80::f816:3eff:fe17:2fa8', '192.168.122.1', '172.24.5.1', '2001:db8::2', 'fe80::64fc:4aff:fe7b:de4b', 'fe80::fc16:3eff:fe29:10fc', 'fe80::d821:f8ff:fe9f:14e6'] Nov 24 14:41:15.353335 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=122429) get_connector_properties /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:109}} Nov 24 14:41:15.353635 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e dsc: {{(pid=122429) get_connector_properties /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:112}} Nov 24 14:41:15.354171 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] <== get_connector_properties: return (46ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '38.102.83.12', 'host': 'np0005533992.novalocal', 'multipath': False, 'enforce_multipath': False, 'initiator': 'iqn.1994-05.com.redhat:5b9c7bcf3f78', 'do_local_attach': False, 'uuid': '3fc68afb-d7e4-475c-9164-ea34f338d965', 'system uuid': 'e2c36df9-c117-4156-b18b-d355565d8e8e', 'nvme_native_multipath': False, 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e', 'found_dsc': '', 'host_ips': ['38.102.83.12', 'fe80::f816:3eff:fe17:2fa8', '192.168.122.1', '172.24.5.1', '2001:db8::2', 'fe80::64fc:4aff:fe7b:de4b', 'fe80::fc16:3eff:fe29:10fc', 'fe80::d821:f8ff:fe9f:14e6']} {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:41:18.863519 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connector [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Factory for iscsi on None {{(pid=122429) factory /usr/local/lib/python3.12/site-packages/os_brick/initiator/connector.py:281}} Nov 24 14:41:18.864673 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] ==> connect_volume: call "{'self': , 'connection_properties': {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7', 'target_lun': 0, 'volume_id': '437b4cfc-c64e-4d3d-a3c2-23e3a27331b7', 'auth_method': 'CHAP', 'auth_username': 'U3BrRNjhH3FLQ4J9HExL', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False}}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:41:18.864966 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Acquiring lock "connect_volume" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:41:18.865578 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:41:18.865873 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Connection properties {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7', 'target_lun': 0, 'volume_id': '437b4cfc-c64e-4d3d-a3c2-23e3a27331b7', 'auth_method': 'CHAP', 'auth_username': 'U3BrRNjhH3FLQ4J9HExL', 'auth_password': 'JhWuZhh35BtAS7Jm', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False, 'enforce_multipath': False} {{(pid=122429) check_multipath /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:133}} Nov 24 14:41:18.867389 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd show status {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:18.872788 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] 'multipathd show status' failed. Not Retrying. {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:41:18.874783 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[86116dce-3599-456a-b93e-aa2618773993]: [Errno 2] No such file or directory: 'multipathd' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:41:18.874783 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:41:18.874783 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:41:18.874783 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:41:18.874783 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:18.874783 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:41:18.874783 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:41:18.874783 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:18.874783 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root Nov 24 14:41:18.874783 np0005533992.novalocal cinder-backup[122429]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Nov 24 14:41:18.874783 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:18.874783 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute Nov 24 14:41:18.874783 np0005533992.novalocal cinder-backup[122429]: return putils.execute(on_execute=on_execute, Nov 24 14:41:18.874783 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:18.874783 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute Nov 24 14:41:18.874783 np0005533992.novalocal cinder-backup[122429]: obj = subprocess.Popen(cmd, Nov 24 14:41:18.874783 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:18.874783 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__ Nov 24 14:41:18.874783 np0005533992.novalocal cinder-backup[122429]: self._execute_child(args, executable, preexec_fn, close_fds, Nov 24 14:41:18.874783 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child Nov 24 14:41:18.874783 np0005533992.novalocal cinder-backup[122429]: raise child_exception_type(errno_num, err_msg, err_filename) Nov 24 14:41:18.874783 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: 'multipathd' Nov 24 14:41:18.876444 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[86116dce-3599-456a-b93e-aa2618773993]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root\n return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute\n return putils.execute(on_execute=on_execute,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute\n obj = subprocess.Popen(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__\n self._execute_child(args, executable, preexec_fn, close_fds,\n File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child\n raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: \'multipathd\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:18.877361 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Acquiring lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:41:18.878071 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:41:18.878435 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Trying to connect to iSCSI portal 38.102.83.12:3260 Nov 24 14:41:18.879474 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 -p 38.102.83.12:3260 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:18.889824 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 -p 38.102.83.12:3260" returned: 21 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:18.890050 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[f500a68e-1c79-4bd7-b808-5242e4db8339]: (4, ('', 'iscsiadm: No records found\n')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:18.890494 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] iscsiadm (): stdout= stderr=iscsiadm: No records found Nov 24 14:41:18.890494 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:41:18.891388 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 -p 38.102.83.12:3260 --interface default --op new {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:18.902440 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 -p 38.102.83.12:3260 --interface default --op new" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:18.902744 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[e2f0d459-ac66-470a-9268-e8b3ce4ca516]: (4, ('New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 38.102.83.12,3260,-1 iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7] added\n', 'iscsiadm: Config file line 381 does not have value\niscsiadm: Config file line 381 does not have value\niscsiadm: Config file line 381 does not have value\n')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:18.903177 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--interface', 'default', '--op', 'new'): stdout=New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 38.102.83.12,3260,-1 iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7] added Nov 24 14:41:18.903177 np0005533992.novalocal cinder-backup[122429]: stderr=iscsiadm: Config file line 381 does not have value Nov 24 14:41:18.903177 np0005533992.novalocal cinder-backup[122429]: iscsiadm: Config file line 381 does not have value Nov 24 14:41:18.903177 np0005533992.novalocal cinder-backup[122429]: iscsiadm: Config file line 381 does not have value Nov 24 14:41:18.903177 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:41:18.903439 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Retrying to connect to iSCSI portal 38.102.83.12:3260 {{(pid=122429) _connect_to_iscsi_portal_unsafe /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1133}} Nov 24 14:41:18.904076 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Finished call to 'os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe' after 0.026(s), this was the 1st time calling it. {{(pid=122429) log_it /usr/local/lib/python3.12/site-packages/tenacity/after.py:44}} Nov 24 14:41:18.904413 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Retrying os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe in 1.0 seconds as it raised BrickException: An unknown exception occurred.. {{(pid=122429) log_it /usr/local/lib/python3.12/site-packages/tenacity/before_sleep.py:65}} Nov 24 14:41:19.906040 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Trying to connect to iSCSI portal 38.102.83.12:3260 Nov 24 14:41:19.907876 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 -p 38.102.83.12:3260 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:19.919016 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 -p 38.102.83.12:3260" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:19.920870 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[8b71736b-27c8-4c68-9b8f-14f46bc2ad21]: (4, ('# BEGIN RECORD 6.2.1.11\nnode.name = iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7\nnode.tpgt = -1\nnode.startup = automatic\nnode.leading_login = No\niface.iscsi_ifacename = default\niface.net_ifacename = \niface.ipaddress = \niface.prefix_len = 0\niface.hwaddress = \niface.transport_name = tcp\niface.initiatorname = \niface.state = \niface.vlan_id = 0\niface.vlan_priority = 0\niface.vlan_state = \niface.iface_num = 0\niface.mtu = 0\niface.port = 0\niface.bootproto = \niface.subnet_mask = \niface.gateway = \niface.dhcp_alt_client_id_state = \niface.dhcp_alt_client_id = \niface.dhcp_dns = \niface.dhcp_learn_iqn = \niface.dhcp_req_vendor_id_state = \niface.dhcp_vendor_id_state = \niface.dhcp_vendor_id = \niface.dhcp_slp_da = \niface.fragmentation = \niface.gratuitous_arp = \niface.incoming_forwarding = \niface.tos_state = \niface.tos = 0\niface.ttl = 0\niface.delayed_ack = \niface.tcp_nagle = \niface.tcp_wsf_state = \niface.tcp_wsf = 0\niface.tcp_timer_scale = 0\niface.tcp_timestamp = \niface.redirect = \niface.def_task_mgmt_timeout = 0\niface.header_digest = \niface.data_digest = \niface.immediate_data = \niface.initial_r2t = \niface.data_seq_inorder = \niface.data_pdu_inorder = \niface.erl = 0\niface.max_receive_data_len = 0\niface.first_burst_len = 0\niface.max_outstanding_r2t = 0\niface.max_burst_len = 0\niface.chap_auth = \niface.bidi_chap = \niface.strict_login_compliance = \niface.discovery_auth = \niface.discovery_logout = \nnode.discovery_address = \nnode.discovery_port = 0\nnode.discovery_type = fw\nnode.session.initial_cmdsn = 0\nnode.session.initial_login_retry_max = 8\nnode.session.xmit_thread_priority = 0\nnode.session.cmds_max = 128\nnode.session.queue_depth = 32\nnode.session.nr_sessions = 1\nnode.session.auth.authmethod = None\nnode.session.auth.username = \nnode.session.auth.password = \nnode.session.auth.username_in = \nnode.session.auth.password_in = \nnode.session.auth.chap_algs = SHA3-256,SHA256\nnode.session.timeo.replacement_timeout = 120\nnode.session.err_timeo.abort_timeout = 15\nnode.session.err_timeo.lu_reset_timeout = 30\nnode.session.err_timeo.tgt_reset_timeout = 30\nnode.session.err_timeo.host_reset_timeout = 60\nnode.session.iscsi.FastAbort = Yes\nnode.session.iscsi.InitialR2T = No\nnode.session.iscsi.ImmediateData = Yes\nnode.session.iscsi.FirstBurstLength = 262144\nnode.session.iscsi.MaxBurstLength = 16776192\nnode.session.iscsi.DefaultTime2Retain = 0\nnode.session.iscsi.DefaultTime2Wait = 2\nnode.session.iscsi.MaxConnections = 1\nnode.session.iscsi.MaxOutstandingR2T = 1\nnode.session.iscsi.ERL = 0\nnode.session.scan = auto\nnode.session.reopen_max = 0\nnode.session.conn_reopen_log_freq = 1\nnode.conn[0].address = 38.102.83.12\nnode.conn[0].port = 3260\nnode.conn[0].startup = manual\nnode.conn[0].tcp.window_size = 524288\nnode.conn[0].tcp.type_of_service = 0\nnode.conn[0].timeo.logout_timeout = 15\nnode.conn[0].timeo.login_timeout = 15\nnode.conn[0].timeo.auth_timeout = 45\nnode.conn[0].timeo.noop_out_interval = 5\nnode.conn[0].timeo.noop_out_timeout = 5\nnode.conn[0].iscsi.MaxXmitDataSegmentLength = 0\nnode.conn[0].iscsi.MaxRecvDataSegmentLength = 262144\nnode.conn[0].iscsi.HeaderDigest = None\nnode.conn[0].iscsi.DataDigest = None\nnode.conn[0].iscsi.IFMarker = No\nnode.conn[0].iscsi.OFMarker = No\n# END RECORD\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] iscsiadm (): stdout=# BEGIN RECORD 6.2.1.11 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.name = iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.tpgt = -1 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.startup = automatic Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.leading_login = No Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.iscsi_ifacename = default Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.net_ifacename = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.ipaddress = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.prefix_len = 0 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.hwaddress = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.transport_name = tcp Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.initiatorname = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.state = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.vlan_id = 0 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.vlan_priority = 0 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.vlan_state = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.iface_num = 0 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.mtu = 0 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.port = 0 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.bootproto = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.subnet_mask = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.gateway = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_alt_client_id_state = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_alt_client_id = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_dns = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_learn_iqn = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_req_vendor_id_state = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_vendor_id_state = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_vendor_id = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_slp_da = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.fragmentation = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.gratuitous_arp = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.incoming_forwarding = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.tos_state = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.tos = 0 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.ttl = 0 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.delayed_ack = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.tcp_nagle = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.tcp_wsf_state = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.tcp_wsf = 0 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.tcp_timer_scale = 0 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.tcp_timestamp = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.redirect = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.def_task_mgmt_timeout = 0 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.header_digest = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.data_digest = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.immediate_data = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.initial_r2t = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.data_seq_inorder = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.data_pdu_inorder = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.erl = 0 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.max_receive_data_len = 0 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.first_burst_len = 0 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.max_outstanding_r2t = 0 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.max_burst_len = 0 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.chap_auth = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.bidi_chap = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.strict_login_compliance = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.discovery_auth = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: iface.discovery_logout = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.discovery_address = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.discovery_port = 0 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.discovery_type = fw Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.initial_cmdsn = 0 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.initial_login_retry_max = 8 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.xmit_thread_priority = 0 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.cmds_max = 128 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.queue_depth = 32 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.nr_sessions = 1 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.auth.authmethod = None Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.auth.username = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.auth.password = *** Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.auth.username_in = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.auth.password_in = Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.auth.chap_algs = SHA3-256,SHA256 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.timeo.replacement_timeout = 120 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.abort_timeout = 15 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.lu_reset_timeout = 30 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.tgt_reset_timeout = 30 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.host_reset_timeout = 60 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.FastAbort = Yes Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.InitialR2T = No Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.ImmediateData = Yes Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.FirstBurstLength = 262144 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxBurstLength = 16776192 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.DefaultTime2Retain = 0 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.DefaultTime2Wait = 2 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxConnections = 1 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxOutstandingR2T = 1 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.ERL = 0 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.scan = auto Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.reopen_max = 0 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.session.conn_reopen_log_freq = 1 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.conn[0].address = 38.102.83.12 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.conn[0].port = 3260 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.conn[0].startup = manual Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.conn[0].tcp.window_size = 524288 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.conn[0].tcp.type_of_service = 0 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.logout_timeout = 15 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.login_timeout = 15 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.auth_timeout = 45 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.noop_out_interval = 5 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.noop_out_timeout = 5 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.MaxXmitDataSegmentLength = 0 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144 Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.HeaderDigest = None Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.DataDigest = None Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.IFMarker = No Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.OFMarker = No Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: # END RECORD Nov 24 14:41:19.921286 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:41:19.924091 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 -p 38.102.83.12:3260 --op update -n node.session.scan -v manual {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:19.933441 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 -p 38.102.83.12:3260 --op update -n node.session.scan -v manual" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:19.933731 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[442bc138-9ff3-4d51-b5f9-f10f454241da]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:19.934224 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.scan', '-v', 'manual'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:41:19.935438 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 -p 38.102.83.12:3260 --op update -n node.session.auth.authmethod -v CHAP {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:19.946410 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 -p 38.102.83.12:3260 --op update -n node.session.auth.authmethod -v CHAP" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:19.946646 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[fa241b32-a952-41fe-bc0c-845c5c39cb1a]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:19.947185 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.authmethod', '-v', 'CHAP'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:41:19.948203 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 -p 38.102.83.12:3260 --op update -n node.session.auth.username -v U3BrRNjhH3FLQ4J9HExL {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:19.959001 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 -p 38.102.83.12:3260 --op update -n node.session.auth.username -v U3BrRNjhH3FLQ4J9HExL" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:19.959314 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[0728c6a9-720d-463c-a5f9-7f0ca4a4e0f6]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:19.959781 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.username', '-v', 'U3BrRNjhH3FLQ4J9HExL'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:41:19.961019 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 -p 38.102.83.12:3260 --op update -n node.session.auth.password -v *** {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:19.971675 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 -p 38.102.83.12:3260 --op update -n node.session.auth.password -v ***" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:19.971896 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[3a362208-1d9c-48b9-9938-adbe014a86bc]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:19.972465 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.password', '-v', '***'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:41:19.973448 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:19.985049 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:19.985344 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[8b0df1c1-5c1d-427d-9b58-92ab807fda30]: (4, ('tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:19.985843 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('-m', 'session'): stdout=tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash) Nov 24 14:41:19.985843 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:41:19.986112 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] iscsi session list stdout=tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash) Nov 24 14:41:19.986112 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:41:19.987360 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 -p 38.102.83.12:3260 --login {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:20.008051 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 -p 38.102.83.12:3260 --login" returned: 0 in 0.019s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:20.008051 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[e9427910-7437-4a56-863c-1a5870ab6f47]: (4, ('Login to [iface: default, target: iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7, portal: 38.102.83.12,3260] successful.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:20.008236 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--login',): stdout=Login to [iface: default, target: iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7, portal: 38.102.83.12,3260] successful. Nov 24 14:41:20.008236 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:41:20.010281 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 -p 38.102.83.12:3260 --op update -n node.startup -v automatic {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:20.021650 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 -p 38.102.83.12:3260 --op update -n node.startup -v automatic" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:20.021923 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[2005e16d-f919-4d5c-b8b2-cc2f1b57229a]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:20.022396 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:41:20.023341 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:20.036163 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.013s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:20.036418 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[6bf645a0-69d0-4b1d-a7a8-5b72b0b9de3b]: (4, ('tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash)\ntcp: [28] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:20.036901 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('-m', 'session'): stdout=tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash) Nov 24 14:41:20.036901 np0005533992.novalocal cinder-backup[122429]: tcp: [28] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 (non-flash) Nov 24 14:41:20.036901 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:41:20.037170 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] iscsi session list stdout=tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash) Nov 24 14:41:20.037170 np0005533992.novalocal cinder-backup[122429]: tcp: [28] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 (non-flash) Nov 24 14:41:20.037170 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:41:20.037694 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: held 1.160s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:41:20.037951 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Connected to 38.102.83.12:3260 {{(pid=122429) _connect_vol /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:659}} Nov 24 14:41:20.038743 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] HCTL ('3', '-', '-', 0) found on session 28 with lun 0 {{(pid=122429) get_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:788}} Nov 24 14:41:20.039018 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Scanning host 3 c: -, t: -, l: 0) {{(pid=122429) scan_iscsi /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:814}} Nov 24 14:41:20.040015 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/class/scsi_host/host3/scan {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:20.067233 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/class/scsi_host/host3/scan" returned: 0 in 0.025s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:20.067233 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[3c4135c7-5a7e-4d94-8005-98ccb9f4fd24]: (4, ('- - 0', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:20.067233 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Searching for a device in session 28 and hctl ['3', '*', '*', 0] yield: None {{(pid=122429) device_name_by_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:808}} Nov 24 14:41:21.069133 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Searching for a device in session 28 and hctl ['3', '*', '*', 0] yield: sdb {{(pid=122429) device_name_by_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:808}} Nov 24 14:41:21.069955 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Connected to sdb using {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7', 'target_lun': 0, 'volume_id': '437b4cfc-c64e-4d3d-a3c2-23e3a27331b7', 'auth_method': 'CHAP', 'auth_username': 'U3BrRNjhH3FLQ4J9HExL', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False} {{(pid=122429) _connect_vol /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:687}} Nov 24 14:41:21.070821 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" :: held 2.205s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:41:21.071167 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] <== connect_volume: return (2206ms) {'type': 'block', 'scsi_wwn': '3600140545226e0442c44d4fb9f1a02a7', 'path': '/dev/sdb'} {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:41:21.071821 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sdb {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:21.093818 np0005533992.novalocal sudo[163365]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sdb Nov 24 14:41:21.409915 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sdb" returned: 0 in 0.338s {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:21.417521 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_db.sqlalchemy.engines [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=122429) _check_effective_sql_mode /usr/local/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:397}} Nov 24 14:41:21.425105 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] _create_container started, container: tempest-volumesbackupstest-backup-container-601227527,backup: 519039aa-9b69-46f1-b149-a260745c08c7. {{(pid=122429) _create_container /opt/stack/cinder/cinder/backup/chunkeddriver.py:252}} Nov 24 14:41:21.505207 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] generate_object_name_prefix: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7 {{(pid=122429) _generate_object_name_prefix /opt/stack/cinder/cinder/backup/drivers/swift.py:433}} Nov 24 14:41:21.510377 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] starting backup of volume: 662c5cd4-cc35-4e90-a911-679b13dfa7a5, volume size: 1073741824, object names prefix volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7, availability zone: nova {{(pid=122429) _prepare_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:362}} Nov 24 14:41:21.908812 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:41:22.040068 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:41:22.040209 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:41:22.194145 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00001: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:41:22.194782 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:41:22.327149 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:41:22.423264 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:41:22.423405 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:41:22.569197 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00002: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:41:22.569458 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:41:23.173215 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:41:23.307773 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:41:23.308243 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:41:23.439071 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00003: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:41:23.439276 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:41:23.647027 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:41:23.755533 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:41:23.755806 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:41:23.899992 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00004: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:41:23.900248 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:41:24.065967 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:41:24.203763 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:41:24.203910 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:41:24.325352 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00005: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:41:24.325764 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:41:24.722872 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:41:24.859479 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:41:24.859786 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:41:25.007144 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00006: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:41:25.007399 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:41:25.404093 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:41:25.535655 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:41:25.536976 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:41:25.715184 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00007: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:41:25.715416 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:41:25.965488 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:41:26.098842 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:41:26.099220 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:41:26.290392 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00008: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:41:26.290756 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:41:26.473027 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:41:26.573104 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:41:26.573219 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:41:26.723894 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00009: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:41:26.724077 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:41:26.901793 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:41:27.062313 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:41:27.062646 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:41:27.198519 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00010: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:41:27.198519 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:41:27.784652 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:41:27.891905 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:41:27.892051 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:41:28.068080 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00011: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:41:28.068309 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:41:28.245329 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:41:28.356524 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:41:28.356880 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:41:28.504930 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00012: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:41:28.505246 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:41:28.886362 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:41:29.020286 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:41:29.020534 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:41:29.172219 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00013: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:41:29.172414 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:41:29.472482 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:41:29.579999 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:41:29.580331 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:41:29.728039 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00014: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:41:29.729473 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:41:30.122295 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:41:30.220624 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:41:30.220773 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:41:30.372421 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00015: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:41:30.372644 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:41:30.813989 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:41:30.947163 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:41:30.947348 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:41:31.095518 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00016: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:41:31.095777 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:41:31.273882 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:41:31.411853 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:41:31.412197 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:41:31.547638 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00017: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:41:31.547898 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:41:31.725787 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:41:31.858996 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:41:31.859459 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:41:32.009919 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00018: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:41:32.009919 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:41:32.163858 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:41:32.268393 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:41:32.268664 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:41:32.422507 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00019: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:41:32.423127 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:41:32.798913 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:41:32.931590 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:41:32.931878 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:41:33.087925 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00020: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:41:33.088057 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:41:33.186250 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:41:33.270684 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Compressed 25165824 bytes of data to 24470 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:41:33.270784 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:41:33.379593 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] backup MD5 for volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00021: 77377273b0a4b61febdbf7bbf52b9db9 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:41:33.379765 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:41:33.383328 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Getting metadata type 'volume-base-metadata' {{(pid=122429) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Nov 24 14:41:33.389494 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=122429) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Nov 24 14:41:33.389494 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Getting metadata type 'volume-metadata' {{(pid=122429) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Nov 24 14:41:33.393087 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] No metadata type 'volume-metadata' available {{(pid=122429) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Nov 24 14:41:33.393249 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Getting metadata type 'volume-glance-metadata' {{(pid=122429) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Nov 24 14:41:33.396747 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] No metadata type 'volume-glance-metadata' available {{(pid=122429) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Nov 24 14:41:33.397012 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] _write_sha256file started, container name: tempest-volumesbackupstest-backup-container-601227527, sha256file filename: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7_sha256file. {{(pid=122429) _write_sha256file /opt/stack/cinder/cinder/backup/chunkeddriver.py:302}} Nov 24 14:41:33.466051 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] _write_sha256file finished. {{(pid=122429) _write_sha256file /opt/stack/cinder/cinder/backup/chunkeddriver.py:318}} Nov 24 14:41:33.466447 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] _write_metadata started, container name: tempest-volumesbackupstest-backup-container-601227527, metadata filename: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7_metadata. {{(pid=122429) _write_metadata /opt/stack/cinder/cinder/backup/chunkeddriver.py:279}} Nov 24 14:41:33.499171 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] _write_metadata finished. Metadata: b'{\n "backup_description": null,\n "backup_id": "519039aa-9b69-46f1-b149-a260745c08c7",\n "backup_name": "tempest-VolumesBackupsTest-Backup-322041849",\n "created_at": "2025-11-24 19:41:13+00:00",\n "objects": [\n {\n "volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00001": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 0\n }\n },\n {\n "volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00002": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 52428800\n }\n },\n {\n "volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00003": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 104857600\n }\n },\n {\n "volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00004": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 157286400\n }\n },\n {\n "volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00005": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 209715200\n }\n },\n {\n "volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00006": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 262144000\n }\n },\n {\n "volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00007": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 314572800\n }\n },\n {\n "volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00008": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 367001600\n }\n },\n {\n "volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00009": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 419430400\n }\n },\n {\n "volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00010": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 471859200\n }\n },\n {\n "volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00011": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 524288000\n }\n },\n {\n "volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00012": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 576716800\n }\n },\n {\n "volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00013": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 629145600\n }\n },\n {\n "volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00014": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 681574400\n }\n },\n {\n "volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00015": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 734003200\n }\n },\n {\n "volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00016": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 786432000\n }\n },\n {\n "volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00017": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 838860800\n }\n },\n {\n "volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00018": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 891289600\n }\n },\n {\n "volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00019": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 943718400\n }\n },\n {\n "volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00020": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 996147200\n }\n },\n {\n "volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00021": {\n "compression": "zlib",\n "length": 25165824,\n "md5": "77377273b0a4b61febdbf7bbf52b9db9",\n "offset": 1048576000\n }\n }\n ],\n "parent_id": null,\n "version": "1.0.0",\n "volume_id": "662c5cd4-cc35-4e90-a911-679b13dfa7a5",\n "volume_meta": "{\\"version\\": 2, \\"volume-base-metadata\\": {\\"id\\": \\"662c5cd4-cc35-4e90-a911-679b13dfa7a5\\", \\"_name_id\\": null, \\"use_quota\\": true, \\"ec2_id\\": null, \\"user_id\\": \\"0e48df3b35f84ee88136fd47ee531c1f\\", \\"project_id\\": \\"c642367dafaa45dd89634e7ec9822ef2\\", \\"snapshot_id\\": null, \\"cluster_name\\": null, \\"host\\": \\"np0005533992.novalocal@lvmdriver-1#lvmdriver-1\\", \\"size\\": 1, \\"availability_zone\\": \\"nova\\", \\"status\\": \\"backing-up\\", \\"attach_status\\": \\"attached\\", \\"migration_status\\": null, \\"scheduled_at\\": \\"2025-11-24T19:40:18.000000\\", \\"launched_at\\": \\"2025-11-24T19:40:19.000000\\", \\"terminated_at\\": null, \\"display_name\\": \\"tempest-VolumesBackupsTest-Volume-1459000726\\", \\"display_description\\": null, \\"provider_location\\": \\"38.102.83.12:3260,iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 0\\", \\"provider_auth\\": \\"CHAP rwfecJCEJ9JcRMgV8PNK ZDyQkauPVCK7HqgF\\", \\"provider_geometry\\": null, \\"provider_id\\": null, \\"volume_type_id\\": \\"1f8776b6-c527-4643-876a-506285adfc66\\", \\"source_volid\\": null, \\"encryption_key_id\\": null, \\"consistencygroup_id\\": null, \\"group_id\\": null, \\"bootable\\": false, \\"multiattach\\": false, \\"replication_status\\": null, \\"replication_extended_status\\": null, \\"replication_driver_data\\": null, \\"previous_status\\": \\"in-use\\", \\"service_uuid\\": \\"6d0941fd-19ec-48ae-82b1-f25526175227\\", \\"shared_targets\\": false, \\"deleted_at\\": null, \\"deleted\\": false, \\"created_at\\": \\"2025-11-24T19:40:18.000000\\", \\"updated_at\\": \\"2025-11-24T19:41:13.000000\\"}}"\n}'. {{(pid=122429) _write_metadata /opt/stack/cinder/cinder/backup/chunkeddriver.py:298}} Nov 24 14:41:33.503943 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] backup 519039aa-9b69-46f1-b149-a260745c08c7 finished. {{(pid=122429) _finalize_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:465}} Nov 24 14:41:33.568137 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sdb {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:33.595522 np0005533992.novalocal sudo[163378]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sdb Nov 24 14:41:33.919420 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sdb" returned: 0 in 0.351s {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:33.920615 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] ==> disconnect_volume: call "{'args': (, {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7', 'target_lun': 0, 'volume_id': '437b4cfc-c64e-4d3d-a3c2-23e3a27331b7', 'auth_method': 'CHAP', 'auth_username': 'U3BrRNjhH3FLQ4J9HExL', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False, 'enforce_multipath': False}, {'type': 'block', 'scsi_wwn': '3600140545226e0442c44d4fb9f1a02a7', 'path': '/dev/sdb'}), 'kwargs': {'force': True}}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:41:33.920615 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Acquiring lock "connect_volume" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:41:33.920934 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: waited 0.000s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:41:33.921921 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m discoverydb -o show -P 1 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:33.932475 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m discoverydb -o show -P 1" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:33.932902 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[01f5528b-7bce-481e-8466-64e5a5869c93]: (4, ('SENDTARGETS:\nNo targets found.\niSNS:\nNo targets found.\nSTATIC:\nTarget: iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5\n\tPortal: 38.102.83.12:3260,-1\n\t\tIface Name: default\nTarget: iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7\n\tPortal: 38.102.83.12:3260,-1\n\t\tIface Name: default\nFIRMWARE:\nNo targets found.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:33.933109 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] iscsiadm ['-m', 'discoverydb', '-o', 'show', '-P', 1]: stdout=SENDTARGETS: Nov 24 14:41:33.933109 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:41:33.933109 np0005533992.novalocal cinder-backup[122429]: iSNS: Nov 24 14:41:33.933109 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:41:33.933109 np0005533992.novalocal cinder-backup[122429]: STATIC: Nov 24 14:41:33.933109 np0005533992.novalocal cinder-backup[122429]: Target: iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 Nov 24 14:41:33.933109 np0005533992.novalocal cinder-backup[122429]: Portal: 38.102.83.12:3260,-1 Nov 24 14:41:33.933109 np0005533992.novalocal cinder-backup[122429]: Iface Name: default Nov 24 14:41:33.933109 np0005533992.novalocal cinder-backup[122429]: Target: iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 Nov 24 14:41:33.933109 np0005533992.novalocal cinder-backup[122429]: Portal: 38.102.83.12:3260,-1 Nov 24 14:41:33.933109 np0005533992.novalocal cinder-backup[122429]: Iface Name: default Nov 24 14:41:33.933109 np0005533992.novalocal cinder-backup[122429]: FIRMWARE: Nov 24 14:41:33.933109 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:41:33.933109 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:41:33.933549 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Regex to get portals from discoverydb: ^SENDTARGETS: Nov 24 14:41:33.933549 np0005533992.novalocal cinder-backup[122429]: .*?^DiscoveryAddress: 38.102.83.12,3260.*? Nov 24 14:41:33.933549 np0005533992.novalocal cinder-backup[122429]: (.*?)^(?:DiscoveryAddress|iSNS):.* {{(pid=122429) _get_discoverydb_portals /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:392}} Nov 24 14:41:33.933549 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Getting connected devices for (ips,iqns,luns)=[('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7', 0)] {{(pid=122429) _get_connection_devices /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:855}} Nov 24 14:41:33.934237 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:33.943930 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:33.944239 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[c4792aab-7368-4ba1-a6ae-a2c4aa69b90a]: (4, ('38.102.83.12:3260,4294967295 iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7\n38.102.83.12:3260,4294967295 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:33.945005 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:33.956095 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:33.956454 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[e9fdb236-957b-4082-9565-5f5c7b877695]: (4, ('tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash)\ntcp: [28] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:33.956667 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('-m', 'session'): stdout=tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash) Nov 24 14:41:33.956667 np0005533992.novalocal cinder-backup[122429]: tcp: [28] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 (non-flash) Nov 24 14:41:33.956667 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:41:33.956826 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] iscsi session list stdout=tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash) Nov 24 14:41:33.956826 np0005533992.novalocal cinder-backup[122429]: tcp: [28] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 (non-flash) Nov 24 14:41:33.956826 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:41:33.957422 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Resulting device map defaultdict(. at 0x7f5b729449a0>, {('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7'): ({'sdb'}, set())}) {{(pid=122429) _get_connection_devices /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:888}} Nov 24 14:41:33.957701 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Removing single pathed devices sdb {{(pid=122429) remove_connection /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:381}} Nov 24 14:41:33.958398 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd show status {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:33.965641 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] 'multipathd show status' failed. Not Retrying. {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:41:33.967907 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[5735b323-731b-4885-91a6-fea980a1b7d1]: [Errno 2] No such file or directory: 'multipathd' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:41:33.967907 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:41:33.967907 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:41:33.967907 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:41:33.967907 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:33.967907 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:41:33.967907 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:41:33.967907 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:33.967907 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root Nov 24 14:41:33.967907 np0005533992.novalocal cinder-backup[122429]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Nov 24 14:41:33.967907 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:33.967907 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute Nov 24 14:41:33.967907 np0005533992.novalocal cinder-backup[122429]: return putils.execute(on_execute=on_execute, Nov 24 14:41:33.967907 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:33.967907 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute Nov 24 14:41:33.967907 np0005533992.novalocal cinder-backup[122429]: obj = subprocess.Popen(cmd, Nov 24 14:41:33.967907 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:33.967907 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__ Nov 24 14:41:33.967907 np0005533992.novalocal cinder-backup[122429]: self._execute_child(args, executable, preexec_fn, close_fds, Nov 24 14:41:33.967907 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child Nov 24 14:41:33.967907 np0005533992.novalocal cinder-backup[122429]: raise child_exception_type(errno_num, err_msg, err_filename) Nov 24 14:41:33.967907 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: 'multipathd' Nov 24 14:41:33.969640 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[5735b323-731b-4885-91a6-fea980a1b7d1]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root\n return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute\n return putils.execute(on_execute=on_execute,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute\n obj = subprocess.Popen(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__\n self._execute_child(args, executable, preexec_fn, close_fds,\n File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child\n raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: \'multipathd\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:33.969923 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Flushing IO for device /dev/sdb {{(pid=122429) flush_device_io /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:441}} Nov 24 14:41:33.970611 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blockdev --flushbufs /dev/sdb {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:33.977679 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "blockdev --flushbufs /dev/sdb" returned: 0 in 0.007s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:33.978007 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[598dce93-6c6f-4d46-ab28-97762459db47]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:33.978176 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Remove SCSI device /dev/sdb with /sys/block/sdb/device/delete {{(pid=122429) remove_scsi_device /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:151}} Nov 24 14:41:33.978905 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/block/sdb/device/delete {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:34.035944 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/block/sdb/device/delete" returned: 0 in 0.056s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:34.036214 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[abafc44e-b3f6-4367-a6e5-79fc6a804a4c]: (4, ('1', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:34.036769 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Checking to see if SCSI volumes sdb have been removed. {{(pid=122429) wait_for_volumes_removal /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:159}} Nov 24 14:41:34.037441 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] SCSI volumes sdb have been removed. {{(pid=122429) wait_for_volumes_removal /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:169}} Nov 24 14:41:34.037728 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Disconnecting from: [('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7')] {{(pid=122429) _disconnect_connection /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1201}} Nov 24 14:41:34.038973 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 -p 38.102.83.12:3260 --op update -n node.startup -v manual {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:34.052199 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 -p 38.102.83.12:3260 --op update -n node.startup -v manual" returned: 0 in 0.013s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:34.052549 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[a72a4735-28ca-4763-bce4-ee4e6f5ee8bf]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:34.053068 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:41:34.054208 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 -p 38.102.83.12:3260 --logout {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:34.086828 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 -p 38.102.83.12:3260 --logout" returned: 0 in 0.032s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:34.087055 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[b5f34c92-ca81-472d-a6cf-388bc59bf6d3]: (4, ('Logging out of session [sid: 28, target: iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7, portal: 38.102.83.12,3260]\nLogout of [sid: 28, target: iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7, portal: 38.102.83.12,3260] successful.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:34.087588 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--logout',): stdout=Logging out of session [sid: 28, target: iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7, portal: 38.102.83.12,3260] Nov 24 14:41:34.087588 np0005533992.novalocal cinder-backup[122429]: Logout of [sid: 28, target: iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7, portal: 38.102.83.12,3260] successful. Nov 24 14:41:34.087588 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:41:34.088632 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 -p 38.102.83.12:3260 --op delete {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:34.100465 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-437b4cfc-c64e-4d3d-a3c2-23e3a27331b7 -p 38.102.83.12:3260 --op delete" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:34.100745 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[9d3f2266-fbf5-4fbb-b231-3c8f618a9273]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:34.101268 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'delete'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:41:34.101784 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: held 0.181s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:41:34.102071 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] <== disconnect_volume: return (181ms) None {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:41:37.643981 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-c13f8c5b-a53b-46a8-ba1e-b7eed865765e tempest-VolumesBackupsTest-1816644175 None] Create backup finished. backup: 519039aa-9b69-46f1-b149-a260745c08c7. Nov 24 14:41:38.656304 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Create backup started, backup: d148fb10-1012-4dca-918c-82420de677fb volume: 662c5cd4-cc35-4e90-a911-679b13dfa7a5. Nov 24 14:41:38.661104 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Call Volume Manager to get_backup_device for Backup(availability_zone='nova',container=None,created_at=2025-11-24T19:41:38Z,data_timestamp=2025-11-24T19:41:38Z,deleted=False,deleted_at=None,display_description=None,display_name=None,encryption_key_id=None,fail_reason=None,host='np0005533992.novalocal',id=d148fb10-1012-4dca-918c-82420de677fb,metadata={},num_dependent_backups=0,object_count=0,parent=Backup(519039aa-9b69-46f1-b149-a260745c08c7),parent_id='519039aa-9b69-46f1-b149-a260745c08c7',project_id='c642367dafaa45dd89634e7ec9822ef2',restore_volume_id=None,service='cinder.backup.drivers.swift.SwiftBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='0e48df3b35f84ee88136fd47ee531c1f',volume_id=662c5cd4-cc35-4e90-a911-679b13dfa7a5) Nov 24 14:41:40.090217 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Using swift URL https://38.102.83.12:8080/v1/AUTH_c642367dafaa45dd89634e7ec9822ef2 {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:286}} Nov 24 14:41:40.090785 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Connect to https://38.102.83.12:8080/v1/AUTH_ in "per_user" mode {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:287}} Nov 24 14:41:40.091058 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '38.102.83.12', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:41:40.092392 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:40.101386 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.009s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:40.101450 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[f2a219b8-5f01-4617-bd98-94d4df11da33]: (4, ('InitiatorName=iqn.1994-05.com.redhat:5b9c7bcf3f78\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:40.103859 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[5e3e5480-48e7-4934-a2e9-37fd5e69ae7b]: [Errno 2] No such file or directory: '/dev/scini' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:41:40.103859 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:41:40.103859 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:41:40.103859 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:41:40.103859 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:40.103859 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:41:40.103859 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:41:40.103859 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:40.103859 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 57, in get_guid Nov 24 14:41:40.103859 np0005533992.novalocal cinder-backup[122429]: with open_scini_device() as fd: Nov 24 14:41:40.103859 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:40.103859 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/contextlib.py", line 137, in __enter__ Nov 24 14:41:40.103859 np0005533992.novalocal cinder-backup[122429]: return next(self.gen) Nov 24 14:41:40.103859 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^ Nov 24 14:41:40.103859 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 40, in open_scini_device Nov 24 14:41:40.103859 np0005533992.novalocal cinder-backup[122429]: fd = os.open(SCINI_DEVICE_PATH, os.O_RDWR) Nov 24 14:41:40.103859 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:40.103859 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: '/dev/scini' Nov 24 14:41:40.104508 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[5e3e5480-48e7-4934-a2e9-37fd5e69ae7b]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 57, in get_guid\n with open_scini_device() as fd:\n ^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/contextlib.py", line 137, in __enter__\n return next(self.gen)\n ^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 40, in open_scini_device\n fd = os.open(SCINI_DEVICE_PATH, os.O_RDWR)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory: \'/dev/scini\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:40.105144 np0005533992.novalocal cinder-backup[122429]: ERROR os_brick.initiator.connectors.scaleio [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Error querying sdc guid: [Errno 2] No such file or directory: FileNotFoundError: [Errno 2] No such file or directory Nov 24 14:41:40.105689 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.scaleio [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Unable to find SDC guid: Error querying sdc guid: [Errno 2] No such file or directory Nov 24 14:41:40.106661 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt -v / -n -o SOURCE {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:40.116249 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt -v / -n -o SOURCE" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:40.116590 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[51e175c7-f2c0-47f9-a7e4-3c53d098d0c8]: (4, ('/dev/vda2\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:40.117685 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/vda2 -s UUID -o value {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:40.128143 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/vda2 -s UUID -o value" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:40.128386 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[3690b01e-f837-4a72-9555-5944fdcb9a30]: (4, ('3fc68afb-d7e4-475c-9164-ea34f338d965\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:40.129643 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[681cda78-9287-4e61-a968-5818560eabf0]: (4, 'e2c36df9-c117-4156-b18b-d355565d8e8e') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:40.130137 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Running cmd (subprocess): nvme version {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:40.144220 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] 'nvme version' failed. Not Retrying. {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:41:40.144606 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.nvmeof [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] nvme not present on system {{(pid=122429) nvme_present /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/nvmeof.py:784}} Nov 24 14:41:40.146490 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=122429) find_dsc /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:132}} Nov 24 14:41:40.147791 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.lightos [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Current host hostNQN nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e and IP(s) are ['38.102.83.12', 'fe80::f816:3eff:fe17:2fa8', '192.168.122.1', '172.24.5.1', '2001:db8::2', 'fe80::64fc:4aff:fe7b:de4b', 'fe80::fc16:3eff:fe29:10fc', 'fe80::d821:f8ff:fe9f:14e6'] Nov 24 14:41:40.148060 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=122429) get_connector_properties /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:109}} Nov 24 14:41:40.148326 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e dsc: {{(pid=122429) get_connector_properties /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:112}} Nov 24 14:41:40.148880 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] <== get_connector_properties: return (56ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '38.102.83.12', 'host': 'np0005533992.novalocal', 'multipath': False, 'enforce_multipath': False, 'initiator': 'iqn.1994-05.com.redhat:5b9c7bcf3f78', 'do_local_attach': False, 'uuid': '3fc68afb-d7e4-475c-9164-ea34f338d965', 'system uuid': 'e2c36df9-c117-4156-b18b-d355565d8e8e', 'nvme_native_multipath': False, 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e', 'found_dsc': '', 'host_ips': ['38.102.83.12', 'fe80::f816:3eff:fe17:2fa8', '192.168.122.1', '172.24.5.1', '2001:db8::2', 'fe80::64fc:4aff:fe7b:de4b', 'fe80::fc16:3eff:fe29:10fc', 'fe80::d821:f8ff:fe9f:14e6']} {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:41:43.658869 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connector [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Factory for iscsi on None {{(pid=122429) factory /usr/local/lib/python3.12/site-packages/os_brick/initiator/connector.py:281}} Nov 24 14:41:43.660090 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] ==> connect_volume: call "{'self': , 'connection_properties': {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076', 'target_lun': 0, 'volume_id': '50ba32d4-4e93-4371-a478-8c6baa9b5076', 'auth_method': 'CHAP', 'auth_username': 'tq5GAo8DNWv2jUiCKkys', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False}}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:41:43.660385 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Acquiring lock "connect_volume" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:41:43.661022 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:41:43.661286 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Connection properties {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076', 'target_lun': 0, 'volume_id': '50ba32d4-4e93-4371-a478-8c6baa9b5076', 'auth_method': 'CHAP', 'auth_username': 'tq5GAo8DNWv2jUiCKkys', 'auth_password': 'HV6XePsKkbiUbTo8', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False, 'enforce_multipath': False} {{(pid=122429) check_multipath /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:133}} Nov 24 14:41:43.662472 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd show status {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:43.669309 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] 'multipathd show status' failed. Not Retrying. {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:41:43.671380 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[66c9ad29-7989-42cd-94bb-a99c8763d562]: [Errno 2] No such file or directory: 'multipathd' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:41:43.671380 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:41:43.671380 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:41:43.671380 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:41:43.671380 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:43.671380 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:41:43.671380 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:41:43.671380 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:43.671380 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root Nov 24 14:41:43.671380 np0005533992.novalocal cinder-backup[122429]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Nov 24 14:41:43.671380 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:43.671380 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute Nov 24 14:41:43.671380 np0005533992.novalocal cinder-backup[122429]: return putils.execute(on_execute=on_execute, Nov 24 14:41:43.671380 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:43.671380 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute Nov 24 14:41:43.671380 np0005533992.novalocal cinder-backup[122429]: obj = subprocess.Popen(cmd, Nov 24 14:41:43.671380 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:43.671380 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__ Nov 24 14:41:43.671380 np0005533992.novalocal cinder-backup[122429]: self._execute_child(args, executable, preexec_fn, close_fds, Nov 24 14:41:43.671380 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child Nov 24 14:41:43.671380 np0005533992.novalocal cinder-backup[122429]: raise child_exception_type(errno_num, err_msg, err_filename) Nov 24 14:41:43.671380 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: 'multipathd' Nov 24 14:41:43.673034 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[66c9ad29-7989-42cd-94bb-a99c8763d562]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root\n return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute\n return putils.execute(on_execute=on_execute,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute\n obj = subprocess.Popen(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__\n self._execute_child(args, executable, preexec_fn, close_fds,\n File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child\n raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: \'multipathd\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:43.673818 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Acquiring lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:41:43.674417 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:41:43.674772 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Trying to connect to iSCSI portal 38.102.83.12:3260 Nov 24 14:41:43.675988 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 -p 38.102.83.12:3260 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:43.688072 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 -p 38.102.83.12:3260" returned: 21 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:43.688288 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[a8790e7a-e524-43f2-80f3-dc991512bb1a]: (4, ('', 'iscsiadm: No records found\n')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:43.688812 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] iscsiadm (): stdout= stderr=iscsiadm: No records found Nov 24 14:41:43.688812 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:41:43.689895 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 -p 38.102.83.12:3260 --interface default --op new {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:43.702138 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 -p 38.102.83.12:3260 --interface default --op new" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:43.702346 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[7c71b591-7695-43a6-868a-81d10bc1f721]: (4, ('New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 38.102.83.12,3260,-1 iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076] added\n', 'iscsiadm: Config file line 381 does not have value\niscsiadm: Config file line 381 does not have value\niscsiadm: Config file line 381 does not have value\n')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:43.702845 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--interface', 'default', '--op', 'new'): stdout=New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 38.102.83.12,3260,-1 iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076] added Nov 24 14:41:43.702845 np0005533992.novalocal cinder-backup[122429]: stderr=iscsiadm: Config file line 381 does not have value Nov 24 14:41:43.702845 np0005533992.novalocal cinder-backup[122429]: iscsiadm: Config file line 381 does not have value Nov 24 14:41:43.702845 np0005533992.novalocal cinder-backup[122429]: iscsiadm: Config file line 381 does not have value Nov 24 14:41:43.702845 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:41:43.703105 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Retrying to connect to iSCSI portal 38.102.83.12:3260 {{(pid=122429) _connect_to_iscsi_portal_unsafe /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1133}} Nov 24 14:41:43.703733 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Finished call to 'os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe' after 0.029(s), this was the 1st time calling it. {{(pid=122429) log_it /usr/local/lib/python3.12/site-packages/tenacity/after.py:44}} Nov 24 14:41:43.704050 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Retrying os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe in 1.0 seconds as it raised BrickException: An unknown exception occurred.. {{(pid=122429) log_it /usr/local/lib/python3.12/site-packages/tenacity/before_sleep.py:65}} Nov 24 14:41:44.705853 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Trying to connect to iSCSI portal 38.102.83.12:3260 Nov 24 14:41:44.708029 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 -p 38.102.83.12:3260 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:44.719245 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 -p 38.102.83.12:3260" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:44.719605 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[42376b3e-68a8-47b7-a74b-8d873c7950f7]: (4, ('# BEGIN RECORD 6.2.1.11\nnode.name = iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076\nnode.tpgt = -1\nnode.startup = automatic\nnode.leading_login = No\niface.iscsi_ifacename = default\niface.net_ifacename = \niface.ipaddress = \niface.prefix_len = 0\niface.hwaddress = \niface.transport_name = tcp\niface.initiatorname = \niface.state = \niface.vlan_id = 0\niface.vlan_priority = 0\niface.vlan_state = \niface.iface_num = 0\niface.mtu = 0\niface.port = 0\niface.bootproto = \niface.subnet_mask = \niface.gateway = \niface.dhcp_alt_client_id_state = \niface.dhcp_alt_client_id = \niface.dhcp_dns = \niface.dhcp_learn_iqn = \niface.dhcp_req_vendor_id_state = \niface.dhcp_vendor_id_state = \niface.dhcp_vendor_id = \niface.dhcp_slp_da = \niface.fragmentation = \niface.gratuitous_arp = \niface.incoming_forwarding = \niface.tos_state = \niface.tos = 0\niface.ttl = 0\niface.delayed_ack = \niface.tcp_nagle = \niface.tcp_wsf_state = \niface.tcp_wsf = 0\niface.tcp_timer_scale = 0\niface.tcp_timestamp = \niface.redirect = \niface.def_task_mgmt_timeout = 0\niface.header_digest = \niface.data_digest = \niface.immediate_data = \niface.initial_r2t = \niface.data_seq_inorder = \niface.data_pdu_inorder = \niface.erl = 0\niface.max_receive_data_len = 0\niface.first_burst_len = 0\niface.max_outstanding_r2t = 0\niface.max_burst_len = 0\niface.chap_auth = \niface.bidi_chap = \niface.strict_login_compliance = \niface.discovery_auth = \niface.discovery_logout = \nnode.discovery_address = \nnode.discovery_port = 0\nnode.discovery_type = fw\nnode.session.initial_cmdsn = 0\nnode.session.initial_login_retry_max = 8\nnode.session.xmit_thread_priority = 0\nnode.session.cmds_max = 128\nnode.session.queue_depth = 32\nnode.session.nr_sessions = 1\nnode.session.auth.authmethod = None\nnode.session.auth.username = \nnode.session.auth.password = \nnode.session.auth.username_in = \nnode.session.auth.password_in = \nnode.session.auth.chap_algs = SHA3-256,SHA256\nnode.session.timeo.replacement_timeout = 120\nnode.session.err_timeo.abort_timeout = 15\nnode.session.err_timeo.lu_reset_timeout = 30\nnode.session.err_timeo.tgt_reset_timeout = 30\nnode.session.err_timeo.host_reset_timeout = 60\nnode.session.iscsi.FastAbort = Yes\nnode.session.iscsi.InitialR2T = No\nnode.session.iscsi.ImmediateData = Yes\nnode.session.iscsi.FirstBurstLength = 262144\nnode.session.iscsi.MaxBurstLength = 16776192\nnode.session.iscsi.DefaultTime2Retain = 0\nnode.session.iscsi.DefaultTime2Wait = 2\nnode.session.iscsi.MaxConnections = 1\nnode.session.iscsi.MaxOutstandingR2T = 1\nnode.session.iscsi.ERL = 0\nnode.session.scan = auto\nnode.session.reopen_max = 0\nnode.session.conn_reopen_log_freq = 1\nnode.conn[0].address = 38.102.83.12\nnode.conn[0].port = 3260\nnode.conn[0].startup = manual\nnode.conn[0].tcp.window_size = 524288\nnode.conn[0].tcp.type_of_service = 0\nnode.conn[0].timeo.logout_timeout = 15\nnode.conn[0].timeo.login_timeout = 15\nnode.conn[0].timeo.auth_timeout = 45\nnode.conn[0].timeo.noop_out_interval = 5\nnode.conn[0].timeo.noop_out_timeout = 5\nnode.conn[0].iscsi.MaxXmitDataSegmentLength = 0\nnode.conn[0].iscsi.MaxRecvDataSegmentLength = 262144\nnode.conn[0].iscsi.HeaderDigest = None\nnode.conn[0].iscsi.DataDigest = None\nnode.conn[0].iscsi.IFMarker = No\nnode.conn[0].iscsi.OFMarker = No\n# END RECORD\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] iscsiadm (): stdout=# BEGIN RECORD 6.2.1.11 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.name = iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.tpgt = -1 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.startup = automatic Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.leading_login = No Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.iscsi_ifacename = default Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.net_ifacename = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.ipaddress = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.prefix_len = 0 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.hwaddress = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.transport_name = tcp Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.initiatorname = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.state = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.vlan_id = 0 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.vlan_priority = 0 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.vlan_state = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.iface_num = 0 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.mtu = 0 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.port = 0 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.bootproto = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.subnet_mask = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.gateway = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_alt_client_id_state = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_alt_client_id = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_dns = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_learn_iqn = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_req_vendor_id_state = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_vendor_id_state = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_vendor_id = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_slp_da = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.fragmentation = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.gratuitous_arp = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.incoming_forwarding = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.tos_state = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.tos = 0 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.ttl = 0 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.delayed_ack = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.tcp_nagle = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.tcp_wsf_state = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.tcp_wsf = 0 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.tcp_timer_scale = 0 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.tcp_timestamp = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.redirect = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.def_task_mgmt_timeout = 0 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.header_digest = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.data_digest = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.immediate_data = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.initial_r2t = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.data_seq_inorder = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.data_pdu_inorder = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.erl = 0 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.max_receive_data_len = 0 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.first_burst_len = 0 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.max_outstanding_r2t = 0 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.max_burst_len = 0 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.chap_auth = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.bidi_chap = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.strict_login_compliance = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.discovery_auth = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: iface.discovery_logout = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.discovery_address = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.discovery_port = 0 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.discovery_type = fw Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.initial_cmdsn = 0 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.initial_login_retry_max = 8 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.xmit_thread_priority = 0 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.cmds_max = 128 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.queue_depth = 32 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.nr_sessions = 1 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.auth.authmethod = None Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.auth.username = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.auth.password = *** Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.auth.username_in = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.auth.password_in = Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.auth.chap_algs = SHA3-256,SHA256 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.timeo.replacement_timeout = 120 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.abort_timeout = 15 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.lu_reset_timeout = 30 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.tgt_reset_timeout = 30 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.host_reset_timeout = 60 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.FastAbort = Yes Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.InitialR2T = No Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.ImmediateData = Yes Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.FirstBurstLength = 262144 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxBurstLength = 16776192 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.DefaultTime2Retain = 0 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.DefaultTime2Wait = 2 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxConnections = 1 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxOutstandingR2T = 1 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.ERL = 0 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.scan = auto Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.reopen_max = 0 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.session.conn_reopen_log_freq = 1 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.conn[0].address = 38.102.83.12 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.conn[0].port = 3260 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.conn[0].startup = manual Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.conn[0].tcp.window_size = 524288 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.conn[0].tcp.type_of_service = 0 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.logout_timeout = 15 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.login_timeout = 15 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.auth_timeout = 45 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.noop_out_interval = 5 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.noop_out_timeout = 5 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.MaxXmitDataSegmentLength = 0 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144 Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.HeaderDigest = None Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.DataDigest = None Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.IFMarker = No Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.OFMarker = No Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: # END RECORD Nov 24 14:41:44.721488 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:41:44.724361 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 -p 38.102.83.12:3260 --op update -n node.session.scan -v manual {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:44.734997 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 -p 38.102.83.12:3260 --op update -n node.session.scan -v manual" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:44.735237 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[31b23adc-fca3-40ec-bb22-24cd5cb6f186]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:44.735706 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.scan', '-v', 'manual'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:41:44.736805 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 -p 38.102.83.12:3260 --op update -n node.session.auth.authmethod -v CHAP {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:44.747104 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 -p 38.102.83.12:3260 --op update -n node.session.auth.authmethod -v CHAP" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:44.747487 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[7fa0982a-dda7-4149-ab31-77a6da54bf3b]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:44.747934 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.authmethod', '-v', 'CHAP'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:41:44.749151 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 -p 38.102.83.12:3260 --op update -n node.session.auth.username -v tq5GAo8DNWv2jUiCKkys {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:44.759964 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 -p 38.102.83.12:3260 --op update -n node.session.auth.username -v tq5GAo8DNWv2jUiCKkys" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:44.760285 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[187a293d-9d4a-4dbd-8264-183e1c733c2e]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:44.760725 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.username', '-v', 'tq5GAo8DNWv2jUiCKkys'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:41:44.761910 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 -p 38.102.83.12:3260 --op update -n node.session.auth.password -v *** {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:44.773234 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 -p 38.102.83.12:3260 --op update -n node.session.auth.password -v ***" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:44.773429 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[28ab3637-de67-440b-a45e-7d2ba148ffed]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:44.773985 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.password', '-v', '***'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:41:44.775119 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:44.786838 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:44.787322 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[2c3aed4e-fb35-45b2-9103-d91d62f9188e]: (4, ('tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:44.787761 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('-m', 'session'): stdout=tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash) Nov 24 14:41:44.787761 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:41:44.788032 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] iscsi session list stdout=tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash) Nov 24 14:41:44.788032 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:41:44.789046 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 -p 38.102.83.12:3260 --login {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:44.809173 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 -p 38.102.83.12:3260 --login" returned: 0 in 0.020s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:44.810510 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[9d638ebf-6c8f-4f03-a8f1-f03eee1e17de]: (4, ('Login to [iface: default, target: iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076, portal: 38.102.83.12,3260] successful.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:44.811004 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--login',): stdout=Login to [iface: default, target: iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076, portal: 38.102.83.12,3260] successful. Nov 24 14:41:44.811004 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:41:44.812202 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 -p 38.102.83.12:3260 --op update -n node.startup -v automatic {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:44.828300 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 -p 38.102.83.12:3260 --op update -n node.startup -v automatic" returned: 0 in 0.016s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:44.828674 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[862442f9-13f7-4445-8e92-0934a7c50556]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:44.829102 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:41:44.831230 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:44.847585 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.016s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:44.847940 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[1f48b138-9927-4da7-9de4-186445c883c9]: (4, ('tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash)\ntcp: [29] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:44.848337 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('-m', 'session'): stdout=tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash) Nov 24 14:41:44.848337 np0005533992.novalocal cinder-backup[122429]: tcp: [29] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 (non-flash) Nov 24 14:41:44.848337 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:41:44.848648 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] iscsi session list stdout=tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash) Nov 24 14:41:44.848648 np0005533992.novalocal cinder-backup[122429]: tcp: [29] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 (non-flash) Nov 24 14:41:44.848648 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:41:44.849142 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: held 1.175s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:41:44.849400 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Connected to 38.102.83.12:3260 {{(pid=122429) _connect_vol /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:659}} Nov 24 14:41:44.850222 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] HCTL ('3', '-', '-', 0) found on session 29 with lun 0 {{(pid=122429) get_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:788}} Nov 24 14:41:44.850482 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Scanning host 3 c: -, t: -, l: 0) {{(pid=122429) scan_iscsi /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:814}} Nov 24 14:41:44.851424 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/class/scsi_host/host3/scan {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:44.881743 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/class/scsi_host/host3/scan" returned: 0 in 0.030s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:44.882077 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[d64abad3-c4a0-45f9-9156-11adfd980aba]: (4, ('- - 0', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:44.882837 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Searching for a device in session 29 and hctl ['3', '*', '*', 0] yield: None {{(pid=122429) device_name_by_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:808}} Nov 24 14:41:45.884678 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Searching for a device in session 29 and hctl ['3', '*', '*', 0] yield: sdb {{(pid=122429) device_name_by_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:808}} Nov 24 14:41:45.885488 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Connected to sdb using {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076', 'target_lun': 0, 'volume_id': '50ba32d4-4e93-4371-a478-8c6baa9b5076', 'auth_method': 'CHAP', 'auth_username': 'tq5GAo8DNWv2jUiCKkys', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False} {{(pid=122429) _connect_vol /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:687}} Nov 24 14:41:45.886266 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" :: held 2.225s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:41:45.886629 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] <== connect_volume: return (2225ms) {'type': 'block', 'scsi_wwn': '36001405c68451d6c6f14ae7ace60a647', 'path': '/dev/sdb'} {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:41:45.887373 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sdb {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:45.916623 np0005533992.novalocal sudo[163633]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sdb Nov 24 14:41:46.220412 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sdb" returned: 0 in 0.333s {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:46.227170 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_db.sqlalchemy.engines [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=122429) _check_effective_sql_mode /usr/local/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:397}} Nov 24 14:41:46.241364 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] _read_sha256file started, container name: tempest-volumesbackupstest-backup-container-601227527, sha256 filename: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7_sha256file. {{(pid=122429) _read_sha256file /opt/stack/cinder/cinder/backup/chunkeddriver.py:336}} Nov 24 14:41:46.290008 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] _read_sha256file finished. {{(pid=122429) _read_sha256file /opt/stack/cinder/cinder/backup/chunkeddriver.py:343}} Nov 24 14:41:46.299255 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] _create_container started, container: volumebackups,backup: d148fb10-1012-4dca-918c-82420de677fb. {{(pid=122429) _create_container /opt/stack/cinder/cinder/backup/chunkeddriver.py:252}} Nov 24 14:41:46.367308 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] generate_object_name_prefix: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194146/az_nova_backup_d148fb10-1012-4dca-918c-82420de677fb {{(pid=122429) _generate_object_name_prefix /opt/stack/cinder/cinder/backup/drivers/swift.py:433}} Nov 24 14:41:46.373322 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] starting backup of volume: 662c5cd4-cc35-4e90-a911-679b13dfa7a5, volume size: 1073741824, object names prefix volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194146/az_nova_backup_d148fb10-1012-4dca-918c-82420de677fb, availability zone: nova {{(pid=122429) _prepare_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:362}} Nov 24 14:41:46.429255 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:41:46.429930 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:41:51.367878 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Getting metadata type 'volume-base-metadata' {{(pid=122429) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Nov 24 14:41:51.375200 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=122429) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Nov 24 14:41:51.375506 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Getting metadata type 'volume-metadata' {{(pid=122429) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Nov 24 14:41:51.380508 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] No metadata type 'volume-metadata' available {{(pid=122429) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Nov 24 14:41:51.380851 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Getting metadata type 'volume-glance-metadata' {{(pid=122429) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Nov 24 14:41:51.385106 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] No metadata type 'volume-glance-metadata' available {{(pid=122429) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Nov 24 14:41:51.385588 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] _write_sha256file started, container name: volumebackups, sha256file filename: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194146/az_nova_backup_d148fb10-1012-4dca-918c-82420de677fb_sha256file. {{(pid=122429) _write_sha256file /opt/stack/cinder/cinder/backup/chunkeddriver.py:302}} Nov 24 14:41:51.475543 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] _write_sha256file finished. {{(pid=122429) _write_sha256file /opt/stack/cinder/cinder/backup/chunkeddriver.py:318}} Nov 24 14:41:51.476113 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] _write_metadata started, container name: volumebackups, metadata filename: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194146/az_nova_backup_d148fb10-1012-4dca-918c-82420de677fb_metadata. {{(pid=122429) _write_metadata /opt/stack/cinder/cinder/backup/chunkeddriver.py:279}} Nov 24 14:41:51.509650 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] _write_metadata finished. Metadata: b'{\n "backup_description": null,\n "backup_id": "d148fb10-1012-4dca-918c-82420de677fb",\n "backup_name": null,\n "created_at": "2025-11-24 19:41:38+00:00",\n "objects": [],\n "parent_id": "519039aa-9b69-46f1-b149-a260745c08c7",\n "version": "1.0.0",\n "volume_id": "662c5cd4-cc35-4e90-a911-679b13dfa7a5",\n "volume_meta": "{\\"version\\": 2, \\"volume-base-metadata\\": {\\"id\\": \\"662c5cd4-cc35-4e90-a911-679b13dfa7a5\\", \\"_name_id\\": null, \\"use_quota\\": true, \\"ec2_id\\": null, \\"user_id\\": \\"0e48df3b35f84ee88136fd47ee531c1f\\", \\"project_id\\": \\"c642367dafaa45dd89634e7ec9822ef2\\", \\"snapshot_id\\": null, \\"cluster_name\\": null, \\"host\\": \\"np0005533992.novalocal@lvmdriver-1#lvmdriver-1\\", \\"size\\": 1, \\"availability_zone\\": \\"nova\\", \\"status\\": \\"backing-up\\", \\"attach_status\\": \\"attached\\", \\"migration_status\\": null, \\"scheduled_at\\": \\"2025-11-24T19:40:18.000000\\", \\"launched_at\\": \\"2025-11-24T19:40:19.000000\\", \\"terminated_at\\": null, \\"display_name\\": \\"tempest-VolumesBackupsTest-Volume-1459000726\\", \\"display_description\\": null, \\"provider_location\\": \\"38.102.83.12:3260,iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 0\\", \\"provider_auth\\": \\"CHAP rwfecJCEJ9JcRMgV8PNK ZDyQkauPVCK7HqgF\\", \\"provider_geometry\\": null, \\"provider_id\\": null, \\"volume_type_id\\": \\"1f8776b6-c527-4643-876a-506285adfc66\\", \\"source_volid\\": null, \\"encryption_key_id\\": null, \\"consistencygroup_id\\": null, \\"group_id\\": null, \\"bootable\\": false, \\"multiattach\\": false, \\"replication_status\\": null, \\"replication_extended_status\\": null, \\"replication_driver_data\\": null, \\"previous_status\\": \\"in-use\\", \\"service_uuid\\": \\"6d0941fd-19ec-48ae-82b1-f25526175227\\", \\"shared_targets\\": false, \\"deleted_at\\": null, \\"deleted\\": false, \\"created_at\\": \\"2025-11-24T19:40:18.000000\\", \\"updated_at\\": \\"2025-11-24T19:41:38.000000\\"}}"\n}'. {{(pid=122429) _write_metadata /opt/stack/cinder/cinder/backup/chunkeddriver.py:298}} Nov 24 14:41:51.514270 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] backup d148fb10-1012-4dca-918c-82420de677fb finished. {{(pid=122429) _finalize_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:465}} Nov 24 14:41:51.607033 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sdb {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:51.633101 np0005533992.novalocal sudo[163646]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sdb Nov 24 14:41:51.966114 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sdb" returned: 0 in 0.359s {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:51.967706 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] ==> disconnect_volume: call "{'args': (, {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076', 'target_lun': 0, 'volume_id': '50ba32d4-4e93-4371-a478-8c6baa9b5076', 'auth_method': 'CHAP', 'auth_username': 'tq5GAo8DNWv2jUiCKkys', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False, 'enforce_multipath': False}, {'type': 'block', 'scsi_wwn': '36001405c68451d6c6f14ae7ace60a647', 'path': '/dev/sdb'}), 'kwargs': {'force': True}}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:41:51.968033 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Acquiring lock "connect_volume" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:41:51.968726 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:41:51.970428 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m discoverydb -o show -P 1 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:51.978001 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m discoverydb -o show -P 1" returned: 0 in 0.008s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:51.978164 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[4b64c0a8-ffac-43b5-a296-e52604c875e9]: (4, ('SENDTARGETS:\nNo targets found.\niSNS:\nNo targets found.\nSTATIC:\nTarget: iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5\n\tPortal: 38.102.83.12:3260,-1\n\t\tIface Name: default\nTarget: iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076\n\tPortal: 38.102.83.12:3260,-1\n\t\tIface Name: default\nFIRMWARE:\nNo targets found.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:51.978516 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ['-m', 'discoverydb', '-o', 'show', '-P', 1]: stdout=SENDTARGETS: Nov 24 14:41:51.978516 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:41:51.978516 np0005533992.novalocal cinder-backup[122429]: iSNS: Nov 24 14:41:51.978516 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:41:51.978516 np0005533992.novalocal cinder-backup[122429]: STATIC: Nov 24 14:41:51.978516 np0005533992.novalocal cinder-backup[122429]: Target: iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 Nov 24 14:41:51.978516 np0005533992.novalocal cinder-backup[122429]: Portal: 38.102.83.12:3260,-1 Nov 24 14:41:51.978516 np0005533992.novalocal cinder-backup[122429]: Iface Name: default Nov 24 14:41:51.978516 np0005533992.novalocal cinder-backup[122429]: Target: iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 Nov 24 14:41:51.978516 np0005533992.novalocal cinder-backup[122429]: Portal: 38.102.83.12:3260,-1 Nov 24 14:41:51.978516 np0005533992.novalocal cinder-backup[122429]: Iface Name: default Nov 24 14:41:51.978516 np0005533992.novalocal cinder-backup[122429]: FIRMWARE: Nov 24 14:41:51.978516 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:41:51.978516 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:41:51.978797 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Regex to get portals from discoverydb: ^SENDTARGETS: Nov 24 14:41:51.978797 np0005533992.novalocal cinder-backup[122429]: .*?^DiscoveryAddress: 38.102.83.12,3260.*? Nov 24 14:41:51.978797 np0005533992.novalocal cinder-backup[122429]: (.*?)^(?:DiscoveryAddress|iSNS):.* {{(pid=122429) _get_discoverydb_portals /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:392}} Nov 24 14:41:51.979056 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Getting connected devices for (ips,iqns,luns)=[('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076', 0)] {{(pid=122429) _get_connection_devices /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:855}} Nov 24 14:41:51.979788 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:51.989323 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:51.989542 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[4cb39d6a-c94c-4899-aee9-e7c23acbc87f]: (4, ('38.102.83.12:3260,4294967295 iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076\n38.102.83.12:3260,4294967295 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:51.990327 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:52.000012 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:52.000319 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[a6247b36-b60f-455a-84eb-d6fe394cf2f4]: (4, ('tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash)\ntcp: [29] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:52.000641 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('-m', 'session'): stdout=tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash) Nov 24 14:41:52.000641 np0005533992.novalocal cinder-backup[122429]: tcp: [29] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 (non-flash) Nov 24 14:41:52.000641 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:41:52.000852 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] iscsi session list stdout=tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash) Nov 24 14:41:52.000852 np0005533992.novalocal cinder-backup[122429]: tcp: [29] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 (non-flash) Nov 24 14:41:52.000852 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:41:52.001380 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Resulting device map defaultdict(. at 0x7f5b90c73b00>, {('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076'): ({'sdb'}, set())}) {{(pid=122429) _get_connection_devices /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:888}} Nov 24 14:41:52.001678 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Removing single pathed devices sdb {{(pid=122429) remove_connection /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:381}} Nov 24 14:41:52.002314 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd show status {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:52.005810 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] 'multipathd show status' failed. Not Retrying. {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:41:52.006914 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[96abd36f-e2a3-4954-84fb-12e1bfb6b4c7]: [Errno 2] No such file or directory: 'multipathd' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:41:52.006914 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:41:52.006914 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:41:52.006914 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:41:52.006914 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:52.006914 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:41:52.006914 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:41:52.006914 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:52.006914 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root Nov 24 14:41:52.006914 np0005533992.novalocal cinder-backup[122429]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Nov 24 14:41:52.006914 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:52.006914 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute Nov 24 14:41:52.006914 np0005533992.novalocal cinder-backup[122429]: return putils.execute(on_execute=on_execute, Nov 24 14:41:52.006914 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:52.006914 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute Nov 24 14:41:52.006914 np0005533992.novalocal cinder-backup[122429]: obj = subprocess.Popen(cmd, Nov 24 14:41:52.006914 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:52.006914 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__ Nov 24 14:41:52.006914 np0005533992.novalocal cinder-backup[122429]: self._execute_child(args, executable, preexec_fn, close_fds, Nov 24 14:41:52.006914 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child Nov 24 14:41:52.006914 np0005533992.novalocal cinder-backup[122429]: raise child_exception_type(errno_num, err_msg, err_filename) Nov 24 14:41:52.006914 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: 'multipathd' Nov 24 14:41:52.008436 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[96abd36f-e2a3-4954-84fb-12e1bfb6b4c7]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root\n return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute\n return putils.execute(on_execute=on_execute,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute\n obj = subprocess.Popen(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__\n self._execute_child(args, executable, preexec_fn, close_fds,\n File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child\n raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: \'multipathd\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:52.008436 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Flushing IO for device /dev/sdb {{(pid=122429) flush_device_io /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:441}} Nov 24 14:41:52.009060 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blockdev --flushbufs /dev/sdb {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:52.017025 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "blockdev --flushbufs /dev/sdb" returned: 0 in 0.008s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:52.017237 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[483c6ff0-7097-490d-8df9-d2aeb71d812c]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:52.017481 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Remove SCSI device /dev/sdb with /sys/block/sdb/device/delete {{(pid=122429) remove_scsi_device /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:151}} Nov 24 14:41:52.018084 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/block/sdb/device/delete {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:52.077726 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/block/sdb/device/delete" returned: 0 in 0.059s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:52.077846 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[54a39dc1-9658-4b0e-9fd7-4d8d0904b6e6]: (4, ('1', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:52.078147 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Checking to see if SCSI volumes sdb have been removed. {{(pid=122429) wait_for_volumes_removal /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:159}} Nov 24 14:41:52.078307 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] SCSI volumes sdb have been removed. {{(pid=122429) wait_for_volumes_removal /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:169}} Nov 24 14:41:52.078597 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Disconnecting from: [('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076')] {{(pid=122429) _disconnect_connection /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1201}} Nov 24 14:41:52.079216 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 -p 38.102.83.12:3260 --op update -n node.startup -v manual {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:52.091241 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 -p 38.102.83.12:3260 --op update -n node.startup -v manual" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:52.092045 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[efe8a6db-437c-482a-afbe-d12620f42eeb]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:52.092630 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:41:52.093878 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 -p 38.102.83.12:3260 --logout {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:52.124322 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 -p 38.102.83.12:3260 --logout" returned: 0 in 0.030s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:52.124612 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[09808eac-e95e-46b5-839f-3b34187067b8]: (4, ('Logging out of session [sid: 29, target: iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076, portal: 38.102.83.12,3260]\nLogout of [sid: 29, target: iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076, portal: 38.102.83.12,3260] successful.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:52.125101 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--logout',): stdout=Logging out of session [sid: 29, target: iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076, portal: 38.102.83.12,3260] Nov 24 14:41:52.125101 np0005533992.novalocal cinder-backup[122429]: Logout of [sid: 29, target: iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076, portal: 38.102.83.12,3260] successful. Nov 24 14:41:52.125101 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:41:52.126485 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 -p 38.102.83.12:3260 --op delete {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:52.138439 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-50ba32d4-4e93-4371-a478-8c6baa9b5076 -p 38.102.83.12:3260 --op delete" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:52.138734 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[80c5f5ad-b235-4d61-8b98-6ca0402a9321]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:52.139251 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'delete'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:41:52.139742 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: held 0.171s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:41:52.140033 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] <== disconnect_volume: return (172ms) None {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:41:55.693601 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-e06e5d55-4f92-4a3a-9388-a6dc7b1cc5b6 tempest-VolumesBackupsTest-1816644175 None] Create backup finished. backup: d148fb10-1012-4dca-918c-82420de677fb. Nov 24 14:41:56.203107 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-4ace9bb5-c409-4e53-aa7c-214446058c06 tempest-VolumesBackupsTest-1816644175 None] Delete backup started, backup: d148fb10-1012-4dca-918c-82420de677fb. Nov 24 14:41:56.204579 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-4ace9bb5-c409-4e53-aa7c-214446058c06 tempest-VolumesBackupsTest-1816644175 None] Using swift URL https://38.102.83.12:8080/v1/AUTH_c642367dafaa45dd89634e7ec9822ef2 {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:286}} Nov 24 14:41:56.204859 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-4ace9bb5-c409-4e53-aa7c-214446058c06 tempest-VolumesBackupsTest-1816644175 None] Connect to https://38.102.83.12:8080/v1/AUTH_ in "per_user" mode {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:287}} Nov 24 14:41:56.205164 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ace9bb5-c409-4e53-aa7c-214446058c06 tempest-VolumesBackupsTest-1816644175 None] delete started, backup: d148fb10-1012-4dca-918c-82420de677fb, container: volumebackups, prefix: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194146/az_nova_backup_d148fb10-1012-4dca-918c-82420de677fb. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:844}} Nov 24 14:41:56.266551 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ace9bb5-c409-4e53-aa7c-214446058c06 tempest-VolumesBackupsTest-1816644175 None] generated object list: ['volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194146/az_nova_backup_d148fb10-1012-4dca-918c-82420de677fb_metadata', 'volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194146/az_nova_backup_d148fb10-1012-4dca-918c-82420de677fb_sha256file']. {{(pid=122429) _generate_object_names /opt/stack/cinder/cinder/backup/chunkeddriver.py:263}} Nov 24 14:41:56.289660 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ace9bb5-c409-4e53-aa7c-214446058c06 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194146/az_nova_backup_d148fb10-1012-4dca-918c-82420de677fb_metadata in container: volumebackups. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:41:56.316941 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ace9bb5-c409-4e53-aa7c-214446058c06 tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194146/az_nova_backup_d148fb10-1012-4dca-918c-82420de677fb_sha256file in container: volumebackups. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:41:56.317395 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ace9bb5-c409-4e53-aa7c-214446058c06 tempest-VolumesBackupsTest-1816644175 None] delete d148fb10-1012-4dca-918c-82420de677fb finished. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:870}} Nov 24 14:41:56.322870 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_db.sqlalchemy.engines [None req-4ace9bb5-c409-4e53-aa7c-214446058c06 tempest-VolumesBackupsTest-1816644175 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=122429) _check_effective_sql_mode /usr/local/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:397}} Nov 24 14:41:56.345605 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.quota [None req-4ace9bb5-c409-4e53-aa7c-214446058c06 tempest-VolumesBackupsTest-1816644175 None] Created reservations ['53a10eb6-bbd8-47cc-ad37-169de41ae46c', 'f468d822-e0e1-4340-b5b2-c607b6788456'] {{(pid=122429) reserve /opt/stack/cinder/cinder/quota.py:752}} Nov 24 14:41:56.378244 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-4ace9bb5-c409-4e53-aa7c-214446058c06 tempest-VolumesBackupsTest-1816644175 None] Delete backup finished, backup d148fb10-1012-4dca-918c-82420de677fb deleted. Nov 24 14:41:57.376100 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Create backup started, backup: c21c7367-55a2-43d2-8456-15fae70970ac volume: 662c5cd4-cc35-4e90-a911-679b13dfa7a5. Nov 24 14:41:57.379268 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Call Volume Manager to get_backup_device for Backup(availability_zone='nova',container='tempest-volumesbackupstest-backup-container-1830262932',created_at=2025-11-24T19:41:57Z,data_timestamp=2025-11-24T19:41:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-1101742960',encryption_key_id=None,fail_reason=None,host='np0005533992.novalocal',id=c21c7367-55a2-43d2-8456-15fae70970ac,metadata={},num_dependent_backups=0,object_count=0,parent=Backup(519039aa-9b69-46f1-b149-a260745c08c7),parent_id='519039aa-9b69-46f1-b149-a260745c08c7',project_id='c642367dafaa45dd89634e7ec9822ef2',restore_volume_id=None,service='cinder.backup.drivers.swift.SwiftBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='0e48df3b35f84ee88136fd47ee531c1f',volume_id=662c5cd4-cc35-4e90-a911-679b13dfa7a5) Nov 24 14:41:58.726503 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Using swift URL https://38.102.83.12:8080/v1/AUTH_c642367dafaa45dd89634e7ec9822ef2 {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:286}} Nov 24 14:41:58.726970 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Connect to https://38.102.83.12:8080/v1/AUTH_ in "per_user" mode {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:287}} Nov 24 14:41:58.727026 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '38.102.83.12', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:41:58.728082 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:58.736275 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.008s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:58.736624 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[7657c68d-d857-4746-9b69-1ade6ce65114]: (4, ('InitiatorName=iqn.1994-05.com.redhat:5b9c7bcf3f78\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:58.738885 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[0e8153ed-30ca-45c9-94a3-edaf3ef63212]: [Errno 2] No such file or directory: '/dev/scini' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:41:58.738885 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:41:58.738885 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:41:58.738885 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:41:58.738885 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:58.738885 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:41:58.738885 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:41:58.738885 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:58.738885 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 57, in get_guid Nov 24 14:41:58.738885 np0005533992.novalocal cinder-backup[122429]: with open_scini_device() as fd: Nov 24 14:41:58.738885 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:58.738885 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/contextlib.py", line 137, in __enter__ Nov 24 14:41:58.738885 np0005533992.novalocal cinder-backup[122429]: return next(self.gen) Nov 24 14:41:58.738885 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^ Nov 24 14:41:58.738885 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 40, in open_scini_device Nov 24 14:41:58.738885 np0005533992.novalocal cinder-backup[122429]: fd = os.open(SCINI_DEVICE_PATH, os.O_RDWR) Nov 24 14:41:58.738885 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:41:58.738885 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: '/dev/scini' Nov 24 14:41:58.740085 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[0e8153ed-30ca-45c9-94a3-edaf3ef63212]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 57, in get_guid\n with open_scini_device() as fd:\n ^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/contextlib.py", line 137, in __enter__\n return next(self.gen)\n ^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 40, in open_scini_device\n fd = os.open(SCINI_DEVICE_PATH, os.O_RDWR)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory: \'/dev/scini\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:58.740085 np0005533992.novalocal cinder-backup[122429]: ERROR os_brick.initiator.connectors.scaleio [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Error querying sdc guid: [Errno 2] No such file or directory: FileNotFoundError: [Errno 2] No such file or directory Nov 24 14:41:58.740352 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.scaleio [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Unable to find SDC guid: Error querying sdc guid: [Errno 2] No such file or directory Nov 24 14:41:58.741189 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt -v / -n -o SOURCE {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:58.751153 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt -v / -n -o SOURCE" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:58.751264 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[76c13c99-1a58-43ad-83bf-4406bf804a05]: (4, ('/dev/vda2\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:58.752140 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/vda2 -s UUID -o value {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:58.763267 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/vda2 -s UUID -o value" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:41:58.763628 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[5db33c46-7da7-4cd2-9f6a-b6eae5d6fcea]: (4, ('3fc68afb-d7e4-475c-9164-ea34f338d965\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:58.764620 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[937704c1-a4c7-462d-b5eb-00ae2780e1f1]: (4, 'e2c36df9-c117-4156-b18b-d355565d8e8e') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:41:58.764926 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Running cmd (subprocess): nvme version {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:41:58.777341 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] 'nvme version' failed. Not Retrying. {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:41:58.777628 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.nvmeof [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] nvme not present on system {{(pid=122429) nvme_present /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/nvmeof.py:784}} Nov 24 14:41:58.778975 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=122429) find_dsc /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:132}} Nov 24 14:41:58.779853 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.lightos [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Current host hostNQN nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e and IP(s) are ['38.102.83.12', 'fe80::f816:3eff:fe17:2fa8', '192.168.122.1', '172.24.5.1', '2001:db8::2', 'fe80::64fc:4aff:fe7b:de4b', 'fe80::fc16:3eff:fe29:10fc', 'fe80::d821:f8ff:fe9f:14e6'] Nov 24 14:41:58.780021 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=122429) get_connector_properties /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:109}} Nov 24 14:41:58.780181 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e dsc: {{(pid=122429) get_connector_properties /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:112}} Nov 24 14:41:58.780509 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] <== get_connector_properties: return (52ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '38.102.83.12', 'host': 'np0005533992.novalocal', 'multipath': False, 'enforce_multipath': False, 'initiator': 'iqn.1994-05.com.redhat:5b9c7bcf3f78', 'do_local_attach': False, 'uuid': '3fc68afb-d7e4-475c-9164-ea34f338d965', 'system uuid': 'e2c36df9-c117-4156-b18b-d355565d8e8e', 'nvme_native_multipath': False, 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e', 'found_dsc': '', 'host_ips': ['38.102.83.12', 'fe80::f816:3eff:fe17:2fa8', '192.168.122.1', '172.24.5.1', '2001:db8::2', 'fe80::64fc:4aff:fe7b:de4b', 'fe80::fc16:3eff:fe29:10fc', 'fe80::d821:f8ff:fe9f:14e6']} {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:42:02.288996 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connector [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Factory for iscsi on None {{(pid=122429) factory /usr/local/lib/python3.12/site-packages/os_brick/initiator/connector.py:281}} Nov 24 14:42:02.290250 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] ==> connect_volume: call "{'self': , 'connection_properties': {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043', 'target_lun': 0, 'volume_id': '9b6df04c-499c-4bdc-8ade-28e84ef85043', 'auth_method': 'CHAP', 'auth_username': '44zTWJikcyMEYU4DirmX', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False}}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:42:02.290592 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Acquiring lock "connect_volume" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:42:02.291299 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:42:02.291542 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Connection properties {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043', 'target_lun': 0, 'volume_id': '9b6df04c-499c-4bdc-8ade-28e84ef85043', 'auth_method': 'CHAP', 'auth_username': '44zTWJikcyMEYU4DirmX', 'auth_password': 'Z9ogPtJ2F8N3gTKs', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False, 'enforce_multipath': False} {{(pid=122429) check_multipath /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:133}} Nov 24 14:42:02.292809 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd show status {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:02.298874 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] 'multipathd show status' failed. Not Retrying. {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:42:02.300866 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[bb18d12b-c2c2-4732-a345-c605b6dbcc48]: [Errno 2] No such file or directory: 'multipathd' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:42:02.300866 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:42:02.300866 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:42:02.300866 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:42:02.300866 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:42:02.300866 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:42:02.300866 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:42:02.300866 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:42:02.300866 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root Nov 24 14:42:02.300866 np0005533992.novalocal cinder-backup[122429]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Nov 24 14:42:02.300866 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:42:02.300866 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute Nov 24 14:42:02.300866 np0005533992.novalocal cinder-backup[122429]: return putils.execute(on_execute=on_execute, Nov 24 14:42:02.300866 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:42:02.300866 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute Nov 24 14:42:02.300866 np0005533992.novalocal cinder-backup[122429]: obj = subprocess.Popen(cmd, Nov 24 14:42:02.300866 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:42:02.300866 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__ Nov 24 14:42:02.300866 np0005533992.novalocal cinder-backup[122429]: self._execute_child(args, executable, preexec_fn, close_fds, Nov 24 14:42:02.300866 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child Nov 24 14:42:02.300866 np0005533992.novalocal cinder-backup[122429]: raise child_exception_type(errno_num, err_msg, err_filename) Nov 24 14:42:02.300866 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: 'multipathd' Nov 24 14:42:02.302629 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[bb18d12b-c2c2-4732-a345-c605b6dbcc48]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root\n return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute\n return putils.execute(on_execute=on_execute,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute\n obj = subprocess.Popen(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__\n self._execute_child(args, executable, preexec_fn, close_fds,\n File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child\n raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: \'multipathd\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:02.303602 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Acquiring lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:42:02.304217 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:42:02.304600 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Trying to connect to iSCSI portal 38.102.83.12:3260 Nov 24 14:42:02.305842 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 -p 38.102.83.12:3260 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:02.317130 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 -p 38.102.83.12:3260" returned: 21 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:02.317375 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[6bd1f261-50c4-4399-bbe6-d3ef2c261360]: (4, ('', 'iscsiadm: No records found\n')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:02.317851 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] iscsiadm (): stdout= stderr=iscsiadm: No records found Nov 24 14:42:02.317851 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:42:02.319027 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 -p 38.102.83.12:3260 --interface default --op new {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:02.329664 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 -p 38.102.83.12:3260 --interface default --op new" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:02.329994 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[93b7a479-13da-446e-bc73-4abde3950c49]: (4, ('New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 38.102.83.12,3260,-1 iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043] added\n', 'iscsiadm: Config file line 381 does not have value\niscsiadm: Config file line 381 does not have value\niscsiadm: Config file line 381 does not have value\n')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:02.330422 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--interface', 'default', '--op', 'new'): stdout=New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 38.102.83.12,3260,-1 iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043] added Nov 24 14:42:02.330422 np0005533992.novalocal cinder-backup[122429]: stderr=iscsiadm: Config file line 381 does not have value Nov 24 14:42:02.330422 np0005533992.novalocal cinder-backup[122429]: iscsiadm: Config file line 381 does not have value Nov 24 14:42:02.330422 np0005533992.novalocal cinder-backup[122429]: iscsiadm: Config file line 381 does not have value Nov 24 14:42:02.330422 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:42:02.330724 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Retrying to connect to iSCSI portal 38.102.83.12:3260 {{(pid=122429) _connect_to_iscsi_portal_unsafe /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1133}} Nov 24 14:42:02.331321 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Finished call to 'os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe' after 0.027(s), this was the 1st time calling it. {{(pid=122429) log_it /usr/local/lib/python3.12/site-packages/tenacity/after.py:44}} Nov 24 14:42:02.331712 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Retrying os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe in 1.0 seconds as it raised BrickException: An unknown exception occurred.. {{(pid=122429) log_it /usr/local/lib/python3.12/site-packages/tenacity/before_sleep.py:65}} Nov 24 14:42:03.333409 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Trying to connect to iSCSI portal 38.102.83.12:3260 Nov 24 14:42:03.334487 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 -p 38.102.83.12:3260 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:03.346191 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 -p 38.102.83.12:3260" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:03.346432 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[4290fbba-a07a-451b-ba7e-164ac16839df]: (4, ('# BEGIN RECORD 6.2.1.11\nnode.name = iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043\nnode.tpgt = -1\nnode.startup = automatic\nnode.leading_login = No\niface.iscsi_ifacename = default\niface.net_ifacename = \niface.ipaddress = \niface.prefix_len = 0\niface.hwaddress = \niface.transport_name = tcp\niface.initiatorname = \niface.state = \niface.vlan_id = 0\niface.vlan_priority = 0\niface.vlan_state = \niface.iface_num = 0\niface.mtu = 0\niface.port = 0\niface.bootproto = \niface.subnet_mask = \niface.gateway = \niface.dhcp_alt_client_id_state = \niface.dhcp_alt_client_id = \niface.dhcp_dns = \niface.dhcp_learn_iqn = \niface.dhcp_req_vendor_id_state = \niface.dhcp_vendor_id_state = \niface.dhcp_vendor_id = \niface.dhcp_slp_da = \niface.fragmentation = \niface.gratuitous_arp = \niface.incoming_forwarding = \niface.tos_state = \niface.tos = 0\niface.ttl = 0\niface.delayed_ack = \niface.tcp_nagle = \niface.tcp_wsf_state = \niface.tcp_wsf = 0\niface.tcp_timer_scale = 0\niface.tcp_timestamp = \niface.redirect = \niface.def_task_mgmt_timeout = 0\niface.header_digest = \niface.data_digest = \niface.immediate_data = \niface.initial_r2t = \niface.data_seq_inorder = \niface.data_pdu_inorder = \niface.erl = 0\niface.max_receive_data_len = 0\niface.first_burst_len = 0\niface.max_outstanding_r2t = 0\niface.max_burst_len = 0\niface.chap_auth = \niface.bidi_chap = \niface.strict_login_compliance = \niface.discovery_auth = \niface.discovery_logout = \nnode.discovery_address = \nnode.discovery_port = 0\nnode.discovery_type = fw\nnode.session.initial_cmdsn = 0\nnode.session.initial_login_retry_max = 8\nnode.session.xmit_thread_priority = 0\nnode.session.cmds_max = 128\nnode.session.queue_depth = 32\nnode.session.nr_sessions = 1\nnode.session.auth.authmethod = None\nnode.session.auth.username = \nnode.session.auth.password = \nnode.session.auth.username_in = \nnode.session.auth.password_in = \nnode.session.auth.chap_algs = SHA3-256,SHA256\nnode.session.timeo.replacement_timeout = 120\nnode.session.err_timeo.abort_timeout = 15\nnode.session.err_timeo.lu_reset_timeout = 30\nnode.session.err_timeo.tgt_reset_timeout = 30\nnode.session.err_timeo.host_reset_timeout = 60\nnode.session.iscsi.FastAbort = Yes\nnode.session.iscsi.InitialR2T = No\nnode.session.iscsi.ImmediateData = Yes\nnode.session.iscsi.FirstBurstLength = 262144\nnode.session.iscsi.MaxBurstLength = 16776192\nnode.session.iscsi.DefaultTime2Retain = 0\nnode.session.iscsi.DefaultTime2Wait = 2\nnode.session.iscsi.MaxConnections = 1\nnode.session.iscsi.MaxOutstandingR2T = 1\nnode.session.iscsi.ERL = 0\nnode.session.scan = auto\nnode.session.reopen_max = 0\nnode.session.conn_reopen_log_freq = 1\nnode.conn[0].address = 38.102.83.12\nnode.conn[0].port = 3260\nnode.conn[0].startup = manual\nnode.conn[0].tcp.window_size = 524288\nnode.conn[0].tcp.type_of_service = 0\nnode.conn[0].timeo.logout_timeout = 15\nnode.conn[0].timeo.login_timeout = 15\nnode.conn[0].timeo.auth_timeout = 45\nnode.conn[0].timeo.noop_out_interval = 5\nnode.conn[0].timeo.noop_out_timeout = 5\nnode.conn[0].iscsi.MaxXmitDataSegmentLength = 0\nnode.conn[0].iscsi.MaxRecvDataSegmentLength = 262144\nnode.conn[0].iscsi.HeaderDigest = None\nnode.conn[0].iscsi.DataDigest = None\nnode.conn[0].iscsi.IFMarker = No\nnode.conn[0].iscsi.OFMarker = No\n# END RECORD\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] iscsiadm (): stdout=# BEGIN RECORD 6.2.1.11 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.name = iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.tpgt = -1 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.startup = automatic Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.leading_login = No Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.iscsi_ifacename = default Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.net_ifacename = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.ipaddress = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.prefix_len = 0 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.hwaddress = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.transport_name = tcp Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.initiatorname = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.state = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.vlan_id = 0 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.vlan_priority = 0 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.vlan_state = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.iface_num = 0 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.mtu = 0 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.port = 0 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.bootproto = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.subnet_mask = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.gateway = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_alt_client_id_state = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_alt_client_id = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_dns = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_learn_iqn = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_req_vendor_id_state = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_vendor_id_state = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_vendor_id = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_slp_da = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.fragmentation = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.gratuitous_arp = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.incoming_forwarding = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.tos_state = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.tos = 0 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.ttl = 0 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.delayed_ack = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.tcp_nagle = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.tcp_wsf_state = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.tcp_wsf = 0 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.tcp_timer_scale = 0 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.tcp_timestamp = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.redirect = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.def_task_mgmt_timeout = 0 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.header_digest = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.data_digest = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.immediate_data = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.initial_r2t = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.data_seq_inorder = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.data_pdu_inorder = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.erl = 0 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.max_receive_data_len = 0 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.first_burst_len = 0 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.max_outstanding_r2t = 0 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.max_burst_len = 0 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.chap_auth = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.bidi_chap = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.strict_login_compliance = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.discovery_auth = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: iface.discovery_logout = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.discovery_address = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.discovery_port = 0 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.discovery_type = fw Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.initial_cmdsn = 0 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.initial_login_retry_max = 8 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.xmit_thread_priority = 0 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.cmds_max = 128 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.queue_depth = 32 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.nr_sessions = 1 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.auth.authmethod = None Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.auth.username = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.auth.password = *** Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.auth.username_in = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.auth.password_in = Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.auth.chap_algs = SHA3-256,SHA256 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.timeo.replacement_timeout = 120 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.abort_timeout = 15 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.lu_reset_timeout = 30 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.tgt_reset_timeout = 30 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.host_reset_timeout = 60 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.FastAbort = Yes Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.InitialR2T = No Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.ImmediateData = Yes Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.FirstBurstLength = 262144 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxBurstLength = 16776192 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.DefaultTime2Retain = 0 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.DefaultTime2Wait = 2 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxConnections = 1 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxOutstandingR2T = 1 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.ERL = 0 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.scan = auto Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.reopen_max = 0 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.session.conn_reopen_log_freq = 1 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.conn[0].address = 38.102.83.12 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.conn[0].port = 3260 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.conn[0].startup = manual Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.conn[0].tcp.window_size = 524288 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.conn[0].tcp.type_of_service = 0 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.logout_timeout = 15 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.login_timeout = 15 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.auth_timeout = 45 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.noop_out_interval = 5 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.noop_out_timeout = 5 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.MaxXmitDataSegmentLength = 0 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144 Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.HeaderDigest = None Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.DataDigest = None Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.IFMarker = No Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.OFMarker = No Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: # END RECORD Nov 24 14:42:03.348299 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:42:03.351008 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 -p 38.102.83.12:3260 --op update -n node.session.scan -v manual {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:03.362317 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 -p 38.102.83.12:3260 --op update -n node.session.scan -v manual" returned: 0 in 0.013s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:03.362655 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[3b17977c-45b0-496e-95fb-638f55369a5f]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:03.363141 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.scan', '-v', 'manual'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:42:03.364098 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 -p 38.102.83.12:3260 --op update -n node.session.auth.authmethod -v CHAP {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:03.375109 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 -p 38.102.83.12:3260 --op update -n node.session.auth.authmethod -v CHAP" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:03.375395 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[4c12df0b-74e4-43ed-83e7-6ad6d406127e]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:03.375919 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.authmethod', '-v', 'CHAP'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:42:03.377038 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 -p 38.102.83.12:3260 --op update -n node.session.auth.username -v 44zTWJikcyMEYU4DirmX {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:03.388214 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 -p 38.102.83.12:3260 --op update -n node.session.auth.username -v 44zTWJikcyMEYU4DirmX" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:03.388490 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[98f815c2-37d6-42a1-a936-153166a36c14]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:03.389013 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.username', '-v', '44zTWJikcyMEYU4DirmX'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:42:03.389996 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 -p 38.102.83.12:3260 --op update -n node.session.auth.password -v *** {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:03.397376 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 -p 38.102.83.12:3260 --op update -n node.session.auth.password -v ***" returned: 0 in 0.007s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:03.397667 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[93572bad-0d2a-4e34-9b23-f3dfd057c971]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:03.398253 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.password', '-v', '***'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:42:03.399343 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:03.411305 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:03.411539 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[90a5329a-a207-4235-9097-c281496ef092]: (4, ('tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:03.411981 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('-m', 'session'): stdout=tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash) Nov 24 14:42:03.411981 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:42:03.412240 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] iscsi session list stdout=tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash) Nov 24 14:42:03.412240 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:42:03.413172 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 -p 38.102.83.12:3260 --login {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:03.431719 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 -p 38.102.83.12:3260 --login" returned: 0 in 0.018s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:03.431979 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[5ce073e7-5f69-4cbb-807c-7c92ae6d565f]: (4, ('Login to [iface: default, target: iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043, portal: 38.102.83.12,3260] successful.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:03.432440 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--login',): stdout=Login to [iface: default, target: iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043, portal: 38.102.83.12,3260] successful. Nov 24 14:42:03.432440 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:42:03.434928 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 -p 38.102.83.12:3260 --op update -n node.startup -v automatic {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:03.445366 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 -p 38.102.83.12:3260 --op update -n node.startup -v automatic" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:03.445659 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[576168f2-2d09-4208-8518-9738aa0652dc]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:03.446087 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:42:03.446934 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:03.461011 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.014s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:03.461246 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[1fc87113-bf7d-443d-bf36-fa627d69bae7]: (4, ('tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash)\ntcp: [30] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:03.461674 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('-m', 'session'): stdout=tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash) Nov 24 14:42:03.461674 np0005533992.novalocal cinder-backup[122429]: tcp: [30] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 (non-flash) Nov 24 14:42:03.461674 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:42:03.461933 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] iscsi session list stdout=tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash) Nov 24 14:42:03.461933 np0005533992.novalocal cinder-backup[122429]: tcp: [30] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 (non-flash) Nov 24 14:42:03.461933 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:42:03.462411 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: held 1.158s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:42:03.462746 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Connected to 38.102.83.12:3260 {{(pid=122429) _connect_vol /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:659}} Nov 24 14:42:03.463526 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] HCTL ('3', '-', '-', 0) found on session 30 with lun 0 {{(pid=122429) get_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:788}} Nov 24 14:42:03.463865 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Scanning host 3 c: -, t: -, l: 0) {{(pid=122429) scan_iscsi /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:814}} Nov 24 14:42:03.464853 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/class/scsi_host/host3/scan {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:03.487613 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/class/scsi_host/host3/scan" returned: 0 in 0.021s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:03.487613 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[f7b6e2ed-fe74-42a3-87b3-1a11cc7d7f7b]: (4, ('- - 0', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:03.487613 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Searching for a device in session 30 and hctl ['3', '*', '*', 0] yield: None {{(pid=122429) device_name_by_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:808}} Nov 24 14:42:04.490321 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Searching for a device in session 30 and hctl ['3', '*', '*', 0] yield: sdb {{(pid=122429) device_name_by_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:808}} Nov 24 14:42:04.491197 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Connected to sdb using {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043', 'target_lun': 0, 'volume_id': '9b6df04c-499c-4bdc-8ade-28e84ef85043', 'auth_method': 'CHAP', 'auth_username': '44zTWJikcyMEYU4DirmX', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False} {{(pid=122429) _connect_vol /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:687}} Nov 24 14:42:04.492049 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" :: held 2.201s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:42:04.492392 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] <== connect_volume: return (2201ms) {'type': 'block', 'scsi_wwn': '36001405aa6041c78d304f83832c7a2f4', 'path': '/dev/sdb'} {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:42:04.493018 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sdb {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:04.511719 np0005533992.novalocal sudo[163876]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sdb Nov 24 14:42:04.834039 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sdb" returned: 0 in 0.341s {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:04.843285 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_db.sqlalchemy.engines [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=122429) _check_effective_sql_mode /usr/local/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:397}} Nov 24 14:42:04.849284 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] _read_sha256file started, container name: tempest-volumesbackupstest-backup-container-601227527, sha256 filename: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7_sha256file. {{(pid=122429) _read_sha256file /opt/stack/cinder/cinder/backup/chunkeddriver.py:336}} Nov 24 14:42:04.897383 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] _read_sha256file finished. {{(pid=122429) _read_sha256file /opt/stack/cinder/cinder/backup/chunkeddriver.py:343}} Nov 24 14:42:04.908987 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] _create_container started, container: tempest-volumesbackupstest-backup-container-1830262932,backup: c21c7367-55a2-43d2-8456-15fae70970ac. {{(pid=122429) _create_container /opt/stack/cinder/cinder/backup/chunkeddriver.py:252}} Nov 24 14:42:04.983480 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] generate_object_name_prefix: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194204/az_nova_backup_c21c7367-55a2-43d2-8456-15fae70970ac {{(pid=122429) _generate_object_name_prefix /opt/stack/cinder/cinder/backup/drivers/swift.py:433}} Nov 24 14:42:04.989507 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] starting backup of volume: 662c5cd4-cc35-4e90-a911-679b13dfa7a5, volume size: 1073741824, object names prefix volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194204/az_nova_backup_c21c7367-55a2-43d2-8456-15fae70970ac, availability zone: nova {{(pid=122429) _prepare_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:362}} Nov 24 14:42:06.062388 np0005533992.novalocal cinder-backup[122429]: DEBUG dbcounter [-] [122429] Writing DB stats cinder:SELECT=91,cinder:UPDATE=24,cinder:INSERT=2 {{(pid=122429) stat_writer /usr/local/lib/python3.12/site-packages/dbcounter.py:115}} Nov 24 14:42:09.987996 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Getting metadata type 'volume-base-metadata' {{(pid=122429) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Nov 24 14:42:09.996467 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=122429) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Nov 24 14:42:09.996791 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Getting metadata type 'volume-metadata' {{(pid=122429) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Nov 24 14:42:10.002463 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] No metadata type 'volume-metadata' available {{(pid=122429) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Nov 24 14:42:10.002824 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Getting metadata type 'volume-glance-metadata' {{(pid=122429) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Nov 24 14:42:10.007354 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] No metadata type 'volume-glance-metadata' available {{(pid=122429) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Nov 24 14:42:10.007844 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] _write_sha256file started, container name: tempest-volumesbackupstest-backup-container-1830262932, sha256file filename: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194204/az_nova_backup_c21c7367-55a2-43d2-8456-15fae70970ac_sha256file. {{(pid=122429) _write_sha256file /opt/stack/cinder/cinder/backup/chunkeddriver.py:302}} Nov 24 14:42:10.110333 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] _write_sha256file finished. {{(pid=122429) _write_sha256file /opt/stack/cinder/cinder/backup/chunkeddriver.py:318}} Nov 24 14:42:10.111056 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] _write_metadata started, container name: tempest-volumesbackupstest-backup-container-1830262932, metadata filename: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194204/az_nova_backup_c21c7367-55a2-43d2-8456-15fae70970ac_metadata. {{(pid=122429) _write_metadata /opt/stack/cinder/cinder/backup/chunkeddriver.py:279}} Nov 24 14:42:10.145987 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] _write_metadata finished. Metadata: b'{\n "backup_description": null,\n "backup_id": "c21c7367-55a2-43d2-8456-15fae70970ac",\n "backup_name": "tempest-VolumesBackupsTest-Backup-1101742960",\n "created_at": "2025-11-24 19:41:57+00:00",\n "objects": [],\n "parent_id": "519039aa-9b69-46f1-b149-a260745c08c7",\n "version": "1.0.0",\n "volume_id": "662c5cd4-cc35-4e90-a911-679b13dfa7a5",\n "volume_meta": "{\\"version\\": 2, \\"volume-base-metadata\\": {\\"id\\": \\"662c5cd4-cc35-4e90-a911-679b13dfa7a5\\", \\"_name_id\\": null, \\"use_quota\\": true, \\"ec2_id\\": null, \\"user_id\\": \\"0e48df3b35f84ee88136fd47ee531c1f\\", \\"project_id\\": \\"c642367dafaa45dd89634e7ec9822ef2\\", \\"snapshot_id\\": null, \\"cluster_name\\": null, \\"host\\": \\"np0005533992.novalocal@lvmdriver-1#lvmdriver-1\\", \\"size\\": 1, \\"availability_zone\\": \\"nova\\", \\"status\\": \\"backing-up\\", \\"attach_status\\": \\"attached\\", \\"migration_status\\": null, \\"scheduled_at\\": \\"2025-11-24T19:40:18.000000\\", \\"launched_at\\": \\"2025-11-24T19:40:19.000000\\", \\"terminated_at\\": null, \\"display_name\\": \\"tempest-VolumesBackupsTest-Volume-1459000726\\", \\"display_description\\": null, \\"provider_location\\": \\"38.102.83.12:3260,iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 0\\", \\"provider_auth\\": \\"CHAP rwfecJCEJ9JcRMgV8PNK ZDyQkauPVCK7HqgF\\", \\"provider_geometry\\": null, \\"provider_id\\": null, \\"volume_type_id\\": \\"1f8776b6-c527-4643-876a-506285adfc66\\", \\"source_volid\\": null, \\"encryption_key_id\\": null, \\"consistencygroup_id\\": null, \\"group_id\\": null, \\"bootable\\": false, \\"multiattach\\": false, \\"replication_status\\": null, \\"replication_extended_status\\": null, \\"replication_driver_data\\": null, \\"previous_status\\": \\"in-use\\", \\"service_uuid\\": \\"6d0941fd-19ec-48ae-82b1-f25526175227\\", \\"shared_targets\\": false, \\"deleted_at\\": null, \\"deleted\\": false, \\"created_at\\": \\"2025-11-24T19:40:18.000000\\", \\"updated_at\\": \\"2025-11-24T19:41:57.000000\\"}}"\n}'. {{(pid=122429) _write_metadata /opt/stack/cinder/cinder/backup/chunkeddriver.py:298}} Nov 24 14:42:10.155041 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] backup c21c7367-55a2-43d2-8456-15fae70970ac finished. {{(pid=122429) _finalize_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:465}} Nov 24 14:42:10.248753 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sdb {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:10.273779 np0005533992.novalocal sudo[163883]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sdb Nov 24 14:42:10.546418 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sdb" returned: 0 in 0.297s {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:10.548780 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] ==> disconnect_volume: call "{'args': (, {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043', 'target_lun': 0, 'volume_id': '9b6df04c-499c-4bdc-8ade-28e84ef85043', 'auth_method': 'CHAP', 'auth_username': '44zTWJikcyMEYU4DirmX', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False, 'enforce_multipath': False}, {'type': 'block', 'scsi_wwn': '36001405aa6041c78d304f83832c7a2f4', 'path': '/dev/sdb'}), 'kwargs': {'force': True}}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:42:10.548780 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Acquiring lock "connect_volume" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:42:10.549207 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:42:10.550802 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m discoverydb -o show -P 1 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:10.562534 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m discoverydb -o show -P 1" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:10.562867 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[b5012f7a-a8f4-4747-b87b-c81373f2b01b]: (4, ('SENDTARGETS:\nNo targets found.\niSNS:\nNo targets found.\nSTATIC:\nTarget: iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5\n\tPortal: 38.102.83.12:3260,-1\n\t\tIface Name: default\nTarget: iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043\n\tPortal: 38.102.83.12:3260,-1\n\t\tIface Name: default\nFIRMWARE:\nNo targets found.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:10.563311 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] iscsiadm ['-m', 'discoverydb', '-o', 'show', '-P', 1]: stdout=SENDTARGETS: Nov 24 14:42:10.563311 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:42:10.563311 np0005533992.novalocal cinder-backup[122429]: iSNS: Nov 24 14:42:10.563311 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:42:10.563311 np0005533992.novalocal cinder-backup[122429]: STATIC: Nov 24 14:42:10.563311 np0005533992.novalocal cinder-backup[122429]: Target: iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 Nov 24 14:42:10.563311 np0005533992.novalocal cinder-backup[122429]: Portal: 38.102.83.12:3260,-1 Nov 24 14:42:10.563311 np0005533992.novalocal cinder-backup[122429]: Iface Name: default Nov 24 14:42:10.563311 np0005533992.novalocal cinder-backup[122429]: Target: iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 Nov 24 14:42:10.563311 np0005533992.novalocal cinder-backup[122429]: Portal: 38.102.83.12:3260,-1 Nov 24 14:42:10.563311 np0005533992.novalocal cinder-backup[122429]: Iface Name: default Nov 24 14:42:10.563311 np0005533992.novalocal cinder-backup[122429]: FIRMWARE: Nov 24 14:42:10.563311 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:42:10.563311 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:42:10.563693 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Regex to get portals from discoverydb: ^SENDTARGETS: Nov 24 14:42:10.563693 np0005533992.novalocal cinder-backup[122429]: .*?^DiscoveryAddress: 38.102.83.12,3260.*? Nov 24 14:42:10.563693 np0005533992.novalocal cinder-backup[122429]: (.*?)^(?:DiscoveryAddress|iSNS):.* {{(pid=122429) _get_discoverydb_portals /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:392}} Nov 24 14:42:10.564144 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Getting connected devices for (ips,iqns,luns)=[('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043', 0)] {{(pid=122429) _get_connection_devices /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:855}} Nov 24 14:42:10.565190 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:10.574199 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node" returned: 0 in 0.009s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:10.574459 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[8dfd909d-5c3e-4a79-b18b-fe5ed8c8e281]: (4, ('38.102.83.12:3260,4294967295 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5\n38.102.83.12:3260,4294967295 iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:10.575504 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:10.582370 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.007s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:10.582660 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[5409d147-f3cc-43ca-a270-271d55336a42]: (4, ('tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash)\ntcp: [30] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:10.583153 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('-m', 'session'): stdout=tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash) Nov 24 14:42:10.583153 np0005533992.novalocal cinder-backup[122429]: tcp: [30] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 (non-flash) Nov 24 14:42:10.583153 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:42:10.583446 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] iscsi session list stdout=tcp: [27] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-662c5cd4-cc35-4e90-a911-679b13dfa7a5 (non-flash) Nov 24 14:42:10.583446 np0005533992.novalocal cinder-backup[122429]: tcp: [30] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 (non-flash) Nov 24 14:42:10.583446 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:42:10.584551 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Resulting device map defaultdict(. at 0x7f5b7298e700>, {('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043'): ({'sdb'}, set())}) {{(pid=122429) _get_connection_devices /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:888}} Nov 24 14:42:10.585025 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Removing single pathed devices sdb {{(pid=122429) remove_connection /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:381}} Nov 24 14:42:10.586055 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd show status {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:10.592956 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] 'multipathd show status' failed. Not Retrying. {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:42:10.595019 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[20a135c3-2a9c-4918-9984-751c9594e2c4]: [Errno 2] No such file or directory: 'multipathd' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:42:10.595019 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:42:10.595019 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:42:10.595019 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:42:10.595019 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:42:10.595019 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:42:10.595019 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:42:10.595019 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:42:10.595019 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root Nov 24 14:42:10.595019 np0005533992.novalocal cinder-backup[122429]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Nov 24 14:42:10.595019 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:42:10.595019 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute Nov 24 14:42:10.595019 np0005533992.novalocal cinder-backup[122429]: return putils.execute(on_execute=on_execute, Nov 24 14:42:10.595019 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:42:10.595019 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute Nov 24 14:42:10.595019 np0005533992.novalocal cinder-backup[122429]: obj = subprocess.Popen(cmd, Nov 24 14:42:10.595019 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:42:10.595019 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__ Nov 24 14:42:10.595019 np0005533992.novalocal cinder-backup[122429]: self._execute_child(args, executable, preexec_fn, close_fds, Nov 24 14:42:10.595019 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child Nov 24 14:42:10.595019 np0005533992.novalocal cinder-backup[122429]: raise child_exception_type(errno_num, err_msg, err_filename) Nov 24 14:42:10.595019 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: 'multipathd' Nov 24 14:42:10.596685 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[20a135c3-2a9c-4918-9984-751c9594e2c4]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root\n return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute\n return putils.execute(on_execute=on_execute,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute\n obj = subprocess.Popen(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__\n self._execute_child(args, executable, preexec_fn, close_fds,\n File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child\n raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: \'multipathd\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:10.597301 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Flushing IO for device /dev/sdb {{(pid=122429) flush_device_io /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:441}} Nov 24 14:42:10.598214 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blockdev --flushbufs /dev/sdb {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:10.603588 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "blockdev --flushbufs /dev/sdb" returned: 0 in 0.005s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:10.603802 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[6178e1c4-d354-47ff-aa54-c19a47f3146c]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:10.604186 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Remove SCSI device /dev/sdb with /sys/block/sdb/device/delete {{(pid=122429) remove_scsi_device /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:151}} Nov 24 14:42:10.605196 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/block/sdb/device/delete {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:10.653040 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/block/sdb/device/delete" returned: 0 in 0.048s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:10.653171 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[de483717-63f6-4552-a4a0-c4bd00e1ca6f]: (4, ('1', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:10.653454 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Checking to see if SCSI volumes sdb have been removed. {{(pid=122429) wait_for_volumes_removal /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:159}} Nov 24 14:42:10.653602 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] SCSI volumes sdb have been removed. {{(pid=122429) wait_for_volumes_removal /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:169}} Nov 24 14:42:10.653943 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Disconnecting from: [('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043')] {{(pid=122429) _disconnect_connection /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1201}} Nov 24 14:42:10.654790 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 -p 38.102.83.12:3260 --op update -n node.startup -v manual {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:10.668778 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 -p 38.102.83.12:3260 --op update -n node.startup -v manual" returned: 0 in 0.014s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:10.668997 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[1d0ac570-17e6-48dd-b8e5-5d18beafa2b4]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:10.669222 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:42:10.670016 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 -p 38.102.83.12:3260 --logout {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:10.704366 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 -p 38.102.83.12:3260 --logout" returned: 0 in 0.034s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:10.704737 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[fb7a7b1f-9bbb-47af-91fc-666874dc80c3]: (4, ('Logging out of session [sid: 30, target: iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043, portal: 38.102.83.12,3260]\nLogout of [sid: 30, target: iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043, portal: 38.102.83.12,3260] successful.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:10.705047 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--logout',): stdout=Logging out of session [sid: 30, target: iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043, portal: 38.102.83.12,3260] Nov 24 14:42:10.705047 np0005533992.novalocal cinder-backup[122429]: Logout of [sid: 30, target: iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043, portal: 38.102.83.12,3260] successful. Nov 24 14:42:10.705047 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:42:10.705771 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 -p 38.102.83.12:3260 --op delete {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:10.716489 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9b6df04c-499c-4bdc-8ade-28e84ef85043 -p 38.102.83.12:3260 --op delete" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:10.716489 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[7d8d5bd3-1426-4de1-a78c-50fbbf4fc441]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:10.716489 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] iscsiadm ('--op', 'delete'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:42:10.716686 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Lock "connect_volume" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: held 0.168s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:42:10.716763 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] <== disconnect_volume: return (168ms) None {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:42:14.266585 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-bb52f41b-7407-4379-b0ae-ed5a1e40d5ca tempest-VolumesBackupsTest-1816644175 None] Create backup finished. backup: c21c7367-55a2-43d2-8456-15fae70970ac. Nov 24 14:42:15.056177 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-f949ed08-c910-4c7f-88dc-e4cf283f03aa tempest-VolumesBackupsTest-1816644175 None] Delete backup started, backup: c21c7367-55a2-43d2-8456-15fae70970ac. Nov 24 14:42:15.058247 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-f949ed08-c910-4c7f-88dc-e4cf283f03aa tempest-VolumesBackupsTest-1816644175 None] Using swift URL https://38.102.83.12:8080/v1/AUTH_c642367dafaa45dd89634e7ec9822ef2 {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:286}} Nov 24 14:42:15.058654 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-f949ed08-c910-4c7f-88dc-e4cf283f03aa tempest-VolumesBackupsTest-1816644175 None] Connect to https://38.102.83.12:8080/v1/AUTH_ in "per_user" mode {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:287}} Nov 24 14:42:15.060422 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f949ed08-c910-4c7f-88dc-e4cf283f03aa tempest-VolumesBackupsTest-1816644175 None] delete started, backup: c21c7367-55a2-43d2-8456-15fae70970ac, container: tempest-volumesbackupstest-backup-container-1830262932, prefix: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194204/az_nova_backup_c21c7367-55a2-43d2-8456-15fae70970ac. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:844}} Nov 24 14:42:15.128933 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f949ed08-c910-4c7f-88dc-e4cf283f03aa tempest-VolumesBackupsTest-1816644175 None] generated object list: ['volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194204/az_nova_backup_c21c7367-55a2-43d2-8456-15fae70970ac_metadata', 'volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194204/az_nova_backup_c21c7367-55a2-43d2-8456-15fae70970ac_sha256file']. {{(pid=122429) _generate_object_names /opt/stack/cinder/cinder/backup/chunkeddriver.py:263}} Nov 24 14:42:15.162320 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f949ed08-c910-4c7f-88dc-e4cf283f03aa tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194204/az_nova_backup_c21c7367-55a2-43d2-8456-15fae70970ac_metadata in container: tempest-volumesbackupstest-backup-container-1830262932. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:15.191861 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f949ed08-c910-4c7f-88dc-e4cf283f03aa tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194204/az_nova_backup_c21c7367-55a2-43d2-8456-15fae70970ac_sha256file in container: tempest-volumesbackupstest-backup-container-1830262932. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:15.192271 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-f949ed08-c910-4c7f-88dc-e4cf283f03aa tempest-VolumesBackupsTest-1816644175 None] delete c21c7367-55a2-43d2-8456-15fae70970ac finished. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:870}} Nov 24 14:42:15.196410 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_db.sqlalchemy.engines [None req-f949ed08-c910-4c7f-88dc-e4cf283f03aa tempest-VolumesBackupsTest-1816644175 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=122429) _check_effective_sql_mode /usr/local/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:397}} Nov 24 14:42:15.213129 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.quota [None req-f949ed08-c910-4c7f-88dc-e4cf283f03aa tempest-VolumesBackupsTest-1816644175 None] Created reservations ['aea3934f-eeb2-4135-8113-8fa55c27f595', '2fd05301-16a5-4825-9f54-f751d0d371c8'] {{(pid=122429) reserve /opt/stack/cinder/cinder/quota.py:752}} Nov 24 14:42:15.240843 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-f949ed08-c910-4c7f-88dc-e4cf283f03aa tempest-VolumesBackupsTest-1816644175 None] Delete backup finished, backup c21c7367-55a2-43d2-8456-15fae70970ac deleted. Nov 24 14:42:16.250437 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] Delete backup started, backup: 519039aa-9b69-46f1-b149-a260745c08c7. Nov 24 14:42:16.251754 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] Using swift URL https://38.102.83.12:8080/v1/AUTH_c642367dafaa45dd89634e7ec9822ef2 {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:286}} Nov 24 14:42:16.252118 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] Connect to https://38.102.83.12:8080/v1/AUTH_ in "per_user" mode {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:287}} Nov 24 14:42:16.252430 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] delete started, backup: 519039aa-9b69-46f1-b149-a260745c08c7, container: tempest-volumesbackupstest-backup-container-601227527, prefix: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:844}} Nov 24 14:42:16.311399 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] generated object list: ['volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00001', 'volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00002', 'volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00003', 'volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00004', 'volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00005', 'volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00006', 'volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00007', 'volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00008', 'volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00009', 'volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00010', 'volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00011', 'volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00012', 'volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00013', 'volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00014', 'volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00015', 'volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00016', 'volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00017', 'volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00018', 'volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00019', 'volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00020', 'volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00021', 'volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7_metadata', 'volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7_sha256file']. {{(pid=122429) _generate_object_names /opt/stack/cinder/cinder/backup/chunkeddriver.py:263}} Nov 24 14:42:16.337183 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00001 in container: tempest-volumesbackupstest-backup-container-601227527. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:16.362915 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00002 in container: tempest-volumesbackupstest-backup-container-601227527. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:16.389924 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00003 in container: tempest-volumesbackupstest-backup-container-601227527. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:16.422288 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00004 in container: tempest-volumesbackupstest-backup-container-601227527. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:16.449838 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00005 in container: tempest-volumesbackupstest-backup-container-601227527. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:16.477429 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00006 in container: tempest-volumesbackupstest-backup-container-601227527. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:16.506147 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00007 in container: tempest-volumesbackupstest-backup-container-601227527. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:16.533850 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00008 in container: tempest-volumesbackupstest-backup-container-601227527. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:16.562121 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00009 in container: tempest-volumesbackupstest-backup-container-601227527. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:16.588451 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00010 in container: tempest-volumesbackupstest-backup-container-601227527. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:16.612620 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00011 in container: tempest-volumesbackupstest-backup-container-601227527. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:16.638686 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00012 in container: tempest-volumesbackupstest-backup-container-601227527. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:16.667635 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00013 in container: tempest-volumesbackupstest-backup-container-601227527. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:16.699345 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00014 in container: tempest-volumesbackupstest-backup-container-601227527. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:16.726922 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00015 in container: tempest-volumesbackupstest-backup-container-601227527. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:16.752841 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00016 in container: tempest-volumesbackupstest-backup-container-601227527. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:16.777032 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00017 in container: tempest-volumesbackupstest-backup-container-601227527. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:16.798471 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00018 in container: tempest-volumesbackupstest-backup-container-601227527. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:16.826095 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00019 in container: tempest-volumesbackupstest-backup-container-601227527. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:16.852791 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00020 in container: tempest-volumesbackupstest-backup-container-601227527. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:16.880608 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7-00021 in container: tempest-volumesbackupstest-backup-container-601227527. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:16.907551 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7_metadata in container: tempest-volumesbackupstest-backup-container-601227527. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:16.932239 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] deleted object: volume_662c5cd4-cc35-4e90-a911-679b13dfa7a5/20251124194121/az_nova_backup_519039aa-9b69-46f1-b149-a260745c08c7_sha256file in container: tempest-volumesbackupstest-backup-container-601227527. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:42:16.932625 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] delete 519039aa-9b69-46f1-b149-a260745c08c7 finished. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:870}} Nov 24 14:42:16.936538 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_db.sqlalchemy.engines [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=122429) _check_effective_sql_mode /usr/local/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:397}} Nov 24 14:42:16.953444 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.quota [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] Created reservations ['08ebbbdb-3235-4806-8914-23b2255aa437', 'b6b95621-522b-4cac-b04d-2250e222fa8f'] {{(pid=122429) reserve /opt/stack/cinder/cinder/quota.py:752}} Nov 24 14:42:16.970950 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-0462b0c6-a538-49d1-bc9b-4c31b46e231c tempest-VolumesBackupsTest-1816644175 None] Delete backup finished, backup 519039aa-9b69-46f1-b149-a260745c08c7 deleted. Nov 24 14:42:26.970283 np0005533992.novalocal cinder-backup[122429]: DEBUG dbcounter [-] [122429] Writing DB stats cinder:SELECT=39,cinder:UPDATE=18,cinder:INSERT=4 {{(pid=122429) stat_writer /usr/local/lib/python3.12/site-packages/dbcounter.py:115}} Nov 24 14:42:46.434953 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:42:46.435522 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:42:56.074130 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Create backup started, backup: 95a3112e-e57e-4a79-bbe8-b4efbc83c5b5 volume: df3774f1-c5c9-492e-802c-e5ce09564b66 snapshot: ac6cdbab-9ae7-4075-a222-a9ae0db7574f. Nov 24 14:42:56.079143 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Call Volume Manager to get_backup_device for Backup(availability_zone='nova',container='tempest-volumessnapshottestjson-backup-container-809404681',created_at=2025-11-24T19:42:55Z,data_timestamp=2025-11-24T19:42:54Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Backup-1226447601',encryption_key_id=None,fail_reason=None,host='np0005533992.novalocal',id=95a3112e-e57e-4a79-bbe8-b4efbc83c5b5,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='f851d93dc40a47668f073fe029189315',restore_volume_id=None,service='cinder.backup.drivers.swift.SwiftBackupDriver',service_metadata=None,size=1,snapshot_id='ac6cdbab-9ae7-4075-a222-a9ae0db7574f',status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='9f1963a65a4341d5b36078d1273654b9',volume_id=df3774f1-c5c9-492e-802c-e5ce09564b66) Nov 24 14:42:57.860607 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Using swift URL https://38.102.83.12:8080/v1/AUTH_f851d93dc40a47668f073fe029189315 {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:286}} Nov 24 14:42:57.860607 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Connect to https://38.102.83.12:8080/v1/AUTH_ in "per_user" mode {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:287}} Nov 24 14:42:57.861665 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '38.102.83.12', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:42:57.863267 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:57.872355 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.009s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:57.872688 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[d7456974-eec8-43df-8345-f5a0d2b9e1c3]: (4, ('InitiatorName=iqn.1994-05.com.redhat:5b9c7bcf3f78\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:57.875334 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[4f0ec67d-d5e2-4f77-ba18-97a95b972e96]: [Errno 2] No such file or directory: '/dev/scini' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:42:57.875334 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:42:57.875334 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:42:57.875334 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:42:57.875334 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:42:57.875334 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:42:57.875334 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:42:57.875334 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:42:57.875334 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 57, in get_guid Nov 24 14:42:57.875334 np0005533992.novalocal cinder-backup[122429]: with open_scini_device() as fd: Nov 24 14:42:57.875334 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^ Nov 24 14:42:57.875334 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/contextlib.py", line 137, in __enter__ Nov 24 14:42:57.875334 np0005533992.novalocal cinder-backup[122429]: return next(self.gen) Nov 24 14:42:57.875334 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^ Nov 24 14:42:57.875334 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 40, in open_scini_device Nov 24 14:42:57.875334 np0005533992.novalocal cinder-backup[122429]: fd = os.open(SCINI_DEVICE_PATH, os.O_RDWR) Nov 24 14:42:57.875334 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:42:57.875334 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: '/dev/scini' Nov 24 14:42:57.876217 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[4f0ec67d-d5e2-4f77-ba18-97a95b972e96]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 57, in get_guid\n with open_scini_device() as fd:\n ^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/contextlib.py", line 137, in __enter__\n return next(self.gen)\n ^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/scaleio.py", line 40, in open_scini_device\n fd = os.open(SCINI_DEVICE_PATH, os.O_RDWR)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory: \'/dev/scini\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:57.877052 np0005533992.novalocal cinder-backup[122429]: ERROR os_brick.initiator.connectors.scaleio [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Error querying sdc guid: [Errno 2] No such file or directory: FileNotFoundError: [Errno 2] No such file or directory Nov 24 14:42:57.877639 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.scaleio [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Unable to find SDC guid: Error querying sdc guid: [Errno 2] No such file or directory Nov 24 14:42:57.878823 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt -v / -n -o SOURCE {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:57.887633 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt -v / -n -o SOURCE" returned: 0 in 0.009s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:57.887920 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[02125ddf-5301-4c0b-a4bb-7cdf397ad35f]: (4, ('/dev/vda2\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:57.888975 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/vda2 -s UUID -o value {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:57.899696 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/vda2 -s UUID -o value" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:42:57.899932 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[9ba4ee73-4f5a-42e5-a385-de750763585a]: (4, ('3fc68afb-d7e4-475c-9164-ea34f338d965\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:57.901160 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[528c1ce3-9a0b-4ee3-aa28-93cb31d245ac]: (4, 'e2c36df9-c117-4156-b18b-d355565d8e8e') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:42:57.901656 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Running cmd (subprocess): nvme version {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:42:57.916095 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] 'nvme version' failed. Not Retrying. {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:42:57.917101 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.nvmeof [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] nvme not present on system {{(pid=122429) nvme_present /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/nvmeof.py:784}} Nov 24 14:42:57.919178 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=122429) find_dsc /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:132}} Nov 24 14:42:57.920163 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.lightos [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Current host hostNQN nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e and IP(s) are ['38.102.83.12', 'fe80::f816:3eff:fe17:2fa8', '192.168.122.1', '172.24.5.1', '2001:db8::2', 'fe80::64fc:4aff:fe7b:de4b'] Nov 24 14:42:57.920421 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=122429) get_connector_properties /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:109}} Nov 24 14:42:57.920705 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.lightos [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e dsc: {{(pid=122429) get_connector_properties /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/lightos.py:112}} Nov 24 14:42:57.921212 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] <== get_connector_properties: return (59ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '38.102.83.12', 'host': 'np0005533992.novalocal', 'multipath': False, 'enforce_multipath': False, 'initiator': 'iqn.1994-05.com.redhat:5b9c7bcf3f78', 'do_local_attach': False, 'uuid': '3fc68afb-d7e4-475c-9164-ea34f338d965', 'system uuid': 'e2c36df9-c117-4156-b18b-d355565d8e8e', 'nvme_native_multipath': False, 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:b75896e9-b5d4-4941-a6e4-36982d60c99e', 'found_dsc': '', 'host_ips': ['38.102.83.12', 'fe80::f816:3eff:fe17:2fa8', '192.168.122.1', '172.24.5.1', '2001:db8::2', 'fe80::64fc:4aff:fe7b:de4b']} {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:43:01.429173 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connector [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Factory for iscsi on None {{(pid=122429) factory /usr/local/lib/python3.12/site-packages/os_brick/initiator/connector.py:281}} Nov 24 14:43:01.430688 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] ==> connect_volume: call "{'self': , 'connection_properties': {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2', 'target_lun': 0, 'volume_id': 'c736c436-e906-48ce-a9bd-077d081f27e2', 'auth_method': 'CHAP', 'auth_username': 'UF26PhGjSTbHXdKVtfU8', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False}}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:43:01.431101 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Acquiring lock "connect_volume" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:43:01.431876 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:43:01.432234 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Connection properties {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2', 'target_lun': 0, 'volume_id': 'c736c436-e906-48ce-a9bd-077d081f27e2', 'auth_method': 'CHAP', 'auth_username': 'UF26PhGjSTbHXdKVtfU8', 'auth_password': 'qwpToGf26t7LhChM', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False, 'enforce_multipath': False} {{(pid=122429) check_multipath /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:133}} Nov 24 14:43:01.433451 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd show status {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:43:01.439367 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] 'multipathd show status' failed. Not Retrying. {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:43:01.441411 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[d4f79adf-db0a-46c4-b034-725906ee0613]: [Errno 2] No such file or directory: 'multipathd' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:43:01.441411 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:43:01.441411 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:43:01.441411 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:43:01.441411 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:43:01.441411 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:43:01.441411 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:43:01.441411 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:43:01.441411 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root Nov 24 14:43:01.441411 np0005533992.novalocal cinder-backup[122429]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Nov 24 14:43:01.441411 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:43:01.441411 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute Nov 24 14:43:01.441411 np0005533992.novalocal cinder-backup[122429]: return putils.execute(on_execute=on_execute, Nov 24 14:43:01.441411 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:43:01.441411 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute Nov 24 14:43:01.441411 np0005533992.novalocal cinder-backup[122429]: obj = subprocess.Popen(cmd, Nov 24 14:43:01.441411 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:43:01.441411 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__ Nov 24 14:43:01.441411 np0005533992.novalocal cinder-backup[122429]: self._execute_child(args, executable, preexec_fn, close_fds, Nov 24 14:43:01.441411 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child Nov 24 14:43:01.441411 np0005533992.novalocal cinder-backup[122429]: raise child_exception_type(errno_num, err_msg, err_filename) Nov 24 14:43:01.441411 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: 'multipathd' Nov 24 14:43:01.443247 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[d4f79adf-db0a-46c4-b034-725906ee0613]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root\n return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute\n return putils.execute(on_execute=on_execute,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute\n obj = subprocess.Popen(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__\n self._execute_child(args, executable, preexec_fn, close_fds,\n File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child\n raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: \'multipathd\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:43:01.444169 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Acquiring lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:43:01.444960 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:43:01.445440 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Trying to connect to iSCSI portal 38.102.83.12:3260 Nov 24 14:43:01.446890 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 -p 38.102.83.12:3260 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:43:01.458082 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 -p 38.102.83.12:3260" returned: 21 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:43:01.458378 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[c9d55c2f-6449-4f38-abfe-bd6d83cdd2fe]: (4, ('', 'iscsiadm: No records found\n')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:43:01.459083 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] iscsiadm (): stdout= stderr=iscsiadm: No records found Nov 24 14:43:01.459083 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:43:01.460333 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 -p 38.102.83.12:3260 --interface default --op new {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:43:01.471090 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 -p 38.102.83.12:3260 --interface default --op new" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:43:01.471446 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[267294f0-24a0-4c23-aa27-6725b54ea1d1]: (4, ('New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 38.102.83.12,3260,-1 iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2] added\n', 'iscsiadm: Config file line 381 does not have value\niscsiadm: Config file line 381 does not have value\niscsiadm: Config file line 381 does not have value\n')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:43:01.471965 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] iscsiadm ('--interface', 'default', '--op', 'new'): stdout=New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 38.102.83.12,3260,-1 iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2] added Nov 24 14:43:01.471965 np0005533992.novalocal cinder-backup[122429]: stderr=iscsiadm: Config file line 381 does not have value Nov 24 14:43:01.471965 np0005533992.novalocal cinder-backup[122429]: iscsiadm: Config file line 381 does not have value Nov 24 14:43:01.471965 np0005533992.novalocal cinder-backup[122429]: iscsiadm: Config file line 381 does not have value Nov 24 14:43:01.471965 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:43:01.472279 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Retrying to connect to iSCSI portal 38.102.83.12:3260 {{(pid=122429) _connect_to_iscsi_portal_unsafe /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1133}} Nov 24 14:43:01.473023 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Finished call to 'os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe' after 0.027(s), this was the 1st time calling it. {{(pid=122429) log_it /usr/local/lib/python3.12/site-packages/tenacity/after.py:44}} Nov 24 14:43:01.473364 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.utils [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Retrying os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe in 1.0 seconds as it raised BrickException: An unknown exception occurred.. {{(pid=122429) log_it /usr/local/lib/python3.12/site-packages/tenacity/before_sleep.py:65}} Nov 24 14:43:02.474479 np0005533992.novalocal cinder-backup[122429]: INFO os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Trying to connect to iSCSI portal 38.102.83.12:3260 Nov 24 14:43:02.475849 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 -p 38.102.83.12:3260 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:43:02.486320 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 -p 38.102.83.12:3260" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:43:02.486729 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[155dd996-8aa2-4f9d-a515-4454317ad77b]: (4, ('# BEGIN RECORD 6.2.1.11\nnode.name = iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2\nnode.tpgt = -1\nnode.startup = automatic\nnode.leading_login = No\niface.iscsi_ifacename = default\niface.net_ifacename = \niface.ipaddress = \niface.prefix_len = 0\niface.hwaddress = \niface.transport_name = tcp\niface.initiatorname = \niface.state = \niface.vlan_id = 0\niface.vlan_priority = 0\niface.vlan_state = \niface.iface_num = 0\niface.mtu = 0\niface.port = 0\niface.bootproto = \niface.subnet_mask = \niface.gateway = \niface.dhcp_alt_client_id_state = \niface.dhcp_alt_client_id = \niface.dhcp_dns = \niface.dhcp_learn_iqn = \niface.dhcp_req_vendor_id_state = \niface.dhcp_vendor_id_state = \niface.dhcp_vendor_id = \niface.dhcp_slp_da = \niface.fragmentation = \niface.gratuitous_arp = \niface.incoming_forwarding = \niface.tos_state = \niface.tos = 0\niface.ttl = 0\niface.delayed_ack = \niface.tcp_nagle = \niface.tcp_wsf_state = \niface.tcp_wsf = 0\niface.tcp_timer_scale = 0\niface.tcp_timestamp = \niface.redirect = \niface.def_task_mgmt_timeout = 0\niface.header_digest = \niface.data_digest = \niface.immediate_data = \niface.initial_r2t = \niface.data_seq_inorder = \niface.data_pdu_inorder = \niface.erl = 0\niface.max_receive_data_len = 0\niface.first_burst_len = 0\niface.max_outstanding_r2t = 0\niface.max_burst_len = 0\niface.chap_auth = \niface.bidi_chap = \niface.strict_login_compliance = \niface.discovery_auth = \niface.discovery_logout = \nnode.discovery_address = \nnode.discovery_port = 0\nnode.discovery_type = fw\nnode.session.initial_cmdsn = 0\nnode.session.initial_login_retry_max = 8\nnode.session.xmit_thread_priority = 0\nnode.session.cmds_max = 128\nnode.session.queue_depth = 32\nnode.session.nr_sessions = 1\nnode.session.auth.authmethod = None\nnode.session.auth.username = \nnode.session.auth.password = \nnode.session.auth.username_in = \nnode.session.auth.password_in = \nnode.session.auth.chap_algs = SHA3-256,SHA256\nnode.session.timeo.replacement_timeout = 120\nnode.session.err_timeo.abort_timeout = 15\nnode.session.err_timeo.lu_reset_timeout = 30\nnode.session.err_timeo.tgt_reset_timeout = 30\nnode.session.err_timeo.host_reset_timeout = 60\nnode.session.iscsi.FastAbort = Yes\nnode.session.iscsi.InitialR2T = No\nnode.session.iscsi.ImmediateData = Yes\nnode.session.iscsi.FirstBurstLength = 262144\nnode.session.iscsi.MaxBurstLength = 16776192\nnode.session.iscsi.DefaultTime2Retain = 0\nnode.session.iscsi.DefaultTime2Wait = 2\nnode.session.iscsi.MaxConnections = 1\nnode.session.iscsi.MaxOutstandingR2T = 1\nnode.session.iscsi.ERL = 0\nnode.session.scan = auto\nnode.session.reopen_max = 0\nnode.session.conn_reopen_log_freq = 1\nnode.conn[0].address = 38.102.83.12\nnode.conn[0].port = 3260\nnode.conn[0].startup = manual\nnode.conn[0].tcp.window_size = 524288\nnode.conn[0].tcp.type_of_service = 0\nnode.conn[0].timeo.logout_timeout = 15\nnode.conn[0].timeo.login_timeout = 15\nnode.conn[0].timeo.auth_timeout = 45\nnode.conn[0].timeo.noop_out_interval = 5\nnode.conn[0].timeo.noop_out_timeout = 5\nnode.conn[0].iscsi.MaxXmitDataSegmentLength = 0\nnode.conn[0].iscsi.MaxRecvDataSegmentLength = 262144\nnode.conn[0].iscsi.HeaderDigest = None\nnode.conn[0].iscsi.DataDigest = None\nnode.conn[0].iscsi.IFMarker = No\nnode.conn[0].iscsi.OFMarker = No\n# END RECORD\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] iscsiadm (): stdout=# BEGIN RECORD 6.2.1.11 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.name = iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.tpgt = -1 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.startup = automatic Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.leading_login = No Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.iscsi_ifacename = default Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.net_ifacename = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.ipaddress = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.prefix_len = 0 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.hwaddress = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.transport_name = tcp Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.initiatorname = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.state = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.vlan_id = 0 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.vlan_priority = 0 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.vlan_state = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.iface_num = 0 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.mtu = 0 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.port = 0 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.bootproto = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.subnet_mask = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.gateway = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_alt_client_id_state = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_alt_client_id = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_dns = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_learn_iqn = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_req_vendor_id_state = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_vendor_id_state = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_vendor_id = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.dhcp_slp_da = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.fragmentation = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.gratuitous_arp = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.incoming_forwarding = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.tos_state = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.tos = 0 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.ttl = 0 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.delayed_ack = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.tcp_nagle = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.tcp_wsf_state = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.tcp_wsf = 0 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.tcp_timer_scale = 0 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.tcp_timestamp = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.redirect = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.def_task_mgmt_timeout = 0 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.header_digest = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.data_digest = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.immediate_data = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.initial_r2t = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.data_seq_inorder = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.data_pdu_inorder = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.erl = 0 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.max_receive_data_len = 0 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.first_burst_len = 0 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.max_outstanding_r2t = 0 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.max_burst_len = 0 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.chap_auth = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.bidi_chap = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.strict_login_compliance = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.discovery_auth = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: iface.discovery_logout = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.discovery_address = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.discovery_port = 0 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.discovery_type = fw Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.initial_cmdsn = 0 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.initial_login_retry_max = 8 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.xmit_thread_priority = 0 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.cmds_max = 128 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.queue_depth = 32 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.nr_sessions = 1 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.auth.authmethod = None Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.auth.username = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.auth.password = *** Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.auth.username_in = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.auth.password_in = Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.auth.chap_algs = SHA3-256,SHA256 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.timeo.replacement_timeout = 120 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.abort_timeout = 15 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.lu_reset_timeout = 30 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.tgt_reset_timeout = 30 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.err_timeo.host_reset_timeout = 60 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.FastAbort = Yes Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.InitialR2T = No Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.ImmediateData = Yes Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.FirstBurstLength = 262144 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxBurstLength = 16776192 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.DefaultTime2Retain = 0 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.DefaultTime2Wait = 2 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxConnections = 1 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.MaxOutstandingR2T = 1 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.iscsi.ERL = 0 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.scan = auto Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.reopen_max = 0 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.session.conn_reopen_log_freq = 1 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.conn[0].address = 38.102.83.12 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.conn[0].port = 3260 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.conn[0].startup = manual Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.conn[0].tcp.window_size = 524288 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.conn[0].tcp.type_of_service = 0 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.logout_timeout = 15 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.login_timeout = 15 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.auth_timeout = 45 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.noop_out_interval = 5 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.conn[0].timeo.noop_out_timeout = 5 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.MaxXmitDataSegmentLength = 0 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144 Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.HeaderDigest = None Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.DataDigest = None Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.IFMarker = No Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: node.conn[0].iscsi.OFMarker = No Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: # END RECORD Nov 24 14:43:02.488648 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:43:02.492108 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 -p 38.102.83.12:3260 --op update -n node.session.scan -v manual {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:43:02.500277 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 -p 38.102.83.12:3260 --op update -n node.session.scan -v manual" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:43:02.500634 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[7c9dc559-0668-4c6c-abf4-bdcd1dd4ead6]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:43:02.501224 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] iscsiadm ('--op', 'update', '-n', 'node.session.scan', '-v', 'manual'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:43:02.502525 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 -p 38.102.83.12:3260 --op update -n node.session.auth.authmethod -v CHAP {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:43:02.513733 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 -p 38.102.83.12:3260 --op update -n node.session.auth.authmethod -v CHAP" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:43:02.514097 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[9a299e7f-85c9-401e-86f3-c3e675555925]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:43:02.514643 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.authmethod', '-v', 'CHAP'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:43:02.515816 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 -p 38.102.83.12:3260 --op update -n node.session.auth.username -v UF26PhGjSTbHXdKVtfU8 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:43:02.525509 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 -p 38.102.83.12:3260 --op update -n node.session.auth.username -v UF26PhGjSTbHXdKVtfU8" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:43:02.525883 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[744a123f-c3d7-4bbf-8992-dfe7bfbcfba2]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:43:02.526434 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.username', '-v', 'UF26PhGjSTbHXdKVtfU8'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:43:02.527823 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 -p 38.102.83.12:3260 --op update -n node.session.auth.password -v *** {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:43:02.539187 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 -p 38.102.83.12:3260 --op update -n node.session.auth.password -v ***" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:43:02.539524 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[e7ee6934-f0b2-4611-a4d5-7f38b403e2d6]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:43:02.540288 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.password', '-v', '***'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:43:02.541619 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:43:02.550245 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 21 in 0.008s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:43:02.550590 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[54b76142-9efc-4663-a5ed-188103fcaf92]: (4, ('', 'iscsiadm: No active sessions.\n')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:43:02.551108 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] iscsiadm ('-m', 'session'): stdout= stderr=iscsiadm: No active sessions. Nov 24 14:43:02.551108 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:43:02.551439 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] iscsi session list stdout= stderr=iscsiadm: No active sessions. Nov 24 14:43:02.551439 np0005533992.novalocal cinder-backup[122429]: {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:43:02.551814 np0005533992.novalocal cinder-backup[122429]: WARNING os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions. Nov 24 14:43:02.551814 np0005533992.novalocal cinder-backup[122429]: Nov 24 14:43:02.552985 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 -p 38.102.83.12:3260 --login {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:43:02.573282 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 -p 38.102.83.12:3260 --login" returned: 0 in 0.019s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:43:02.573603 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[78af1c9e-a7bf-4a80-ac1d-51155b10dd08]: (4, ('Login to [iface: default, target: iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2, portal: 38.102.83.12,3260] successful.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:43:02.574182 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] iscsiadm ('--login',): stdout=Login to [iface: default, target: iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2, portal: 38.102.83.12,3260] successful. Nov 24 14:43:02.574182 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:43:02.575514 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 -p 38.102.83.12:3260 --op update -n node.startup -v automatic {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:43:02.588060 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 -p 38.102.83.12:3260 --op update -n node.startup -v automatic" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:43:02.588420 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[f7c294a6-3841-4db8-98e6-cecabe9fd0d6]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:43:02.589007 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:43:02.590186 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:43:02.601848 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.012s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:43:02.602134 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[eb7fbe53-b5e6-4ae7-9411-c8756aab0122]: (4, ('tcp: [31] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:43:02.602583 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] iscsiadm ('-m', 'session'): stdout=tcp: [31] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 (non-flash) Nov 24 14:43:02.602583 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:43:02.602863 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] iscsi session list stdout=tcp: [31] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 (non-flash) Nov 24 14:43:02.602863 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:43:02.603341 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Lock "connect_to_iscsi_portal-38.102.83.12:3260-iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: held 1.159s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:43:02.603624 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Connected to 38.102.83.12:3260 {{(pid=122429) _connect_vol /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:659}} Nov 24 14:43:02.604286 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] HCTL ('2', '-', '-', 0) found on session 31 with lun 0 {{(pid=122429) get_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:788}} Nov 24 14:43:02.604630 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Scanning host 2 c: -, t: -, l: 0) {{(pid=122429) scan_iscsi /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:814}} Nov 24 14:43:02.606017 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/class/scsi_host/host2/scan {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:43:02.626543 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/class/scsi_host/host2/scan" returned: 0 in 0.020s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:43:02.626971 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[f9ea09fc-cbd1-43a4-b2d2-c64c3cfc633d]: (4, ('- - 0', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:43:02.628014 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Searching for a device in session 31 and hctl ['2', '*', '*', 0] yield: None {{(pid=122429) device_name_by_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:808}} Nov 24 14:43:03.629828 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Searching for a device in session 31 and hctl ['2', '*', '*', 0] yield: sda {{(pid=122429) device_name_by_hctl /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:808}} Nov 24 14:43:03.630742 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Connected to sda using {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2', 'target_lun': 0, 'volume_id': 'c736c436-e906-48ce-a9bd-077d081f27e2', 'auth_method': 'CHAP', 'auth_username': 'UF26PhGjSTbHXdKVtfU8', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False} {{(pid=122429) _connect_vol /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:687}} Nov 24 14:43:03.631855 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Lock "connect_volume" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" :: held 2.200s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:43:03.632220 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] <== connect_volume: return (2201ms) {'type': 'block', 'scsi_wwn': '3600140544099ea752cc437b8a8bb4ec2', 'path': '/dev/sda'} {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:43:03.632984 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sda {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:43:03.661463 np0005533992.novalocal sudo[164490]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sda Nov 24 14:43:04.013501 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 1003 /dev/sda" returned: 0 in 0.380s {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:43:04.020404 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_db.sqlalchemy.engines [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=122429) _check_effective_sql_mode /usr/local/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:397}} Nov 24 14:43:04.028893 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] _create_container started, container: tempest-volumessnapshottestjson-backup-container-809404681,backup: 95a3112e-e57e-4a79-bbe8-b4efbc83c5b5. {{(pid=122429) _create_container /opt/stack/cinder/cinder/backup/chunkeddriver.py:252}} Nov 24 14:43:04.187704 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] generate_object_name_prefix: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5 {{(pid=122429) _generate_object_name_prefix /opt/stack/cinder/cinder/backup/drivers/swift.py:433}} Nov 24 14:43:04.191344 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] starting backup of volume: df3774f1-c5c9-492e-802c-e5ce09564b66, volume size: 1073741824, object names prefix volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5, availability zone: nova {{(pid=122429) _prepare_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:362}} Nov 24 14:43:04.616209 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:43:04.750401 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:43:04.751008 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:43:04.906132 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] backup MD5 for volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00001: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:43:04.906329 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:43:05.052431 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:43:05.183843 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:43:05.184093 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:43:05.329072 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] backup MD5 for volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00002: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:43:05.329199 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:43:06.037150 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:43:06.172292 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:43:06.172430 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:43:06.303003 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] backup MD5 for volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00003: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:43:06.303302 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:43:06.481280 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:43:06.614989 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:43:06.615239 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:43:06.764833 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] backup MD5 for volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00004: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:43:06.765088 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:43:07.180386 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:43:07.326003 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:43:07.326317 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:43:07.473612 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] backup MD5 for volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00005: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:43:07.473741 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:43:07.649545 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:43:07.759192 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:43:07.759451 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:43:07.885841 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] backup MD5 for volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00006: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:43:07.886109 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:43:08.088695 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:43:08.186300 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:43:08.187090 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:43:08.289599 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] backup MD5 for volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00007: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:43:08.289763 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:43:08.657125 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:43:08.792584 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:43:08.792877 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:43:08.943668 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] backup MD5 for volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00008: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:43:08.943865 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:43:09.312328 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:43:09.446083 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:43:09.446365 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:43:09.592531 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] backup MD5 for volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00009: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:43:09.592829 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:43:09.780218 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:43:09.934408 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:43:09.934728 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:43:10.079446 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] backup MD5 for volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00010: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:43:10.079833 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:43:10.505011 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:43:10.642847 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:43:10.643186 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:43:10.789305 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] backup MD5 for volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00011: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:43:10.789548 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:43:10.934816 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:43:11.066953 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:43:11.067091 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:43:11.225436 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] backup MD5 for volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00012: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:43:11.225760 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:43:11.408740 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:43:11.542239 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:43:11.542806 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:43:11.692954 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] backup MD5 for volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00013: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:43:11.692954 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:43:11.857109 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:43:11.996312 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:43:11.996596 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:43:12.139411 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] backup MD5 for volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00014: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:43:12.139747 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:43:12.357260 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:43:12.455717 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:43:12.455960 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:43:12.611663 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] backup MD5 for volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00015: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:43:12.611663 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:43:12.748769 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:43:12.856594 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:43:12.856848 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:43:13.003078 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] backup MD5 for volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00016: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:43:13.003205 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:43:13.183801 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:43:13.375601 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:43:13.375736 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:43:13.519112 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] backup MD5 for volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00017: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:43:13.519266 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:43:13.689875 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:43:13.834107 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:43:13.834473 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:43:13.989715 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] backup MD5 for volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00018: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:43:13.989715 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:43:14.212425 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:43:14.343717 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:43:14.344052 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:43:14.495031 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] backup MD5 for volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00019: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:43:14.495293 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:43:14.671771 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:43:14.805383 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Compressed 52428800 bytes of data to 50976 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:43:14.805784 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:43:14.970819 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] backup MD5 for volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00020: 25e317773f308e446cc84c503a6d1f85 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:43:14.971083 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:43:15.107935 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Backing up chunk of data from volume. {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:394}} Nov 24 14:43:15.197989 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Compressed 25165824 bytes of data to 24470 bytes using zlib. {{(pid=122429) _prepare_output_data /opt/stack/cinder/cinder/backup/chunkeddriver.py:433}} Nov 24 14:43:15.198071 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] About to put_object {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:397}} Nov 24 14:43:15.310479 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] backup MD5 for volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00021: 77377273b0a4b61febdbf7bbf52b9db9 {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:405}} Nov 24 14:43:15.310670 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Calling eventlet.sleep(0) {{(pid=122429) _backup_chunk /opt/stack/cinder/cinder/backup/chunkeddriver.py:412}} Nov 24 14:43:15.315317 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Getting metadata type 'volume-base-metadata' {{(pid=122429) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Nov 24 14:43:15.323801 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=122429) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Nov 24 14:43:15.323982 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Getting metadata type 'volume-metadata' {{(pid=122429) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Nov 24 14:43:15.327511 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] No metadata type 'volume-metadata' available {{(pid=122429) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Nov 24 14:43:15.327717 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Getting metadata type 'volume-glance-metadata' {{(pid=122429) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Nov 24 14:43:15.331165 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.driver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] No metadata type 'volume-glance-metadata' available {{(pid=122429) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Nov 24 14:43:15.331466 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] _write_sha256file started, container name: tempest-volumessnapshottestjson-backup-container-809404681, sha256file filename: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5_sha256file. {{(pid=122429) _write_sha256file /opt/stack/cinder/cinder/backup/chunkeddriver.py:302}} Nov 24 14:43:15.407199 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] _write_sha256file finished. {{(pid=122429) _write_sha256file /opt/stack/cinder/cinder/backup/chunkeddriver.py:318}} Nov 24 14:43:15.408190 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] _write_metadata started, container name: tempest-volumessnapshottestjson-backup-container-809404681, metadata filename: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5_metadata. {{(pid=122429) _write_metadata /opt/stack/cinder/cinder/backup/chunkeddriver.py:279}} Nov 24 14:43:15.444247 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] _write_metadata finished. Metadata: b'{\n "backup_description": null,\n "backup_id": "95a3112e-e57e-4a79-bbe8-b4efbc83c5b5",\n "backup_name": "tempest-VolumesSnapshotTestJSON-Backup-1226447601",\n "created_at": "2025-11-24 19:42:55+00:00",\n "objects": [\n {\n "volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00001": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 0\n }\n },\n {\n "volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00002": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 52428800\n }\n },\n {\n "volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00003": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 104857600\n }\n },\n {\n "volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00004": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 157286400\n }\n },\n {\n "volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00005": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 209715200\n }\n },\n {\n "volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00006": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 262144000\n }\n },\n {\n "volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00007": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 314572800\n }\n },\n {\n "volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00008": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 367001600\n }\n },\n {\n "volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00009": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 419430400\n }\n },\n {\n "volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00010": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 471859200\n }\n },\n {\n "volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00011": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 524288000\n }\n },\n {\n "volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00012": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 576716800\n }\n },\n {\n "volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00013": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 629145600\n }\n },\n {\n "volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00014": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 681574400\n }\n },\n {\n "volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00015": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 734003200\n }\n },\n {\n "volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00016": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 786432000\n }\n },\n {\n "volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00017": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 838860800\n }\n },\n {\n "volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00018": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 891289600\n }\n },\n {\n "volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00019": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 943718400\n }\n },\n {\n "volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00020": {\n "compression": "zlib",\n "length": 52428800,\n "md5": "25e317773f308e446cc84c503a6d1f85",\n "offset": 996147200\n }\n },\n {\n "volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00021": {\n "compression": "zlib",\n "length": 25165824,\n "md5": "77377273b0a4b61febdbf7bbf52b9db9",\n "offset": 1048576000\n }\n }\n ],\n "parent_id": null,\n "version": "1.0.0",\n "volume_id": "df3774f1-c5c9-492e-802c-e5ce09564b66",\n "volume_meta": "{\\"version\\": 2, \\"volume-base-metadata\\": {\\"id\\": \\"df3774f1-c5c9-492e-802c-e5ce09564b66\\", \\"_name_id\\": null, \\"use_quota\\": true, \\"ec2_id\\": null, \\"user_id\\": \\"9f1963a65a4341d5b36078d1273654b9\\", \\"project_id\\": \\"f851d93dc40a47668f073fe029189315\\", \\"snapshot_id\\": null, \\"cluster_name\\": null, \\"host\\": \\"np0005533992.novalocal@lvmdriver-1#lvmdriver-1\\", \\"size\\": 1, \\"availability_zone\\": \\"nova\\", \\"status\\": \\"available\\", \\"attach_status\\": \\"detached\\", \\"migration_status\\": null, \\"scheduled_at\\": \\"2025-11-24T19:42:53.000000\\", \\"launched_at\\": \\"2025-11-24T19:42:54.000000\\", \\"terminated_at\\": null, \\"display_name\\": \\"tempest-VolumesSnapshotTestJSON-Volume-870092034\\", \\"display_description\\": null, \\"provider_location\\": null, \\"provider_auth\\": null, \\"provider_geometry\\": null, \\"provider_id\\": null, \\"volume_type_id\\": \\"1f8776b6-c527-4643-876a-506285adfc66\\", \\"source_volid\\": null, \\"encryption_key_id\\": null, \\"consistencygroup_id\\": null, \\"group_id\\": null, \\"bootable\\": false, \\"multiattach\\": false, \\"replication_status\\": null, \\"replication_extended_status\\": null, \\"replication_driver_data\\": null, \\"previous_status\\": null, \\"service_uuid\\": \\"6d0941fd-19ec-48ae-82b1-f25526175227\\", \\"shared_targets\\": false, \\"deleted_at\\": null, \\"deleted\\": false, \\"created_at\\": \\"2025-11-24T19:42:53.000000\\", \\"updated_at\\": \\"2025-11-24T19:42:54.000000\\"}}"\n}'. {{(pid=122429) _write_metadata /opt/stack/cinder/cinder/backup/chunkeddriver.py:298}} Nov 24 14:43:15.449378 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] backup 95a3112e-e57e-4a79-bbe8-b4efbc83c5b5 finished. {{(pid=122429) _finalize_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:465}} Nov 24 14:43:15.551475 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sda {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:43:15.576405 np0005533992.novalocal sudo[164510]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sda Nov 24 14:43:15.895547 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sda" returned: 0 in 0.344s {{(pid=122429) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:43:15.897184 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] ==> disconnect_volume: call "{'args': (, {'target_discovered': False, 'target_portal': '38.102.83.12:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2', 'target_lun': 0, 'volume_id': 'c736c436-e906-48ce-a9bd-077d081f27e2', 'auth_method': 'CHAP', 'auth_username': 'UF26PhGjSTbHXdKVtfU8', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw', 'cacheable': False, 'enforce_multipath': False}, {'type': 'block', 'scsi_wwn': '3600140544099ea752cc437b8a8bb4ec2', 'path': '/dev/sda'}), 'kwargs': {'force': True}}" {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:177}} Nov 24 14:43:15.897511 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Acquiring lock "connect_volume" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:68}} Nov 24 14:43:15.898222 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: waited 0.001s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:73}} Nov 24 14:43:15.899694 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m discoverydb -o show -P 1 {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:43:15.910576 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m discoverydb -o show -P 1" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:43:15.910907 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[4481cd3e-2784-4ff8-9a4e-519346a0e472]: (4, ('SENDTARGETS:\nNo targets found.\niSNS:\nNo targets found.\nSTATIC:\nTarget: iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2\n\tPortal: 38.102.83.12:3260,-1\n\t\tIface Name: default\nFIRMWARE:\nNo targets found.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:43:15.911379 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] iscsiadm ['-m', 'discoverydb', '-o', 'show', '-P', 1]: stdout=SENDTARGETS: Nov 24 14:43:15.911379 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:43:15.911379 np0005533992.novalocal cinder-backup[122429]: iSNS: Nov 24 14:43:15.911379 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:43:15.911379 np0005533992.novalocal cinder-backup[122429]: STATIC: Nov 24 14:43:15.911379 np0005533992.novalocal cinder-backup[122429]: Target: iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 Nov 24 14:43:15.911379 np0005533992.novalocal cinder-backup[122429]: Portal: 38.102.83.12:3260,-1 Nov 24 14:43:15.911379 np0005533992.novalocal cinder-backup[122429]: Iface Name: default Nov 24 14:43:15.911379 np0005533992.novalocal cinder-backup[122429]: FIRMWARE: Nov 24 14:43:15.911379 np0005533992.novalocal cinder-backup[122429]: No targets found. Nov 24 14:43:15.911379 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:43:15.911736 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Regex to get portals from discoverydb: ^SENDTARGETS: Nov 24 14:43:15.911736 np0005533992.novalocal cinder-backup[122429]: .*?^DiscoveryAddress: 38.102.83.12,3260.*? Nov 24 14:43:15.911736 np0005533992.novalocal cinder-backup[122429]: (.*?)^(?:DiscoveryAddress|iSNS):.* {{(pid=122429) _get_discoverydb_portals /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:392}} Nov 24 14:43:15.912161 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Getting connected devices for (ips,iqns,luns)=[('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2', 0)] {{(pid=122429) _get_connection_devices /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:855}} Nov 24 14:43:15.913230 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:43:15.924133 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:43:15.924432 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[5920f00c-fb44-426b-b67d-7cabde7d0639]: (4, ('38.102.83.12:3260,4294967295 iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:43:15.925432 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:43:15.935492 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:43:15.935747 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[9ee11f16-2afa-458e-a618-62e5c7c8d19f]: (4, ('tcp: [31] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 (non-flash)\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:43:15.936172 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] iscsiadm ('-m', 'session'): stdout=tcp: [31] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 (non-flash) Nov 24 14:43:15.936172 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm_bare /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1223}} Nov 24 14:43:15.936425 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] iscsi session list stdout=tcp: [31] 38.102.83.12:3260,1 iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 (non-flash) Nov 24 14:43:15.936425 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsi_session /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1212}} Nov 24 14:43:15.937364 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Resulting device map defaultdict(. at 0x7f5b90fdf600>, {('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2'): ({'sda'}, set())}) {{(pid=122429) _get_connection_devices /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:888}} Nov 24 14:43:15.937796 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Removing single pathed devices sda {{(pid=122429) remove_connection /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:381}} Nov 24 14:43:15.938799 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd show status {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:43:15.946280 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] 'multipathd show status' failed. Not Retrying. {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:423}} Nov 24 14:43:15.948819 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[0f68b892-3c73-4409-b883-606f6209de95]: [Errno 2] No such file or directory: 'multipathd' {{(pid=159746) _process_cmd /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:492}} Nov 24 14:43:15.948819 np0005533992.novalocal cinder-backup[122429]: Traceback (most recent call last): Nov 24 14:43:15.948819 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd Nov 24 14:43:15.948819 np0005533992.novalocal cinder-backup[122429]: ret = func(*f_args, **f_kwargs) Nov 24 14:43:15.948819 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:43:15.948819 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap Nov 24 14:43:15.948819 np0005533992.novalocal cinder-backup[122429]: return func(*args, **kwargs) Nov 24 14:43:15.948819 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:43:15.948819 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root Nov 24 14:43:15.948819 np0005533992.novalocal cinder-backup[122429]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Nov 24 14:43:15.948819 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:43:15.948819 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute Nov 24 14:43:15.948819 np0005533992.novalocal cinder-backup[122429]: return putils.execute(on_execute=on_execute, Nov 24 14:43:15.948819 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:43:15.948819 np0005533992.novalocal cinder-backup[122429]: File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute Nov 24 14:43:15.948819 np0005533992.novalocal cinder-backup[122429]: obj = subprocess.Popen(cmd, Nov 24 14:43:15.948819 np0005533992.novalocal cinder-backup[122429]: ^^^^^^^^^^^^^^^^^^^^^ Nov 24 14:43:15.948819 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__ Nov 24 14:43:15.948819 np0005533992.novalocal cinder-backup[122429]: self._execute_child(args, executable, preexec_fn, close_fds, Nov 24 14:43:15.948819 np0005533992.novalocal cinder-backup[122429]: File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child Nov 24 14:43:15.948819 np0005533992.novalocal cinder-backup[122429]: raise child_exception_type(errno_num, err_msg, err_filename) Nov 24 14:43:15.948819 np0005533992.novalocal cinder-backup[122429]: FileNotFoundError: [Errno 2] No such file or directory: 'multipathd' Nov 24 14:43:15.950645 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[0f68b892-3c73-4409-b883-606f6209de95]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory'), 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 489, in _process_cmd\n ret = func(*f_args, **f_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 270, in _wrap\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 198, in execute_root\n return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute\n return putils.execute(on_execute=on_execute,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute\n obj = subprocess.Popen(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__\n self._execute_child(args, executable, preexec_fn, close_fds,\n File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child\n raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: \'multipathd\'\n') {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:43:15.951204 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Flushing IO for device /dev/sda {{(pid=122429) flush_device_io /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:441}} Nov 24 14:43:15.952071 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blockdev --flushbufs /dev/sda {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:43:15.959603 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "blockdev --flushbufs /dev/sda" returned: 0 in 0.007s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:43:15.959817 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[7dc62153-de2d-45f1-b9af-28b5a4c89dcd]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:43:15.960221 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Remove SCSI device /dev/sda with /sys/block/sda/device/delete {{(pid=122429) remove_scsi_device /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:151}} Nov 24 14:43:15.961227 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/block/sda/device/delete {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:43:16.007328 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/block/sda/device/delete" returned: 0 in 0.046s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:43:16.007688 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[85d6a893-aabf-45a8-9a88-1ac29bc9fd64]: (4, ('1', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:43:16.008388 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Checking to see if SCSI volumes sda have been removed. {{(pid=122429) wait_for_volumes_removal /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:159}} Nov 24 14:43:16.008835 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.linuxscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] SCSI volumes sda have been removed. {{(pid=122429) wait_for_volumes_removal /usr/local/lib/python3.12/site-packages/os_brick/initiator/linuxscsi.py:169}} Nov 24 14:43:16.009359 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Disconnecting from: [('38.102.83.12:3260', 'iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2')] {{(pid=122429) _disconnect_connection /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1201}} Nov 24 14:43:16.010505 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 -p 38.102.83.12:3260 --op update -n node.startup -v manual {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:43:16.021875 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 -p 38.102.83.12:3260 --op update -n node.startup -v manual" returned: 0 in 0.011s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:43:16.022489 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[36497f5c-b130-48f8-8b18-c2b2cf541957]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:43:16.022812 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:43:16.024374 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 -p 38.102.83.12:3260 --logout {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:43:16.053311 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 -p 38.102.83.12:3260 --logout" returned: 0 in 0.029s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:43:16.053794 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[5c46d04d-a36c-4d80-b0eb-75104242a382]: (4, ('Logging out of session [sid: 31, target: iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2, portal: 38.102.83.12,3260]\nLogout of [sid: 31, target: iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2, portal: 38.102.83.12,3260] successful.\n', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:43:16.054013 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] iscsiadm ('--logout',): stdout=Logging out of session [sid: 31, target: iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2, portal: 38.102.83.12,3260] Nov 24 14:43:16.054013 np0005533992.novalocal cinder-backup[122429]: Logout of [sid: 31, target: iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2, portal: 38.102.83.12,3260] successful. Nov 24 14:43:16.054013 np0005533992.novalocal cinder-backup[122429]: stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:43:16.054844 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 -p 38.102.83.12:3260 --op delete {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} Nov 24 14:43:16.064762 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c736c436-e906-48ce-a9bd-077d081f27e2 -p 38.102.83.12:3260 --op delete" returned: 0 in 0.010s {{(pid=159746) execute /usr/local/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} Nov 24 14:43:16.065131 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo.privsep.daemon [-] privsep: reply[4875824f-b0d9-4e15-8663-9e3c62aa8572]: (4, ('', '')) {{(pid=159746) _call_back /usr/local/lib/python3.12/site-packages/oslo_privsep/daemon.py:515}} Nov 24 14:43:16.065318 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] iscsiadm ('--op', 'delete'): stdout= stderr= {{(pid=122429) _run_iscsiadm /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/iscsi.py:1067}} Nov 24 14:43:16.065628 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.base [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Lock "connect_volume" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: held 0.168s {{(pid=122429) inner /usr/local/lib/python3.12/site-packages/os_brick/initiator/connectors/base.py:87}} Nov 24 14:43:16.065721 np0005533992.novalocal cinder-backup[122429]: DEBUG os_brick.initiator.connectors.iscsi [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] <== disconnect_volume: return (168ms) None {{(pid=122429) trace_logging_wrapper /usr/local/lib/python3.12/site-packages/os_brick/utils.py:204}} Nov 24 14:43:19.610389 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-1221aed5-80f9-454b-ae95-92869b7aeb08 tempest-VolumesSnapshotTestJSON-1740062394 None] Create backup finished. backup: 95a3112e-e57e-4a79-bbe8-b4efbc83c5b5. Nov 24 14:43:19.836369 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] Delete backup started, backup: 95a3112e-e57e-4a79-bbe8-b4efbc83c5b5. Nov 24 14:43:19.840210 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] Using swift URL https://38.102.83.12:8080/v1/AUTH_f851d93dc40a47668f073fe029189315 {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:286}} Nov 24 14:43:19.840491 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.drivers.swift [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] Connect to https://38.102.83.12:8080/v1/AUTH_ in "per_user" mode {{(pid=122429) initialize /opt/stack/cinder/cinder/backup/drivers/swift.py:287}} Nov 24 14:43:19.840878 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] delete started, backup: 95a3112e-e57e-4a79-bbe8-b4efbc83c5b5, container: tempest-volumessnapshottestjson-backup-container-809404681, prefix: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:844}} Nov 24 14:43:19.922174 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] generated object list: ['volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00001', 'volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00002', 'volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00003', 'volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00004', 'volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00005', 'volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00006', 'volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00007', 'volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00008', 'volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00009', 'volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00010', 'volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00011', 'volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00012', 'volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00013', 'volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00014', 'volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00015', 'volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00016', 'volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00017', 'volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00018', 'volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00019', 'volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00020', 'volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00021', 'volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5_metadata', 'volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5_sha256file']. {{(pid=122429) _generate_object_names /opt/stack/cinder/cinder/backup/chunkeddriver.py:263}} Nov 24 14:43:19.948448 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] deleted object: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00001 in container: tempest-volumessnapshottestjson-backup-container-809404681. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:43:19.973455 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] deleted object: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00002 in container: tempest-volumessnapshottestjson-backup-container-809404681. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:43:19.999657 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] deleted object: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00003 in container: tempest-volumessnapshottestjson-backup-container-809404681. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:43:20.024638 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] deleted object: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00004 in container: tempest-volumessnapshottestjson-backup-container-809404681. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:43:20.049493 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] deleted object: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00005 in container: tempest-volumessnapshottestjson-backup-container-809404681. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:43:20.077823 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] deleted object: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00006 in container: tempest-volumessnapshottestjson-backup-container-809404681. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:43:20.106195 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] deleted object: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00007 in container: tempest-volumessnapshottestjson-backup-container-809404681. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:43:20.137519 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] deleted object: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00008 in container: tempest-volumessnapshottestjson-backup-container-809404681. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:43:20.169495 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] deleted object: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00009 in container: tempest-volumessnapshottestjson-backup-container-809404681. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:43:20.202974 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] deleted object: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00010 in container: tempest-volumessnapshottestjson-backup-container-809404681. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:43:20.231741 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] deleted object: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00011 in container: tempest-volumessnapshottestjson-backup-container-809404681. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:43:20.262426 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] deleted object: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00012 in container: tempest-volumessnapshottestjson-backup-container-809404681. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:43:20.290884 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] deleted object: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00013 in container: tempest-volumessnapshottestjson-backup-container-809404681. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:43:20.320866 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] deleted object: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00014 in container: tempest-volumessnapshottestjson-backup-container-809404681. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:43:20.350231 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] deleted object: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00015 in container: tempest-volumessnapshottestjson-backup-container-809404681. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:43:20.378356 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] deleted object: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00016 in container: tempest-volumessnapshottestjson-backup-container-809404681. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:43:20.407416 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] deleted object: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00017 in container: tempest-volumessnapshottestjson-backup-container-809404681. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:43:20.437110 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] deleted object: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00018 in container: tempest-volumessnapshottestjson-backup-container-809404681. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:43:20.466596 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] deleted object: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00019 in container: tempest-volumessnapshottestjson-backup-container-809404681. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:43:20.495910 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] deleted object: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00020 in container: tempest-volumessnapshottestjson-backup-container-809404681. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:43:20.526108 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] deleted object: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5-00021 in container: tempest-volumessnapshottestjson-backup-container-809404681. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:43:20.554048 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] deleted object: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5_metadata in container: tempest-volumessnapshottestjson-backup-container-809404681. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:43:20.580610 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] deleted object: volume_df3774f1-c5c9-492e-802c-e5ce09564b66/20251124194304/az_nova_backup_95a3112e-e57e-4a79-bbe8-b4efbc83c5b5_sha256file in container: tempest-volumessnapshottestjson-backup-container-809404681. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:860}} Nov 24 14:43:20.580961 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.backup.chunkeddriver [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] delete 95a3112e-e57e-4a79-bbe8-b4efbc83c5b5 finished. {{(pid=122429) delete_backup /opt/stack/cinder/cinder/backup/chunkeddriver.py:870}} Nov 24 14:43:20.585823 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_db.sqlalchemy.engines [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=122429) _check_effective_sql_mode /usr/local/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:397}} Nov 24 14:43:20.607674 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.quota [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] Created reservations ['df215fcc-d5fb-4507-89d1-f20a2c18c916', 'fe386ec9-4104-4797-869e-39c586fc1909'] {{(pid=122429) reserve /opt/stack/cinder/cinder/quota.py:752}} Nov 24 14:43:20.631462 np0005533992.novalocal cinder-backup[122429]: INFO cinder.backup.manager [None req-4ad3599b-9976-404b-9899-7ae28563c698 tempest-VolumesSnapshotTestJSON-1740062394 None] Delete backup finished, backup 95a3112e-e57e-4a79-bbe8-b4efbc83c5b5 deleted. Nov 24 14:43:30.626788 np0005533992.novalocal cinder-backup[122429]: DEBUG dbcounter [-] [122429] Writing DB stats cinder:SELECT=41,cinder:UPDATE=12,cinder:INSERT=2 {{(pid=122429) stat_writer /usr/local/lib/python3.12/site-packages/dbcounter.py:115}} Nov 24 14:43:46.437165 np0005533992.novalocal cinder-backup[122429]: DEBUG dbcounter [-] [122429] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=122429) stat_writer /usr/local/lib/python3.12/site-packages/dbcounter.py:115}} Nov 24 14:43:46.439467 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:43:46.439875 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:44:46.444383 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:44:46.444383 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:45:46.436795 np0005533992.novalocal cinder-backup[122429]: DEBUG dbcounter [-] [122429] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=122429) stat_writer /usr/local/lib/python3.12/site-packages/dbcounter.py:115}} Nov 24 14:45:46.441019 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:45:46.441252 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:46:46.445228 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:46:46.445808 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:47:46.438511 np0005533992.novalocal cinder-backup[122429]: DEBUG dbcounter [-] [122429] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=122429) stat_writer /usr/local/lib/python3.12/site-packages/dbcounter.py:115}} Nov 24 14:47:46.450445 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:47:46.450835 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:48:46.456114 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:48:46.456590 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:49:46.436288 np0005533992.novalocal cinder-backup[122429]: DEBUG dbcounter [-] [122429] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=122429) stat_writer /usr/local/lib/python3.12/site-packages/dbcounter.py:115}} Nov 24 14:49:46.452761 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:49:46.453056 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:50:46.457243 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:50:46.457887 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:51:46.436444 np0005533992.novalocal cinder-backup[122429]: DEBUG dbcounter [-] [122429] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=122429) stat_writer /usr/local/lib/python3.12/site-packages/dbcounter.py:115}} Nov 24 14:51:46.461531 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:51:46.461762 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:52:46.467418 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:52:46.467749 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:53:46.436398 np0005533992.novalocal cinder-backup[122429]: DEBUG dbcounter [-] [122429] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=122429) stat_writer /usr/local/lib/python3.12/site-packages/dbcounter.py:115}} Nov 24 14:53:46.463655 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:53:46.463882 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:54:46.469372 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:54:46.469372 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:55:46.439164 np0005533992.novalocal cinder-backup[122429]: DEBUG dbcounter [-] [122429] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=122429) stat_writer /usr/local/lib/python3.12/site-packages/dbcounter.py:115}} Nov 24 14:55:46.473539 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:55:46.473800 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:56:46.478481 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:56:46.478896 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:57:46.439109 np0005533992.novalocal cinder-backup[122429]: DEBUG dbcounter [-] [122429] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=122429) stat_writer /usr/local/lib/python3.12/site-packages/dbcounter.py:115}} Nov 24 14:57:46.485154 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:57:46.485154 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:58:46.479124 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:58:46.479487 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 14:59:46.438048 np0005533992.novalocal cinder-backup[122429]: DEBUG dbcounter [-] [122429] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=122429) stat_writer /usr/local/lib/python3.12/site-packages/dbcounter.py:115}} Nov 24 14:59:46.474833 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 14:59:46.475042 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Nov 24 15:00:46.481113 np0005533992.novalocal cinder-backup[122429]: DEBUG oslo_service.periodic_task [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=122429) run_periodic_tasks /usr/local/lib/python3.12/site-packages/oslo_service/periodic_task.py:210}} Nov 24 15:00:46.481113 np0005533992.novalocal cinder-backup[122429]: DEBUG cinder.manager [None req-a5212422-deb9-4610-ac42-4320c442ce66 None None] Notifying Schedulers of capabilities ... {{(pid=122429) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}