rsync - Rackspace cloudfiles - files appear as folders in file listing and don't appear in rclone - Stack Overflow

admin管理员组

文章数量:1395352

I am attempting to migrate some legacy storage buckets from Rackspace cloudfiles to an S3 provider. I'm using rclone to do this and the copy process has largely worked successfully. I say largely because....

I have discovered that some files are omitted from the copy. When I investigate the affected path/folder to these files in the source bucket, I find what appear to be empty folders with the name of the file e.g. somevideoname.mp4 (156Mb). If I directly download these I get an empty folder, but if I download the "folder" via it's Akamai CDN link, I get the video file.

Testing for missing files using this approach in rsync shows no differences:

rclone check racksyd:mybucketwasabi:mybucket --one-way --missing-
on-dst=mybucket-missing.txt --progress

I speculate that these files were originally uploaded as file parts and assembled via a manifest into the file. For some reason nothing is visible in rackspace other than an "empty" folder. Here's a screenshot of one of these (from Cyberduck OSX software)

I've put an enquiry into Rackspace support, but has anyone run across this before? I would be interested if there are any additional flags or options in rclone I can use to ensure these files are included in my copy and to allow me to check the completed copies for true fidelity without omitting these files.

I am attempting to migrate some legacy storage buckets from Rackspace cloudfiles to an S3 provider. I'm using rclone to do this and the copy process has largely worked successfully. I say largely because....

I have discovered that some files are omitted from the copy. When I investigate the affected path/folder to these files in the source bucket, I find what appear to be empty folders with the name of the file e.g. somevideoname.mp4 (156Mb). If I directly download these I get an empty folder, but if I download the "folder" via it's Akamai CDN link, I get the video file.

Testing for missing files using this approach in rsync shows no differences:

rclone check racksyd:mybucketwasabi:mybucket --one-way --missing-
on-dst=mybucket-missing.txt --progress

I speculate that these files were originally uploaded as file parts and assembled via a manifest into the file. For some reason nothing is visible in rackspace other than an "empty" folder. Here's a screenshot of one of these (from Cyberduck OSX software)

I've put an enquiry into Rackspace support, but has anyone run across this before? I would be interested if there are any additional flags or options in rclone I can use to ensure these files are included in my copy and to allow me to check the completed copies for true fidelity without omitting these files.

• rsync
• rackspace-cloud
• rackspace-cloudfiles

Share Improve this question Follow edited Mar 31 at 7:00 Mark Rotteveel 110k229229 gold badges156156 silver badges224224 bronze badges asked Mar 27 at 4:54 rodneytrodneyt 14299 bronze badges 1

• OK progress, Rackspace Object Storage, uses a pseudo-hierarchy to make it human-friendly to manage it, please refer to the next documentation for a deeper understanding of it docs-ospc.rackspace/cloud-files/v1/general-api-info/… It seems Swiftly CLI tool is more reliable. Will update as I learn more. – rodneyt Commented Mar 28 at 2:26

Add a comment  | 

1 Answer 1

Sorted by: Reset to default -1

For anyone else arriving here. The solution was provided by Rackspace support. I went with the option of editing the get.py file below which worked for me.

Here's an example command to recursively download files/folders. Note trailing slash on local path folder.

swiftly --auth-url=https://identity.api.rackspacecloud/v2.0 --auth-user=MYRACKUSER --auth-key=MYAPIKEY --auth-methods=auth2key,auth2password --region=MY_REGION_eg_SYD --concurrency=10 get MY_BUCKETNAME -p MY_PATH_WITHIN_BUCKET_TO_FOLDER -o '/MYLOCALPATH/FOLDER/' --all-objects

Swiftly is actually a Python 2.7 application, this usually happens when using Python 3.  So you can either:

1. Use Python 2.7

2. Or you can fix this you can edit get.py that's included with Swiftly. For me this was in /usr/local/lib/python3.9/site-packages/swiftly/cli/get.py. If you're using virtualenv then it's most likely in a similar path within that environment. You will need to change line 235 from:

new_path = path + '/' + item['name'].encode('utf8')

to

new_path = path + '/' + item['name']#.encode('utf8')

本文标签： rsyncRackspace cloudfilesfiles appear as folders in file listing and don39t appear in rcloneStack Overflow