Guild icon
S3Drive
Community / support / Problem downloading huge number of files
Avatar
User experiences issues with reliably selecting and then downloading media files. Originally from @Emission, snapshot:
Deleted the app. What's the point of Cloud storage if you can't access it in a practical way from another device?
The app only lets me select 1000 or 2000 photos while there are 5700 in thr folder, so you can't really download them all back to device either.
Can't select more than 3000 files. Eventhough there are 5534 in the folder. Can't scroll further down than 3000 and select all just increment in steps of 1000 and only if you scroll past that number.
Any solutions? I really want my photos back.
Installed again. Tried again. Doesn't work. App is buggy.
Rebuilt cache several times. Deleted app again. Pity, cause I purchased the 1200 GB plan and was really hopeful. But the app needs soms serious work. I know it's work in progress and I don't want a refund, but can you please help me out here? Tried also without VPN.
Avatar
Hi @Emission, Thanks for your feedback. If you need to reliably download/upload couple hundreds, thousands (or more) files then my advice would be to skip upload/download/selection UI part and set up Sync instead. Sync is pretty rock-solid and has built-in retry, verification mechanism to make sure that all files that you've chose to upload/download will end up where they're supposed to. In the future we'll certainly improve native app download/upload queue persistency, retry and verification, but given difficulty of this task and the fact that there is actually an alternative, we've been focused on other feature items: https://s3drive.canny.io/feature-requests/p/upload-retry-improvements-including-timeout-4xx-5xx https://s3drive.canny.io/feature-requests/p/improve-transfers-tab-persistence Please let me know if that's any help/use for you.
Files > 100M are uploaded using multipart upload using 5MB chunks. If upload gets interrupted, next time when file gets uploaded we verify MD5 (only if E2E
Internal task to improve robustness of the Transfers tab. Part of this task is to migrate away from Hive and use Isar.
Avatar
Ok. I'll try that. Thanks.
7:44 AM
So sync the folder instead of select all files in a certain folder?
7:45 AM
And after that set it to sync one way so I can delete photos locally but keep them in the cloud?
Avatar
Avatar
Emission
So sync the folder instead of select all files in a certain folder?
Pretty much yes. It will internally take care of each file. Selecting 5000 files through UI reliably, is quite impractical and whilst I hope we will get there eventually, it's not that easy to reliably implement.
Avatar
I understand.
Avatar
Avatar
Emission
And after that set it to sync one way so I can delete photos locally but keep them in the cloud?
It really depends what are you up to. My understanding is that you've already backed up your media to the cloud. Once you set up sync from cloud to e.g. DCIM folder then you will create a copy of it. I would advice to have your media stored in at least 3 different locations (e.g. your phone, your desktop and cloud), not just cloud alone.
Avatar
Yes I downloaded them all again from my NAS.
7:52 AM
But in future I would like to sync my NAS with the cloud as well.
Avatar
Avatar
Emission
But in future I would like to sync my NAS with the cloud as well.
You can use Media Backup to auto-backup your media from Android to cloud. You can then setup Sync jobs (be aware there are different modes, for backups Copy is preferred, for sync, Sync is preferred - check attached screenshot) to make sure that relevant locations end up where they're supposed to. E.g. Using S3Drive, you can set up on your desktop periodic sync from cloud (media auto backup location) to NAS... additionally you can set up whole NAS (or just selected folder) to be backed up back to S3Drive (or other remote location). Depending on the NAS, it may be possible to set up periodic sync jobs using Rclone CLI (without installing S3Drive GUI). Depending on the NAS you could try connecting to it directly using (e.g. WebDav) and make Media Backup upload to some specific location. This workflow isn't something I would always recommend due to security implications. Depending on NAS and/or your network admin setup it might open up your NAS unnecessarily to the internet. There are plenty options, challenge is understanding what you really need and finding what works for you exactly. (edited)
Avatar
Ok. I use Truenas. So it'll be my first backup point. From there I want to backup the folders on my NAS to the cloud. My NAS would be the primary source locally. But when I'm on the road and want to access mh files. I wouldn't want to access my NAS through port forwarding. (Perhaps later with an overlay network like Netbird or Headscale/Tailscale) What I seek to accomplish is when I alter or delete a file on a remote device outside of my LAN, it syncs it with the NAS and all the other clients. I let Truenas handle versioning and Snapshots. So when a file gets deleted on say a laptop on the road, it should be synced to be deleted on another laptop or phone on another location. But the NAS keeps a backup (snapshot) of that file for a period of time or until manually deleted from the snapshots.
Avatar
Avatar
Emission
Ok. I use Truenas. So it'll be my first backup point. From there I want to backup the folders on my NAS to the cloud. My NAS would be the primary source locally. But when I'm on the road and want to access mh files. I wouldn't want to access my NAS through port forwarding. (Perhaps later with an overlay network like Netbird or Headscale/Tailscale) What I seek to accomplish is when I alter or delete a file on a remote device outside of my LAN, it syncs it with the NAS and all the other clients. I let Truenas handle versioning and Snapshots. So when a file gets deleted on say a laptop on the road, it should be synced to be deleted on another laptop or phone on another location. But the NAS keeps a backup (snapshot) of that file for a period of time or until manually deleted from the snapshots.
Not an expert regarding best strategy. Given your approach, it may be worth trying two-way sync between cloud <> NAS. If you're locally, you modify NAS. If you're on the road, you modify cloud. Bear in mind that in principle this shouldn't be treated as backup, but more like live working copies. If you screw up on NAS, it will get propagated to cloud and vice-versa. Given that your NAS handles snapshot and versioning this might not be a big deal though. I would probably setup two-way sync job on the Truenas box using https://rclone.org/commands/rclone_bisync/ and Cron (with a lock file to prevent running another job before previous finalizes), running e.g. every 15 minutes (depending on the needs). You would also need to monitor its output and resolve conflicts from time to time. We certainly plan to support some workflows out of the box and provide better integrations with NAS (e.g. https://s3drive.canny.io/feature-requests/p/file-offloading-cloud-nas-tiered-storage), but we're not there yet. (edited)
Perform bidirectional synchronization between two paths.
Avatar
I'll look into that. Thanks. I used to have Synology Drive. It was excellent for syncing. The software is fantastic. But their hardware is very expensive and recently vendor locked in. I'm trying to build something like that Synology Drive feature with backup to 2 clouds.
9:14 AM
One live cloud. One Archive cloud.
9:15 AM
I can also create 2 buckets. One for syncing. One for backup retentions.
9:15 AM
Just a thought.
9:15 AM
Thanks for advice.
Exported 18 message(s)
Timezone: UTC+0