S3Drive - macOS mount issues

Continuation of: https://discord.com/channels/1069654792902815845/1069654792902815848/1365642460486828093

The files in the app and on the mounted drive are not the same (app seems to be up-to-date)

Unfortunately whilst app will always show up-to-date view, mount might not (due to caching). There is a relevant feature item we need to address: https://s3drive.canny.io/feature-requests/p/drive-mount-webdav-sync-changes-online-with-s3drive If you keep using mount only, the cache shall be intact. As long as you edit your storage outside of mount, the mount itself won't reflect that until some time (likely couple minutes) or until remounted.

Copying a larger file (few MBs up to GBs) into the mounted drive causes an IO error.

We'll look on this issue on macOS mount specifically. By any chance if you feel like doing so, please give it a go trying to connect to external storage (now that you have an Ultimate trial) and see if problem also applies for external storage.

For each file I copy, there seems to be a hidden shadow copy of the same file (. file prefix)

Where can you see these, is that S3Drive app or in the mount folder? I am wondering if these are just temporary files that rclone may produce or something that's being uploaded and persisted to remote storage.

Finder and sometime also the app is not responding for a while until the transfer stops and reports error

I would be curious to know if same happens if you use any external storage. We're aware that macOS mount performance is subpar, although in principle it should be usable, which by your description sounds like it isn't. There is mid-long term feature item that will address most of these issues by introducing native macOS mount: https://s3drive.canny.io/feature-requests/p/macos-native-file-mount In the meantime, we'll try doing our best to improve the FUSE mount, so if you find any more patterns where it fails, freeze, please let us know. (edited)

I try to provide an update with external storage.

Regarding:

Copying a larger file (few MBs up to GBs) into the mounted drive causes an IO error. Doesn't matter whether I use Finder, Forklift or the CLI

and

Finder and sometime also the app is not responding for a while until the transfer stops and reports error

Can you try setting: Mount cache mode to Full? We've noticed in recent versions that it's actually much slower when different caching mode is used.

Hi Tom, I did last evening some more testing, but haven't seen your most recent post. I installed a pending MacOS Update and rebooted twice to be sure, having a clean booted system. Almost everything has been resolved by that. The mount seems to be much more stable, and I can copy and paste data with Finder / Forklist as well as with cp in the CLI. I guess either the pending MacOS played not so nicely into the system's stability or a missing reboot after the S3Drive installation. The only thing that I still could observe - but was confirmed from your side - the App and Fuse mount aren't immediately in-sync. But I guess this is okay. As a result of the Fuse mount internals, one can copy files, but the real upload kicks in much later, after the copy job has been done (observed by monitoring bandwidth and traffic, which started to increase way after the copy job. Does this mean:

that a computer must stay online to no disrupt the real upload? Is there any indicator when it's safe to go into sleep?
that initially the files will be copied to a temp directory? Is this a file system copy again or an emulated CIFS / NFS transfer?
that my drive (which path mount / partition) must provide at least the double capacity of the file until it's uploaded? Sorry, that I'm asking that much. It's absolutely not to criticize the product. Quite the opposite, love to find a new solution and to move away from Google / Proton. I do some last tests these days by cloning my Proton drive to my private MinIO storage and observe how it behaves.

Another observation (upload through S3Drive, not mount), uploaded / synced files keep their metadata such as timestamps whereas folder are reset to 01.01.2000. Not a big deal for me, but probably something that could be fixed (if unintended).

that a computer must stay online to no disrupt the real upload? Is there any indicator when it's safe to go into sleep?

That's right, there isn't indicator as such, but I've just added a feature request to improve that: https://s3drive.canny.io/feature-requests/p/drive-mount-pending-uploads-indicator

that initially the files will be copied to a temp directory? Is this a file system copy again or an emulated CIFS / NFS transfer?

Yes, that would be temp directory copy: https://rclone.org/commands/rclone_mount/#vfs-file-caching with the default location of: --cache-dir string Directory rclone will use for caching. I can find out exactly what path is that on macOS.

that my drive (which path mount / partition) must provide at least the double capacity of the file until it's uploaded?

Unfortunately yes, whilst it me sound sub-optimal, that's deliberate decision that Rclone authors have made. I guess one reason would be to lock the file version as it was exactly, without actually performing lock on the original file. If you need "in-place" method to transfer files then using Sync (from the S3Drive tab) might be a better choice. If you need "block" copy, that is copying only part of the file that was edited, then protocols like: https://linux.die.net/man/1/rsync might be a better choice.

Drive mount pending uploads indicator | Feature Requests | S3Drive

Currently when user uploads files through the mount they don't exactly have clue if file is already uploaded and whether they can safely disconnect the app.

rclone mount

Mount the remote as file system on a mountpoint.

11:08 AM

Sorry, that I'm asking that much. It's absolutely not to criticize the product. Quite the opposite, love to find a new solution and to move away from Google / Proton.

We're not afraid of constructive criticism and we love your feedback so far!

dannyyy

That's the rclone limitation which corresponds to limitations of most remote back-ends where folder timestamps aren't supported. On S3 for instance, folders are just emulated prefixes of the underlying paths and don't hold any metadata. Maybe in the future there will be consensus what hacky approach to use to store timestamp on the folder level. Related: https://forum.rclone.org/t/need-to-persist-dates-for-directories-with-dropbox-how/39675/15