S3Drive - Android file mount

It would be awesome to have the content of the S3 mounts as Android File provider. I am not an Android developer but the Davx5 app does exactly that with their WebDav mounts. I went ahead and scanned their AndroidManifest.xml and this is how they do it (plus implementation of course). https://github.com/bitfireAT/davx5-ose/blob/273deecbe49b9f0c5ae753353ad0f8a514c4c401/app/src/main/AndroidManifest.xml#L288-L296 Thank you for your hard work, using S3 Drive and liking it a lot!

davx5-ose/app/src/main/AndroidManifest.xml at 273deecbe49b9f0c5ae75...

DAVx⁵ is an open-source CalDAV/CardDAV suite and sync app for Android. You can also access your online files (WebDAV) with it. - bitfireAT/davx5-ose

3

Hi @Ari, Thanks for requesting this feature and providing some examples. We've got it on the radar: https://s3drive.canny.io/feature-requests/p/android-virtual-file-system, we don't have any exact ETA, but since it's pretty important feature we will have it implemented rather sooner than later.

Hey thank you! It would definitely save me some sync troubles. My current workflow is:

download keepass db from S3 Drive
keepass merge from local
upload db to S3 Drive I could not find a better way but maybe there are others

Hi @Ari, I've wanted to understand this feature little bit better. At the moment you can open Keepass DB using S3Drive, the issue is that you can't save it back and have it synced automatically. I believe that you need to use "Save as" and then manually upload it to the remote location using S3Drive. If we allowed to "Save" back to the same file and then detected the file change and subsequently synced file using S3Drive automatically would this resolve your use case? Is there anything else that you were proposing with the DavX5 example?

yes, basically DAV mounts are visible in Android's main menu when you save

6:58 PM

Keepassxc contains a save as feature but you cannot see the "Mount point" because S3 drive does not register a provider

7:00 PM

this is what I see when I trigger the Save To

7:00 PM

oops cannot send screenshots?

I see, so it's actually different thing. I haven't investigated the "Mount point" as such, so far I've checked the workflow that can be used with e.g. Google Drive. In other words you can open file and don't need to use "Save as" to select the "Mount point", but instead you simply click Save and the Google Drive can pick up the change. That's something we could implement initially. In the long-run we could implement both (including the File Provider mount). (edited)

1

7:03 PM

I see g drive but no option for S3 drive yeah

Fair enough, thanks for the example. We're going to investigate how we could implement it for S3Drive.

1

Tom changed the channel name: Android file mount 9/6/2023 10:41 AM

There is a clunky workaround before this gets implemented. Since S3Drive is 1:1 compatible with Rclone format (including the encryption) it would be possible to setup the WebDav endpoint to S3 using Rclone: https://rclone.org/commands/rclone_serve_webdav/ and then mount it using the DavX5. This would require hosting Rclone on a desktop / server though. It may be possible to actually do it from within the mobile (no desktop/sever required) using Rclone for Android: https://github.com/x0b/rcx (it seem to support encryption and WebDav), but I haven't tried that option. Since 1.5.1 S3Drive (due to be released) there will be an option to set up WebDav server on desktop. This also could be used together with DavX5. For a truly independent mobile experience though we need to implement it natively. We're building spec and will be able provide some timelines soon.

Yeah I think that would work - I was actually thinking of mounting the S3 folder with s3fs on linux and expose it via a WebDav server (cause Davx can use that as file provider)

3:46 PM

Thank you Tom, I would honestly gladly pay for such a premium feature (alongside the rest of S3 drive) cause it saves so much time

@Ari WebDav server is already available on desktop clients: https://s3drive.app/changelog Alternatively you can use: s3fs and run the WebDav server yourself as you say, but you could also achieve the same with our native rclone mount which is likely going to be more performant than a POSIX compatible s3fs.

Changelog - S3Drive

I'll have to try rclone - do you happen to know how come it is more performant then a file system mount?

3:20 PM

s3fs is indeed quite slow here

In both cases they are FUSE mounts (virtual), but Rclone has additional VFS caching: https://rclone.org/commands/rclone_mount/#vfs-file-caching There is a nice comparison of 3 different approaches: http://gaul.org/talks/s3fs-tradeoffs/#1 Tradeoffs: http://gaul.org/talks/s3fs-tradeoffs/#24 If I am correct, Rclone approach is similar to goofys, whereas s3fs offers maximum POSIX compatibility at huge cost. E.g. listing directory with 1000 files will take up to 1000! requests with s3fs, however it will take just one with rclone / goofys. (edited)

1

thank you that's great to know!

@Tom sorry to ask you this but I am not an expert here I wonder what the implications are for me if there is no uid/gid support... Does that mean that if mounted locally I won't be able to specify group permissions for, say, a dir?

I, too, was wondering if this is possible for Android. So, if I'm understanding this request correctly, we're able to mount S3Drive as a file system to replace Google Drive/One Drive? If so, that is something I'd be interested in. However, self-hosting a server is something I'd like to avoid.

Ari

@Tom sorry to ask you this but I am not an expert here I wonder what the implications are for me if there is no uid/gid support... Does that mean that if mounted locally I won't be able to specify group permissions for, say, a dir?

@Ari In principle S3 and S3Drive as is currently doesn't support/preserve native permissions or FS metadata (e.g. date modified / created). It is possible to emulate this (and that's what s3fs does - if I am correct) by storing that metadata on S3. The issue is that there is no easy way to query such data. Native S3 listing methods return up to 1000 results at once, but support only couple fields (file name, size, data uploaded etc.) and doesn't list any additional metadata, such metadata must be queried separately. Separate query means 1 request per single file, so basically s3fs in order to be POSIX compatible in some cases needs to issue 1000x times more requests. This has dramatic performance consequences. You're right you won't be able to specify group permissions for a dir. (Actually it may work, but such data won't be preserved when you remount). (edited)

1

I see, thanks! Maybe I don't need Posix if I use rclone, I definitely need it for exposing it via web server cause I don't want to use root or root group (edited)

6:57 PM

the dir I am using has got ten files and it is kinda slow

InfiniteAds559

I, too, was wondering if this is possible for Android. So, if I'm understanding this request correctly, we're able to mount S3Drive as a file system to replace Google Drive/One Drive? If so, that is something I'd be interested in. However, self-hosting a server is something I'd like to avoid.

Can't tell exactly when, but native mount for Android will come this year without any Webdav mounts or other tricks. There is some development required which isn't entirely easy given our cross-platform framework which is great, but some native functionality requires little bit more work.

1

oh no worries you guys are doing great work already and as I said I'd gladly pay for such a feature

1

Ari

I see, thanks! Maybe I don't need Posix if I use rclone, I definitely need it for exposing it via web server cause I don't want to use root or root group (edited)

I am not exactly sure what's your setup / use case, but if I mount bucket directly from S3Drive (using tray icon), then the mount has the same user permission that app uses. No root required. Permissions of file&dirs: -rw-rw-r-- which is pretty default Rclone setting, but in principle we could add settings configuration to change it.

yeah I am not using a GUI though on the server, if your app works as daemon I can do that

Ari

yeah I am not using a GUI though on the server, if your app works as daemon I can do that

We haven't actually thought about this use case, but given that this mount can be directly replaced with rclone command it's probably not a high priority. When you go to the app logs, you will see exactly what commands app executed on rclone binary. You can replicate the same on your server if you wish.

cool

7:09 PM

I'll try that out at some point

Great, if you need my input I am happy to help out. I am pretty excited to see S3Drive to be used in a different scenarios

1

FWIW I followed your advice @Tom and I am now mounting my folder with goofys - it's another ball game

Ari

FWIW I followed your advice @Tom and I am now mounting my folder with goofys - it's another ball game

Great to hear that. We've been playing with goofys and even had an AWS client-side encryption compatible prototype, ultimately we've replaced it with Rclone, however experience with goofys and its codebase was pretty good. (edited)

I'm interested in the same thing for iOS, using File Provider API. Same use-case of KeePass DB (edited)

Tom

Great to hear that. We've been playing with goofys and even had an AWS client-side encryption compatible prototype, ultimately we've replaced it with Rclone, however experience with goofys and its codebase was pretty good. (edited)

Client-side encryption is also important for my use case but fortunately KeePass databases are already encrypted

Paper

I'm interested in the same thing for iOS, using File Provider API. Same use-case of KeePass DB (edited)

We definitely plan to support provider API's on both Android and iOS, we're still looking for viable options of implementing this, as given our cross-platform framework this implementation isn't the easiest. Can't promise anything, but do hope we can get it running this quarter.

Ari

Client-side encryption is also important for my use case but fortunately KeePass databases are already encrypted

I am not sure if I missed that, but was there any specific reason you went for goofys instead of https://rclone.org/commands/rclone_mount/ ? Did it perform better or perhaps being pure S3 was easier to set up?

goofys was easier to setup - I just added a line in my /etc/fstab

7:20 PM

rclone would have been "a very tiny bit" more work (like, writing/finding a systemd service or a wrapper script) (edited)

I was hoping Flutter would take care of it

That perfectly makes sense. Actually interesting how such a relatively small points play role in decision making.

Paper

I was hoping Flutter would take care of it

On the contrary

yeah - well you are right, this is my home server and I have to do things during my "free" time - which of course is already close to zero

7:23 PM

(I know you perfectly know what I am talking about lol) (edited)

7:30 PM

one quirk of goofys is that it does not seem to work with the systemd automount feature (that mounts on demand)

7:32 PM

oh wait a second - this rclone feature would have solved my issue

rclone mount

Mount the remote as file system on a mountpoint.

Hi @Ari, How are things? I've just wanted to let you know since couple weeks there is a new sync feature which may satisfy your Keepass workflow (https://discord.com/channels/1069654792902815845/1144327683073908797/1147971175415304274). You can use: "Open with" feature on Android and open file externally in Keepass app. When you save your file this change shall be captured by the S3Drive and synced back to your S3 back-end. Similar mechanism apply for any other file type, e.g. you can open .txt file externally, modify it and then when you save it back it will be captured by the S3Drive. We are still working on File provider as requested, but perhaps this can also improve your workflow. I would be keen to hear if this works for you, thanks ! (edited)

Discord - A New Way to Chat with Friends & Communities

Discord is the easiest way to communicate over voice, video, and text. Chat, hang out, and stay close with your friends and communities.

I will definitely try out, seems to fit well