S3Drive - general

Kakarot1925 joined the server. 7/29/2023 8:39 AM

Hi @Kakarot1925, welcome to S3Drive !

Tom

Hi @zer0, thanks ! What platform(s) are you using? I could send you the promo code for Android. Web has no paywall at the moment. Other platforms require registering an account. Since we're closely integrated with Stripe we don't yet have a workflow for "enabling Pro" outside of it, however if you get it, I could offer you a refund once you're done testing.

I do use Android

zer0

I do use Android

Cool, I've sent you the code, if you need anything just let me know.

Chernsha joined the server. 7/29/2023 10:33 AM

The new rclone stuff is nice

1

12:52 PM

Will test large file downloads from browser

Hmm I cannot seem to access the files using rclone

1:24 AM

What's the correct way to configure?

Tamara joined the server. 7/30/2023 2:28 AM

zer0

Hmm I cannot seem to access the files using rclone

If you use mount feature on Desktop you can find the exact commands in the application logs. If file content encryption together with filename encryption is used, then please find below sample config.

# Obscure password
echo "YourPlaintextPassword" | rclone obscure -

# Add it to Rclone config, config file location: `rclone config file`
[s3drive_remote]
type = s3
provider = Other
access_key_id = <access_key_id>
secret_access_key = <secret_access_key>
endpoint = <endpoint>
region = <region>

[s3drive_crypt]
type = crypt
filename_encoding = base64
remote = s3drive_remote:<bucket_name>
password = <obscuredPassword>
filename_encryption = standard
directory_name_encryption = true
suffix = none

Then you can use: s3drive_crypt as your remote encrypted location. Please note that whilst we support both encrypted and unencrypted files in the same location, Rclone doesn't seem to like the mix and won't display existing unencrypted files for the encrypted remote. In such case it's better to either keep everything encrypted globally or have dedicate paths with encrypted-only or unencrypted-only files. (edited)

1

Morethanevil joined the server. 7/31/2023 6:40 AM

Deleted User joined the server. 7/31/2023 9:10 AM

Deleted User

Click to see original message

Deleted User

Click to see original message

9:53 AM

Hi

just on the encryption theme, i don't know what the process is so this might seem a newbie question, but i'm assuming if you encrypt and access your bucket on a mobile device, then goto the desktop and use the same pw etc and try and view the mobile encrypted data you can do that? ie the key to decrypt is with the id and pw so is interchangeable between devices?

I have a question about the pro version.. I bought it via Playstore and now I wanted to know, how I can connect the desktop app to it?

Buzz69

just on the encryption theme, i don't know what the process is so this might seem a newbie question, but i'm assuming if you encrypt and access your bucket on a mobile device, then goto the desktop and use the same pw etc and try and view the mobile encrypted data you can do that? ie the key to decrypt is with the id and pw so is interchangeable between devices?

Yes

1

Morethanevil

I have a question about the pro version.. I bought it via Playstore and now I wanted to know, how I can connect the desktop app to it?

Play store only unlocks android app

Okay so I can not create an account in the app with pro enabled and link it to desktop app?

Morethanevil

I have a question about the pro version.. I bought it via Playstore and now I wanted to know, how I can connect the desktop app to it?

Hi and thank you for supporting us. It seems you've bought the Pro just on Android. If you need Pro working on all devices you would need to register an account and buy it through our website. When you use that account on any device it will obtain the license from our server and enable you the Pro. For iOS/Android, we as a developer were forced to use respective in-app payments, by default that's assigned to your Apple/Google account and it's not visible on other S3Drive clients. We haven't implemented any propagation mechanism and weren't sure about the exact approach. Speaking of how you manage credentials between devices, at the moment they need to be manually copied over, but we're looking to improve that: https://s3drive.canny.io/feature-requests/p/qr-code-credentials-exchange If you're not happy with your purchase, I could make an exception and we could switch the "plans" over.

Morethanevil

Okay so I can not create an account in the app with pro enabled and link it to desktop app?

It may work like that in the future, but that's not the case yet... and even if it was, the price would have to be different as technically you would buy lifetime version on all devices. The difference is that on our website we sell perpetual fallback license, whereas on iOS/Google we're forced (EDIT: well there are also subscriptions, but there isn't concept of perpetual fallback license - https://s3drive.app/faq?q=perpetual as such) to sell the lifetime with their 15% cut. (edited)

Thanks for the explanation, I will wait for QR Code feature, it is the easiest way atm. I got a warning after executing the installer from windows defender.

10:58 AM

For the Linux version Flatpak would be nicer than appimage, because it is updateable

Morethanevil

Thanks for the explanation, I will wait for QR Code feature, it is the easiest way atm. I got a warning after executing the installer from windows defender.

Microsoft implements some heuristic mechanism to flag the executable if not enough people used the package. We've recently switched location of where we host our packages to Github release: https://github.com/s3drive/app/releases It might go away eventually, we're also thinking of buying "trusted certificate" from some provider, which should mitigate these warnings sooner. Alternatively you can use MSIX package, but we update it rarely due to pretty crappy experience with Microsoft Store: https://apps.microsoft.com/store/detail/s3drive-cloud-storage/9NX2DN9Q37NS (edited)

Ahh cool, please link the store app on github too

Morethanevil

For the Linux version Flatpak would be nicer than appimage, because it is updateable

Flatpak is pending: https://github.com/flathub/flathub/pull/4364 however there is some licensing stuff we're not sure about and didn't have time to do the legal research. It will come probably in a few weeks. (edited)

Flatpak is as great as the store app. They can selfupdate, so it is easier to manage versions. You can be listed on flathub too

11:03 AM

= more users

Morethanevil

Flatpak is as great as the store app. They can selfupdate, so it is easier to manage versions. You can be listed on flathub too

Yes, we think the same. It's just there are two ways the "proprietary" packages can be distributed and the technical approach is quite different. One way or another, it will appear on Flathub. As you say more users and pretty convenient to distribute.

1

I installed the store app, works perfect without warnings. Should be the prefered method for installing, and exe for people who don't like the store

Morethanevil

I installed the store app, works perfect without warnings. Should be the prefered method for installing, and exe for people who don't like the store

Point taken. We'll have it updated, as the version is quite old.

1

Morethanevil

I installed the store app, works perfect without warnings. Should be the prefered method for installing, and exe for people who don't like the store

Just to give you an idea, how we love this process. This is the first hand experience trying to login to dev account:

After clearing cookies different one. Sorry, I've just had to vent

If it don't works as expected, than add a hint about the warning from the installer. A notice is better than a surprise i think

1

Just hot the new update on Android, thanks

I will try out

️

Update works well so far, but I don't get the E2EE running. I set a new crypt in rclone, with only one password, since s3drive does not support 2 passwords. I set filenames and directory names to encrypt too. Then I added the bucket to the app and activated the encryption with the same password. I get access, but folders and filesnames stay encrypted. Did I miss something? Then I saw that my other buckets are missing in the app, after I added the encrypted one. The profiles got overwritten. Securitysettings should be moved to bucket settings, since they only apply to the current bucket. It is a little confusing at first. Dark mode and dotfiles are appwide settings.

6:40 AM

Steps to reproduce on the missing profiles: I already had unecnrypted buckets. I added another one and set encrypted with password from rclone. Since it did not work, I tried to toggle off and on again to set the password again (Maybe I mistyped) and then the other buckets were gone

6:41 AM

Import / Export settings would be great in this case

Morethanevil

Update works well so far, but I don't get the E2EE running. I set a new crypt in rclone, with only one password, since s3drive does not support 2 passwords. I set filenames and directory names to encrypt too. Then I added the bucket to the app and activated the encryption with the same password. I get access, but folders and filesnames stay encrypted. Did I miss something? Then I saw that my other buckets are missing in the app, after I added the encrypted one. The profiles got overwritten. Securitysettings should be moved to bucket settings, since they only apply to the current bucket. It is a little confusing at first. Dark mode and dotfiles are appwide settings.

Thanks for your input. Please find the other post with sample Rclone config: https://discord.com/channels/1069654792902815845/1069654792902815848/1135157727216279585 For filename encryption to work, please make sure that these two landed in your settings:

filename_encoding = base64
suffix = none

By default the Rclone's encoding is base32: https://github.com/rclone/rclone/blob/88c72d1f4de94a5db75e6b685efdbe525adf70b8/backend/crypt/crypt.go#L140 unless overriden by the config creator.

Morethanevil

Steps to reproduce on the missing profiles: I already had unecnrypted buckets. I added another one and set encrypted with password from rclone. Since it did not work, I tried to toggle off and on again to set the password again (Maybe I mistyped) and then the other buckets were gone

In principle this shouldn't happen, the E2EE settings regardless of UI are applied per S3 credentials separately. If your other credentials are lost, well it sounds like we may've screwed up. I am going to reproduce it. Is this Android?

Morethanevil

Import / Export settings would be great in this case

It will definitely come soon.

Morethanevil

Update works well so far, but I don't get the E2EE running. I set a new crypt in rclone, with only one password, since s3drive does not support 2 passwords. I set filenames and directory names to encrypt too. Then I added the bucket to the app and activated the encryption with the same password. I get access, but folders and filesnames stay encrypted. Did I miss something? Then I saw that my other buckets are missing in the app, after I added the encrypted one. The profiles got overwritten. Securitysettings should be moved to bucket settings, since they only apply to the current bucket. It is a little confusing at first. Dark mode and dotfiles are appwide settings.

For the 2nd password, I've just added this item: https://s3drive.canny.io/feature-requests/p/support-2nd-rclone-crypt-password Rclone support is still "hot" and our aim was to deliver MVP with contents, filepath encryption and drive mount. We also hope that we'll be able to influence Rclone crypt improvements first, as it would affect the end derived key for the file encryption: https://github.com/rclone/rclone/issues/7192 (edited)

Morethanevil

Steps to reproduce on the missing profiles: I already had unecnrypted buckets. I added another one and set encrypted with password from rclone. Since it did not work, I tried to toggle off and on again to set the password again (Maybe I mistyped) and then the other buckets were gone

When you say: "Since it did not work", at the moment when you enable/disable filename encryption and go back to the listing, you need to explicitly refresh it on the S3Drive side in order for changes to be applied to your existing already loaded listing. This will get improved as well, so it's less confusing. (edited)

Tom

Thanks for your input. Please find the other post with sample Rclone config: https://discord.com/channels/1069654792902815845/1069654792902815848/1135157727216279585 For filename encryption to work, please make sure that these two landed in your settings:

filename_encoding = base64
suffix = none

By default the Rclone's encoding is base32: https://github.com/rclone/rclone/blob/88c72d1f4de94a5db75e6b685efdbe525adf70b8/backend/crypt/crypt.go#L140 unless overriden by the config creator.

It works now, second notice was the hint from the app to encode the password in base64 using https://www.base64encode.org and then enter the password as base64, not plaintext. Is there a possibilty to enter a plain password, which is then calculated in base64?

Base64 Encode and Decode - Online

Encode to Base64 format or decode from it with various advanced options. Our site has an easy to use online tool to convert your data.

Tom

In principle this shouldn't happen, the E2EE settings regardless of UI are applied per S3 credentials separately. If your other credentials are lost, well it sounds like we may've screwed up. I am going to reproduce it. Is this Android?

Yes Android Playstore release from yesterday

Tom

For the 2nd password, I've just added this item: https://s3drive.canny.io/feature-requests/p/support-2nd-rclone-crypt-password Rclone support is still "hot" and our aim was to deliver MVP with contents, filepath encryption and drive mount. We also hope that we'll be able to influence Rclone crypt improvements first, as it would affect the end derived key for the file encryption: https://github.com/rclone/rclone/issues/7192 (edited)

One password is okay for the beginning, it works now. Maybe a quickstart guide with screenshots would be nice, so people can reproduce

Morethanevil

It works now, second notice was the hint from the app to encode the password in base64 using https://www.base64encode.org and then enter the password as base64, not plaintext. Is there a possibilty to enter a plain password, which is then calculated in base64?

Yes, please click: "Start" instead of "Existing key (base64)". The existing key is legacy option from previous AES-GCM encryption. It will work for Rclone if you properly encode your password, but it's mostly to satisfy previous user password if they want to decrypt files encrypted using legacy encryption.

Morethanevil

One password is okay for the beginning, it works now. Maybe a quickstart guide with screenshots would be nice, so people can reproduce

We're working on some documentation pages and tutorials using: https://squidfunk.github.io/mkdocs-material/ It wasn't our priority when features API wasn't exactly stable, but now things are getting final shape.

1

Tom

Yes, please click: "Start" instead of "Existing key (base64)". The existing key is legacy option from previous AES-GCM encryption. It will work for Rclone if you properly encode your password, but it's mostly to satisfy previous user password if they want to decrypt files encrypted using legacy encryption.

I thought start would setup a new password and encryption. This is why i used existing

Tom

Thanks for your input. Please find the other post with sample Rclone config: https://discord.com/channels/1069654792902815845/1069654792902815848/1135157727216279585 For filename encryption to work, please make sure that these two landed in your settings:

filename_encoding = base64
suffix = none

By default the Rclone's encoding is base32: https://github.com/rclone/rclone/blob/88c72d1f4de94a5db75e6b685efdbe525adf70b8/backend/crypt/crypt.go#L140 unless overriden by the config creator.

BTW. I am wondering in the end if you had to set Base64 encoding (in the Rclone config) explicitly or was it already set by the Rclone's config creator? (edited)

Zev joined the server. 8/2/2023 10:06 AM

Welcome @Zev

(edited)

Tom

BTW. I am wondering in the end if you had to set Base64 encoding (in the Rclone config) explicitly or was it already set by the Rclone's config creator? (edited)

The config creator uses base32 as default, as you said before. It only shows options if you say yes to the advanced configuration when creating the config file

Just ran the configuration assistant

Morethanevil

Just ran the configuration assistant

Great, so indeed possible.

Morethanevil

Steps to reproduce on the missing profiles: I already had unecnrypted buckets. I added another one and set encrypted with password from rclone. Since it did not work, I tried to toggle off and on again to set the password again (Maybe I mistyped) and then the other buckets were gone

There is a concurrency issue combined with a bug in library that we use: https://pub.dev/packages/flutter_secure_storage. When setting E2E password, we run the key derivation: https://rclone.org/crypt/#key-derivation which takes some xxx ms. When user taps/clicks on password confirmation, more than once, this all starts over and chances are it will clean up the other credentials. We need to apply quick workaround first, so we have time to properly fix this in the underlying library. (edited)

Tom

There is a concurrency issue combined with a bug in library that we use: https://pub.dev/packages/flutter_secure_storage. When setting E2E password, we run the key derivation: https://rclone.org/crypt/#key-derivation which takes some xxx ms. When user taps/clicks on password confirmation, more than once, this all starts over and chances are it will clean up the other credentials. We need to apply quick workaround first, so we have time to properly fix this in the underlying library. (edited)

I will setup the buckets again and start with the encrypted one

10:49 AM

But it is great that you found the bug, so a workaround or fix can be made

10:53 AM

I wanted to use my backups a better way than just rclone via sftp. So I learned about MinIO with Versioning and so on... It is a little tricky to bind a user to a bucket but I found a solution for this. Then I wanted to use S3 on android too, but there were no good apps. Mostly they only work with AWS, your app was the first one which worked and has not a UI from the 1990. Now with the encryption and the app, I am able to restore files and Folders without copy rclone.conf over, than run a long mounting and hoping to get my file

Morethanevil

I wanted to use my backups a better way than just rclone via sftp. So I learned about MinIO with Versioning and so on... It is a little tricky to bind a user to a bucket but I found a solution for this. Then I wanted to use S3 on android too, but there were no good apps. Mostly they only work with AWS, your app was the first one which worked and has not a UI from the 1990. Now with the encryption and the app, I am able to restore files and Folders without copy rclone.conf over, than run a long mounting and hoping to get my file

Great to hear, I am amazed to see that there are so many different user stories with S3Drive. Speaking of "bind a user to a bucket", is it something you would like to share more about? Is it MinIO specific or something we could help with S3Drive as well?

It is for MinIO only, the user managment is aweful there

11:00 AM

Every user with read write can create and delete every bucket.

11:00 AM

Useful ACLs are missing

11:00 AM

But it is possible via JSON Format to bind a user to one bucket, I was looking around last night and finally found a solution in a blogpost

11:01 AM

https://blog.nikhilbhardwaj.in/2020/02/25/minio-bucket-policy/

Minio Bucket Policy Notes

Minio is a really cool opensource project which democratizes cloud storage. The feature that I love most about it is S3 compatibility which means that you ca...

11:01 AM

{ "Version": "2012-10-17", "Statement": [ { "Action": [ "s3:ListBucket", "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::BUCKET-NAME/*", "arn:aws:s3:::BUCKET-NAME" ], "Sid": "Description" } ] }

11:01 AM

Please copy from the blog, not discord

11:01 AM

Discord always screws up

11:02 AM

As for S3 Drive it could check the permissions from the user and do an autodiscovery for the buckets. Then the user could select the ones to have in the app

11:03 AM

The only thing to do would be setup the passwords for encrypted buckets

11:03 AM

Serverside encryption is aweful too in minio, you need a kms

Morethanevil

It is for MinIO only, the user managment is aweful there

I see, I believe Minio's goal was to build a replacement for AWS S3, with all of the IAM / KMS clunky, but enterprise-ready stuff

Morethanevil

As for S3 Drive it could check the permissions from the user and do an autodiscovery for the buckets. Then the user could select the ones to have in the app

On the S3 login screen you can select single bucket manually, but autodiscovery sounds also interesting. Technically we could ask user (dialog) and import all buckets as separate profiles (or perhaps there is a neater way to manage it). (edited)

A policy generator for the json files would be cool from the devs... selecvt user, select bucket, select ACL...

Tom

On the S3 login screen you can select single bucket manually, but autodiscovery sounds also interesting. Technically we could ask user (dialog) and import all buckets as separate profiles (or perhaps there is a neater way to manage it). (edited)

If you use more than 5 buckets it is a little work to add them all, and retype the credentials everytime

Morethanevil

If you use more than 5 buckets it is a little work to add them all, and retype the credentials everytime

FYI, https://s3drive.canny.io/feature-requests/p/bucket-auto-discovery

Thanks

️ I upvoted

if you want a self-service MinIO policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::${aws:username}"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::${aws:username}/*"
            ]
        }
    ]
}

11:13 AM

that's what I always add to any MinIO instance I configure, full control over your own bucket that's your username

And it only allows the user to use his own buckets?

only a single bucket that is their own username

11:42 AM

they can even delete it or create it if it doesn't exist

11:44 AM

you can always replace ${aws:username} by anything you want, be it a variable or a fixed bucket name, there unfortunately isn't any group name variable

So the first aws:username means the username of the user and the second one is for the bucket? I am not much into the policy thing, the docs are not very userfriendly

no, both time it's the bucket name

11:45 AM

the policy can be assigned to either a user or a group

11:46 AM

in my case I have a users group to which I assign the selfservice policy, then I add whoever I want to the users group and they'll be able to manage their very own bucket

Okay so the only difference to the other would be, that your json provides full access to the bucket, including deletion and creation

it can be simplified a lot actually, I kept the resources separate but you can merge them

Xenthys®

you can always replace ${aws:username} by anything you want, be it a variable or a fixed bucket name, there unfortunately isn't any group name variable

That's cool

One policy for a group and then just add users

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::${aws:username}",
                "arn:aws:s3:::${aws:username}/*"
            ]
        }
    ]
}

11:48 AM

by default I don't merge them because the first resource is the bucket itself, while the other is for its content

11:51 AM

I also had a "public-ro" and a "public-rw" policy, useful if you want all users to be able to access a common bucket but only allow a subgroup to write there

Morethanevil

Okay so the only difference to the other would be, that your json provides full access to the bucket, including deletion and creation

yeah, I'm afraid the one you sent doesn't allow multiparts or versions usage, it also allows users to set policies on their buckets if they want to allow other users to access it, or make a directory public

I will try this after work, thank you

Xenthys®

if you want a self-service MinIO policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::${aws:username}"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::${aws:username}/*"
            ]
        }
    ]
}

Username needs to be replaced with an existing bucket? How do I use this with groups?

no, that one can be used as-is