Guild icon
S3Drive
Community / general
For all on-topic discussion about S3Drive or related storage providers.
Avatar
Just realized there are concerns with sharing data because idiots will share illegal material. Maybe stay out of the business of hosting the data for now. I'm just thinking out loud but I'll leave the above writing there in case anyone finds it helpful.
Avatar
Avatar
myfrogger
Thank you for the detailed responses. I'm using I use Windows and iPhone. My initial thought with the favicon is the current folder with the letters S3 on it. I'd have to do a bit of thinking and asking around at identifying the target market for this product. I'm not even sure S3Drive is the best name for marketing purposes. I don't dislike it though; just rambling. As far as those coming from Dropbox, Onedrive, etc, etc.... I think that the E2EE/control or own your data is a significant selling point and also the "pay for what you need". I don't know why all these companies have different "packages" when it seems easier to onboard people to a pay-as-you-go model. Apple seems to be the only one that figured it out that millions of people don't care about a $0.99 fee but they do care about $5-10. I think that you could monetize by having your own built in cloud (simply putting everyone's data into your own BackBlaze or Wasabi account). At least then when someone downloads the app, there is a usable setting without any further configuration. Maybe offer X 50gb free storage like everyone else. I remember reading years ago that getting someone to use the app the moment they download it is critical to keeping them as a user. I think you should embrace accepting bitcoin, lightning, and maybe some other crypto payments. Also embrace the community that wants their data private. Don't ask for names, don't log their details, etc. For cards use stripe. Maybe this not a project that you'll monetize but I feel like this could be a full time job, and an opportunity to make a great product that people would pay for. Anyway---I agree with you that you can't do everything at once, and the priorities appear to be ordered in a logical way. Keep up the good work 🙂 I'm happy to be an early user and maybe able to contribute somehow along the way. (edited)
@myfrogger You're absolutely spot on. We're researching what cloud we could use or whether we need to build our own one (likely MinIO based). Ideally we would pay predictable amount for the resources and resell it at some margin as a complete solution back to the users. The issue with Wasabi, Backblaze and many other clouds is that we can't control the costs as a business (there are many components) and can't easily specify quota per user account, so we don't bankrupt if one user decides to abuse the S3 credentials. Building own cloud on the other hand is additional chunk of work that we would like to avoid, more importantly it comes with huge responsibility, some legal burden and maintenance costs (infrastructure updates, security fixes, SLA, outages, on-call...). We'd also like to leave a room for power users which would like to use their own S3 back-end, so we're not locking anyone with our ecosystem. ~90-95% users would likely use our default offering, because it's convenient, but if someone would like to set up their own MinIO instance using their NAS or whatever, then go ahead. We'll come back to that, but first priority is to build a stable S3 protocol clients. I would love this to be a full-time gig, at the moment it is a full time volunteering and won't be source of income for long, but that's fine. I am extremely excited building technology which helps people to keep their data private without giving it up to seemingly "free" alternatives. It's quite challenging explaining people that "free" software that they use it's actually not free (they pay by their data being sold and mismanaged), these very same people will say: "I've nothing to hide", they're majority and not a target of our offering. There is still significant community of people who actually do mind protecting their privacy. I would rather stay in that circle. ... and I am glad to have a users like you who open up and give proper feedback. Thank you ! (edited)
🚀 1
Avatar
Avatar
myfrogger
Just realized there are concerns with sharing data because idiots will share illegal material. Maybe stay out of the business of hosting the data for now. I'm just thinking out loud but I'll leave the above writing there in case anyone finds it helpful.
As long as a business provides an easy way for DMCA to take down the copyrighted material I think it should be fine. (edited)
👍 1
Alisa joined the server. 2/16/2023 4:28 PM
Avatar
Hi @Alisa Welcome here 🙂
Avatar
Avatar
myfrogger
Hi @Alisa Welcome here 🙂
Hey:)
Avatar
Since 1.1.1 there is an improved upload performance (using multi-threaded workers), as well as additional functions in the Transfers section. Previously uploading ~100+ files could crash the app depending on the machine. Currently ~1000 files is a piece of cake. We've tested ~11k and except that scrolling the items isn't exactly a smooth experience, it's uploading just fine. It is possible to retry the failed requests with the retry arrow. There are also bulk operation buttons to stop all and delete all items within the tab you're in. (edited)
9:29 AM
9:29 AM
Waiting list
9:30 AM
Failed ones (auto-retry will get there in one of the next releases) (edited)
Avatar
Awesome!!! Great work 🙂
👍 1
Tillerson joined the server. 2/26/2023 10:40 AM
Avatar
Hi @Tillerson !
10:41 AM
Welcome to our modest community 🙂
Avatar
hello
Deleted User joined the server. 3/1/2023 11:59 PM
Deleted User joined the server. 3/2/2023 12:03 AM
Avatar
Avatar
Deleted User
Click to see original message
Deleted User 3/2/2023 12:03 AM
Avatar
Deleted User 3/2/2023 12:20 AM
Hello, I am trying the photosync for s3 and I'm getting invalid url on the generated access key
Avatar
Deleted User 3/2/2023 12:53 AM
made it work - I had to add https:// in front of it
12:53 AM
Trying S3Drive with the same but cannot connect
Avatar
Hi @Deleted User and welcome to our channel. We've recently added the photo & video backup module in our S3Drive app which is now recommended way of interacting with S3 endpoints. If you would like to keep using PhotoSync you're free to do so, however going forwards we're actively expanding S3Drive. Thanks
12:51 PM
Hi @Deleted User and welcome to our channel ! 👋 (edited)
Avatar
Deleted User 3/2/2023 10:19 PM
Hi @Tom thanks. I've already switched to s3Drive but that one cannot connect to my idrive s3 ... not sure what to do, I think I've tried everything ...
10:21 PM
I'm interested in the premium version with encryption. Quick question: can I also decrypt the files with some standard command or software?
Avatar
Avatar
Deleted User
I'm interested in the premium version with encryption. Quick question: can I also decrypt the files with some standard command or software?
Encryption is compatible with V2 client-side cipher that AWS introduced. I am not aware of CLI that supports it, but it is supported by AWS SDK's, please search table for "Content Encryption: AES/GCM" and "Key Wrap: AESWrap"(https://docs.aws.amazon.com/general/latest/gr/aws_sdk_cryptography.html#crypto_features) which is supported by Java, .NET, Ruby V2, and C++. We haven't published any script or CLI tool yet, but we will provide one as an open-source, so user can always download and decrypt data at their discretion. We haven't prioritized that yet, as due to some limitations of both AES-GCM and AWS approach, we're having problem with implementing encrypted video streams and considering switching to different cipher and/or encryption scheme: https://s3drive.canny.io/feature-requests/p/support-streaming-of-encrypted-videos (edited)
Avatar
Avatar
Deleted User
Hi @Tom thanks. I've already switched to s3Drive but that one cannot connect to my idrive s3 ... not sure what to do, I think I've tried everything ...
I've tried connecting to iDrive myself and it worked just fine. Do you get any specific error? What endpoint are you using (I got e.g.: https://f4n6.par.idrivee2-46.com) ? Can you please long tap on version number at the bottom of the login screen, so you go to the Logs and perhaps see some error or warning that might render something useful? (edited)
Avatar
Deleted User 3/3/2023 12:25 PM
Multiple issues: (a) can't verify the secret key (the other app has a view button) (b) can't place cursor to add or delete a letter (c) not clear if I need to add the https:// part to the endpoint string
12:25 PM
I get "Failed to authorize"
12:25 PM
same credentials worked on the other app
12:26 PM
endpoint looks similar to yours
Avatar
Avatar
Deleted User
same credentials worked on the other app
Regarding a) We will add such option, thanks for mentioning that. b) That's something we can't easily reproduce. What's your Android version and phone model? c) Both: "https://f4n6.par.idrivee2-46.com" and "f4n6.par.idrivee2-46.com" would behave the same and use secure HTTPS context. Quick question, does the API key that you generate have both: "Read and write" permissions? S3Drive verifies the credentials, by saving a small test file to the bucket.
Avatar
Deleted User 3/3/2023 2:32 PM
For b) I have an Samsung S22 Ultra.
2:33 PM
Android 13
2:33 PM
It is read/write ... Photo sync did save files with same account
2:34 PM
And it also works on syncaware ...
Avatar
Avatar
Deleted User
endpoint looks similar to yours
I've managed to reproduce the "Failed to authorize" issue once: "Default encryption" is enabled for an iDrive bucket. It seems that they've implemented encryption in a way which modifies the ETag / MD5 hashes on the S3 protocol side. S3Drive performs strict ETag validation which is required for not yet released syncing features. After we write an object to iDrive E2, we expect ETag to match content of file that we've sent to the server (file integrity), instead they seem to return ETag of their encrypted blob. We will think of workaround, e.g. by disabling syncing features (and notifying user) if we detect ETag issue instead of preventing user from logging in by displaying: "Failed to authorize", however before it's there the only way is to disable iDrive encryption. https://s3drive.canny.io/feature-requests/p/workaround-idrive-e2-auth-issue-if-default-encryption-is-used PhotoSync doesn't perform ETag validation, since it's only one-way upload. Technically speaking the iDrive approach isn't correct from an S3 protocol perspective and is unique amongst S3 providers, e.g. Backblaze's encryption works just fine. Thanks for reporting this issue ! (edited)
👍 1
Avatar
Deleted User 3/5/2023 1:11 PM
there's gonna be a better way to connect to the account with QR code or something like on the web instead of typing all that ... Maybe with a small tool to generate the QR code or something ...
Avatar
Avatar
Deleted User
there's gonna be a better way to connect to the account with QR code or something like on the web instead of typing all that ... Maybe with a small tool to generate the QR code or something ...
Thanks, I know it's indeed painful with 4 details to be provided. There will be improvement coming within a few weeks. We've already implemented QR code auth in our previous app (PhotoSync). The old web client does indeed support that: https://web.syncaware.com (edited)
Avatar
Avatar
Deleted User
there's gonna be a better way to connect to the account with QR code or something like on the web instead of typing all that ... Maybe with a small tool to generate the QR code or something ...
Basically the idea is that desktop or web will display QR codes, which then mobile app will scan and securely (encrypted !) provide the credentials over.
Avatar
Deleted User 3/5/2023 1:21 PM
I've signed in after disabling the idrive default encryption as you've indicated
👍 1
1:21 PM
not sure how to transfer a folder?
1:22 PM
is album thumbnail view enabled?
1:22 PM
or is it only for media backup?
Avatar
Avatar
Deleted User
not sure how to transfer a folder?
It's not yet possible, but eventually will be.
👍 1
Avatar
Avatar
Deleted User
or is it only for media backup?
Initially it's media backup only preview functionality.
Avatar
Deleted User 3/5/2023 1:23 PM
got it
Avatar
Avatar
Deleted User
or is it only for media backup?
By saying thumbnail view, you mean grid view?
Avatar
Deleted User 3/5/2023 1:23 PM
can i select which media?
1:23 PM
i.e. camera album ...
Avatar
Avatar
Tom
By saying thumbnail view, you mean grid view?
Deleted User 3/5/2023 1:24 PM
yes
Avatar
Avatar
Deleted User
can i select which media?
You can upload: "All" media from select album/albums or "Only new", there is a dialog prompt the moment when album is selected. In next turn we will implement manual mode where you can manually select media to backup (or conversely mark them "fresh" for reupload).
Avatar
Avatar
Deleted User
yes
Yes, we might actually kill the current view, since filenames + list isn't condensed enough for bigger albums. The issue is that on iOS there is no way to reliably get the filename for display anyway, so we'll likely default to thumbnail / grid view. (edited)
1:28 PM
There is an active development on all this, it's all coming. Bear with us please and stay tuned 🙂
Avatar
Avatar
Tom
Yes, we might actually kill the current view, since filenames + list isn't condensed enough for bigger albums. The issue is that on iOS there is no way to reliably get the filename for display anyway, so we'll likely default to thumbnail / grid view. (edited)
Deleted User 3/5/2023 1:28 PM
👍
1:29 PM
after upload preview is not working for me (I have added an encryption key) (edited)
Avatar
Avatar
Deleted User
after upload preview is not working for me (I have added an encryption key) (edited)
Is it a photo or video?
Avatar
Deleted User 3/5/2023 1:31 PM
photos
1:31 PM
actually I only have the thumbs folder
1:31 PM
I'll check on idrive
1:32 PM
yup ... only the thumbs folder exists (edited)
Avatar
What are your numbers here after backup has finished? Is media that was supposed to be backed up media appearing in: "Backed up" counter?
Avatar
Deleted User 3/5/2023 1:37 PM
it says 2 and 2
1:38 PM
2-2-0
1:38 PM
But I have another issue, in that folder/album I believe I should only have 2 photos
Avatar
Great, so when you navigate to Root folder, can you see: Automatic uploads folder?
Avatar
Deleted User 3/5/2023 1:38 PM
when I go in it I see pics I have just received in whatsapp
1:39 PM
Root folder of?
Avatar
Basically Top of the Files section.
Avatar
Deleted User 3/5/2023 1:39 PM
there is no no, I don't see that
1:39 PM
but auto upload is not enabled
1:39 PM
I used "Start backup"
1:40 PM
now in the album I see 2 files only
Avatar
That's fine, since your numbers claim that media was backed up, it should be there. I am wondering if it has something to do with iDrive read consistency.
1:41 PM
Album tab shall display all pictures. There are two annotations, green (backed up) and white cloud (not backed up yet), no annotation mean media is not selected for backup.
Avatar
Deleted User 3/5/2023 1:42 PM
some strange things are happening ... files appear in the root
1:42 PM
then disappear ...
1:43 PM
the 2 photos show as backed up
Avatar
... but have you seen Automatic upload folder at least once?
Avatar
Deleted User 3/5/2023 1:43 PM
no
1:43 PM
I can enable the auto backup to see
Avatar
It won't change anything really. It's just poor name of mine, the automatic upload folder is where the backed up media are supposed to appear regardless if it's user clicking "Start backup" or if auto mode is enabled (edited)
Avatar
Deleted User 3/5/2023 1:44 PM
says: app needs to be running, background service will be available soon
Avatar
What do you mean that files appear in the root? You mean like just at the first top level in the Files tab? (edited)
Avatar
Deleted User 3/5/2023 1:46 PM
yes
Avatar
Avatar
Deleted User
says: app needs to be running, background service will be available soon
That's right, background service is being worked on. PhotoSync had that, we can do the same here.
Avatar
Deleted User 3/5/2023 1:46 PM
in the listing ... a screen refresh or cache issue??
1:47 PM
now they are not there anymore
1:47 PM
I should take a screenshot if it happens again
Avatar
That looks to me like hitting different iDrive server every refresh which have different version of cache. We haven't used iDrive actively really.
1:47 PM
Never experienced that with Backblaze for instance.
Avatar
Deleted User 3/5/2023 1:47 PM
could be ...
Avatar
I am still worried that Automatic uploads folder isn't displayed, as this is where files are supposed to be uploaded. We will allow overriding that location by user in the Settings, but it's not there yet.
Avatar
Avatar
Deleted User
could be ...
I've found this topic on Reddit, which in general highlights multiple issues that iDrive E2 is having. https://www.reddit.com/r/selfhosted/comments/vyu2c7/comment/iw0pgdx/?utm_source=share&utm_medium=web2x&context=3 One of them 4 months ago: "It’s impossible to access newly created folders" I am not claiming that this is what's happening, as there well may be an issue on our side (we're still beta). Is it possible by any chance for you to switch to e.g. Backblaze or other provider at least temporarily? (edited)
Avatar
Deleted User 3/5/2023 1:56 PM
I suppose it is ... but I am in no hurry
Avatar
Deleted User 3/5/2023 2:08 PM
maybe I can try the MinIO instead ...
Avatar
Avatar
Deleted User
maybe I can try the MinIO instead ...
MinIO is great, however ironically with the default settings using their Docker image we've faced similar listing cache issues (LS command was returning stale objects which were not fetchable via GET method), we have it in our backlog and didn't have chance to investigate since priorities are elsewhere. I guess it must be related to default cache settings which are maybe performance focused. Let's be honest, S3 protocol implementations for a long time weren't suitable for thing like S3Drive, due to consistency issues which were addressed not that long time ago. For AWS it was December 2020: https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/
exploit joined the server. 3/7/2023 6:35 PM
Avatar
Avatar
exploit
Click to see original message
Avatar
Hello 🙂
Exported 100 message(s)
Timezone: UTC+0