S3Drive - general

hmm

10:16 AM

it is set up as us-east-1 in toml

10:16 AM

sending you my test creds

zer0

it is set up as us-east-1 in toml

Thanks, got this. In fact there is an issue on our side. The detection logic (pretty stupid regex) incorrectly detects the region from your endpoint. When I've tested Garage locally using just IP addresses it wasn't an issue. I've sent you possible temporary resolution via PM. (edited)

zer0

I see pro features mentioned. Do you see a bitwarden-like business model?

I think I haven't responded to that properly. We haven't decided exactly, our closest business plans include managed accounts with storage plans instead of selling "Pro" features. We've still left a room for ourselves if we ever decided to sell "Pro" features... and since we wouldn't like to disrupt current early adopters there is the Enrol API mentioned in our Privacy section that I believe you refer to. Realistically speaking, we'll keep certain Pro features always free (E2E encryption, photo backup etc.) to lower the barrier of entry and likely provide some additional Pro extensions and possibly per seat licensing model for companies in the future. We'll certainly stay cautious with changes and we extremely value existing users who contribute by providing valuable feedback. (edited)

1

slipstream joined the server. 3/24/2023 10:30 PM

@Tom working nicely with garage now

10:34 PM

zer0

@Tom working nicely with garage now

Thanks for letting me know. Do you mind if I ask how are you finding Garage and did you try any S3 compatible alternatives that is MinIO / Seaweed FS / Ceph or anything like that?

I did use minio before using garage

4:53 PM

I think I'll be moving back to minio

4:55 PM

Garage store a lot of chunk files which are hard to deal with on ext4

4:55 PM

It takes ages to run ncdu on the data folder

zer0

Garage store a lot of chunk files which are hard to deal with on ext4

Ah yes, I've seen in their documentation that they actually recommend XFS instead due to possible inode limit issues with EXT4.

Yea

4:56 PM

I like the CLI tho

... and it's a single Go binary with less baggage that MinIO already have.

4:57 PM

I would be curious to try SeaweedFS though. (edited)

I was looking into SeaweedFS today

4:58 PM

Looks very interesting

4:58 PM

I need to read the docs a bit more

KiraUsagi joined the server. 3/26/2023 3:24 AM

Hi @KiraUsagi, welcome to our channel !

Deleted User joined the server. 3/28/2023 5:39 AM

Hi @Deleted User, welcome to our small world !

dsdsdsdsdsdsdsds joined the server. 3/30/2023 1:26 AM

I'd go with whatever option had the easiest backups tbh. I find that setting up an environment is not that hard, having redundancy too, but having good backups isin't that simple.

AraAra ( •̀ .̫ •́ )✧ joined the server. 4/7/2023 6:43 PM

Sup people

Hi @AraAra ( •̀ .̫ •́ )✧, welcome to the channel, how are things?

Tom

Hi @AraAra ( •̀ .̫ •́ )✧, welcome to the channel, how are things?

Good good. Just recently finished college. Now looking to get into a university in Canada

AraAra ( •̀ .̫ •́ )✧

Good good. Just recently finished college. Now looking to get into a university in Canada

Exciting times. What degree you aiming for? Let me guess... CS?

(edited)

Tom

Exciting times. What degree you aiming for? Let me guess... CS?

(edited)

Aahh not really

6:46 PM

I was considering it

6:46 PM

But given my ADHD. I decided to go towards English literature/writing/media etc.

6:46 PM

As I like to be creative

6:47 PM

I am also however, a tech nerd

Right, so my guess wasn't exactly right. It's just so many people I've talked to are either CS students or converting to IT.

I really did want to

6:48 PM

But I also didn't want to make myself suffer mentally (edited)

6:48 PM

As I've suffered enough

Yeah sure. We forget in our society that in the long run mental health wins over any aspect. I also know some people who tried hard to convert to IT for the luxurious pay being the reason one, but ultimately decided it's not for them. (edited)

eg. Due to family pressure, I went with the science department starting from 9th grade to college

6:51 PM

And oh boy, worst mistake of my life

6:51 PM

I was in a mentally dark place. Almost decided to "game over" myself

6:52 PM

I've recovered in the past few months. Life's much better now

Yep I know that, it's like if someone else decides what you like doing... and noone asks you. Happened a lot around my circles. Sorry to hear that, but I am glad that you've took some steps and recovered. Good luck with your submission in Canada, stay strong ! (edited)

Thanks

6:53 PM

Feel like this server isn't promoted enough

6:53 PM

Really good app btw

Thanks, it's all pretty new and still expanding. We're not doing much of a marketing, little bit overwhelmed by outstanding tech items right now. (edited)

That's alright

Is there anything in particular I can assist you with?

I've personally done some android development myself

6:56 PM

It can be quite interesting

Indeed, we've started with Android and decided to expand on all platforms.

Tom

Is there anything in particular I can assist you with?

Oh yeah forgot, I can't seem to be able to delete folders

6:58 PM

It loads for a few moments but the folder persists

AraAra ( •̀ .̫ •́ )✧

Oh yeah forgot, I can't seem to be able to delete folders

I see, what cloud are you using? We've seen such behaviour with MinIO and iDrive.

Tom

I see, what cloud are you using? We've seen such behaviour with MinIO and iDrive.

Storj

AraAra ( •̀ .̫ •́ )✧

Oh yeah forgot, I can't seem to be able to delete folders

Also, there are lots of improvements that we will be releasing regarding folder operations: https://discord.com/channels/1069654792902815845/1069654792902815848/1085780599962349648 "Deleting item is a different beast. Currently we're not deleting items, but rather "copy" them to trash one by one, and then delete files in the original location. It's not that delete is slow, but the "copy" operation takes time and it's safe to assume that "thousand" files operation will be roughly 1000x slower then a single file. What can help here is to switch to S3 provider/region that has closest proximity to your internet connection, but no matter what results won't be satisfying enough. No worries though, we plan to address this in ~April 2023 by adding support for native S3 versioning: https://s3drive.canny.io/feature-requests/p/version-preview-restore-integration-with-trash so we can actually skip Copy operation... and Delete itself is fast enough. " (edited)

AraAra ( •̀ .̫ •́ )✧

Storj

Thanks for your report. At this stage all I can say is sorry that it doesn't work as expected, we will have a closer look regarding Folder delete and Storj. It's usually some tricky caching behaviour that we've hasn't managed consistently reproduce... but I am sure that we'll eventually get to this. Ultimately we can come up with some compromise, e.g. hide folder for some time after delete. (edited)

AraAra ( •̀ .̫ •́ )✧

Storj

One question that I have, does it stay like this forever or eventually the folder entry disappears?

Tom

Thanks for your report. At this stage all I can say is sorry that it doesn't work as expected, we will have a closer look regarding Folder delete and Storj. It's usually some tricky caching behaviour that we've hasn't managed consistently reproduce... but I am sure that we'll eventually get to this. Ultimately we can come up with some compromise, e.g. hide folder for some time after delete. (edited)

Ah no need to be sorry

1

Tom

One question that I have, does it stay like this forever or eventually the folder entry disappears?

Seems to stay like that

AraAra ( •̀ .̫ •́ )✧

Ah no need to be sorry

@AraAra ( •̀ .̫ •́ )✧ I will let you know once I find out something with Storj, in the meantime was wondering how are you finding other aspects of S3Drive? Is there anything that you don't like / like in particular? It's an early stage and every single feedback is extremely valuable for us. (edited)

Tom

@AraAra ( •̀ .̫ •́ )✧ I will let you know once I find out something with Storj, in the meantime was wondering how are you finding other aspects of S3Drive? Is there anything that you don't like / like in particular? It's an early stage and every single feedback is extremely valuable for us. (edited)

Ability to turn off Trash/Recycle bin would be pretty nice

1

7:52 PM

As for the Ui, I would like to see Monet theming.

prematrix joined the server. 4/8/2023 12:37 AM

Hello

12:37 AM

12:38 AM

Why can't I connect to my iDrive E2?

Hi @prematrix, this happens when iDrive E2 encryption is enabled. In general they use non-standard approach to reporting object hashes that we rely on. There are some ways around, but we haven't got to that yet. More on that here: https://discord.com/channels/1069654792902815845/1069654792902815848/1081247124580204574 and: https://s3drive.canny.io/feature-requests/p/workaround-idrive-e2-auth-issue-if-default-encryption-is-used The recommended path for iDrive E2 is to disable their encryption and enable in-app E2E encryption. (edited)

Thank you for the response. I will use cryptomator, until the issue is fixed.

prematrix

Thank you for the response. I will use cryptomator, until the issue is fixed.

Thanks for letting us know. We will definitely have it sorted pretty soon. Stay tuned !

Xenthys® joined the server. 4/10/2023 10:14 PM

Hello there! ablobwave

The Discord server is hard to join from the app since it opens a webview, I'd recommend opening the invite externally so the Discord application can pick it up, or at least the standard web browser. Trying out the application with MinIO, so far so good, pleased by the rename function that likely performs a copy + delete under the hood. It does what one would expect from an S3 client, no ads either, really liking it. (edited)

Not sure what the .aa* file and folder are about, but some "don't touch my bucket" parameter would be nice if the app doesn't strictly need them, otherwise that sounds like an additional bucket policy :D EDIT: looks like the file is for some kind of init feature within the app, and one of the two folders is the trash. I've seen the versioning feature request, but the trash folder could be opt-in if possible. (edited)

Hi @Xenthys®, welcome to our channel. Thanks for your post. Being relatively new Discord user myself such feedback is extremely useful, thanks ! You're right, "rename" performs a "copy" and then "delete" (but only if "copy" was successful operation.). The .aainit file is our write test, as well as ETag response validation (which is required for not yet released syncing features), as some providers (talking mostly about iDrive E2 with SSE enabled) don't generate valid ETags. BTW. Would you like S3Drive to support read-only mode? Regardless, we will try to improve clarity of this operation, so user feels more confident that we're not doing some shady write/reads. Speaking of Trash itself, likely this week starting on Android first there will be a Settings option to disable Trash feature altogether (which is a soft-delete emulation, but slow and pointless if bucket already supports versioning). Versioning UI with restore options will come little bit later. (edited)

Tom

Hi @Xenthys®, welcome to our channel. Thanks for your post. Being relatively new Discord user myself such feedback is extremely useful, thanks ! You're right, "rename" performs a "copy" and then "delete" (but only if "copy" was successful operation.). The .aainit file is our write test, as well as ETag response validation (which is required for not yet released syncing features), as some providers (talking mostly about iDrive E2 with SSE enabled) don't generate valid ETags. BTW. Would you like S3Drive to support read-only mode? Regardless, we will try to improve clarity of this operation, so user feels more confident that we're not doing some shady write/reads. Speaking of Trash itself, likely this week starting on Android first there will be a Settings option to disable Trash feature altogether (which is a soft-delete emulation, but slow and pointless if bucket already supports versioning). Versioning UI with restore options will come little bit later. (edited)

Thank you for your reply! Let me know if you have any question about Discord, I love to help around. For the .aainit file it's fine, but I'd prefer if the app saved the test results locally then deleted the file. I want to be able to write files so I wouldn't use a read-only mode, and we can always create read-only access keys if we want to be sure that's how the app will behave! I'm very interested by the share link expiry slider or date picker though, I never share for 7 days, it's either a smaller duration or permanent. Cool, I don't mind not having the versioning UI yet, but had to delete my file versions + the trash versions to cleanup my bucket so… yeah, trash is cool but I assume most people who want that have versioning enabled. I assume you already have quite a few buckets on various providers to test your features, but I can provide a MinIO one if it could be of interest. There was a 2nd folder with an HTML page in it, not sure what it was about but same thing I'd say, that's probably the least expected action from an S3 browser… While I audited the actions and indeed didn't find anything malicious, that could get me assassinated by my colleagues if I ever connected a more important bucket to the app. doggokek

Xenthys®

Thank you for your reply! Let me know if you have any question about Discord, I love to help around. For the .aainit file it's fine, but I'd prefer if the app saved the test results locally then deleted the file. I want to be able to write files so I wouldn't use a read-only mode, and we can always create read-only access keys if we want to be sure that's how the app will behave! I'm very interested by the share link expiry slider or date picker though, I never share for 7 days, it's either a smaller duration or permanent. Cool, I don't mind not having the versioning UI yet, but had to delete my file versions + the trash versions to cleanup my bucket so… yeah, trash is cool but I assume most people who want that have versioning enabled. I assume you already have quite a few buckets on various providers to test your features, but I can provide a MinIO one if it could be of interest. There was a 2nd folder with an HTML page in it, not sure what it was about but same thing I'd say, that's probably the least expected action from an S3 browser… While I audited the actions and indeed didn't find anything malicious, that could get me assassinated by my colleagues if I ever connected a more important bucket to the app. doggokek

Thanks for your help. Currently sharing uses S3 presigned URL, they're part of the protocol, however the signature validity is limited to 7 days max. The HTML is in fact created when sharing files externally to create the "album" feel. The workflow is that we create "presigned" URL for every resource that you share, then we embed them in the HTML which is itself "presigned". There will be in-app sharing later this year, which will basically open shared resources within the Web app: https://s3drive.canny.io/feature-requests/p/in-app-sharing It will still have to follow the S3 protocol limitations though. We're exploring different ways, so we can generate permanent links and allow users revoking them. There are multiple ways, one way is to create a middle-man (link shortener/alias and presigned links generator) to constantly extend the validity and keep the same URL scheme. The other way is to aim more into closer integration, e.g. with permission management and STS using MinIO we could revoke shared resources, something which isn't possible with pure S3 protocol. We haven't exactly agreed on the best way forwards. Thanks for all your feedback and I am more than happy to hear how are you finding different parts of the app and whether it suits your workflow. Speaking of MinIO, last time when we've tried to test the S3Drive with MinIO we've came across same listing caching issues when folder was deleted with all of its contents. The contents was removed properly, but the folder entry stayed in the s3 ls even though headObject couldn't retrieve it as a valid S3 entry. I am curious if you came across of something similar. (edited)

Yes, 7 days is the maximum but not the minimum, when I share a file however I'd like to keep using raw presigned URLs without having app-specific features included without me being aware. I'm looking for an S3 browser to use S3 features, if I wanted a fully-featured platform I'd install Nextcloud instead and use S3 as the backend storage, be careful about feature creep! I expect the majority of users who are technical enough to have an S3 bucket to use the app just for a quick file access or upload as well. I haven't encountered any issue with MinIO yet, but I deleted content through the console with their versions when I figured out the trash feature. I know MinIO doesn't immediately update the file list after a create or delete operation, even the console shows an invalid object count if you check too soon after, so maybe that's related?

While my messages are very opinionated, in the end it remains your application of course, I would never order an independent developer to do what I want, unless we have a paid contract haha

Xenthys®

Yes, 7 days is the maximum but not the minimum, when I share a file however I'd like to keep using raw presigned URLs without having app-specific features included without me being aware. I'm looking for an S3 browser to use S3 features, if I wanted a fully-featured platform I'd install Nextcloud instead and use S3 as the backend storage, be careful about feature creep! I expect the majority of users who are technical enough to have an S3 bucket to use the app just for a quick file access or upload as well. I haven't encountered any issue with MinIO yet, but I deleted content through the console with their versions when I figured out the trash feature. I know MinIO doesn't immediately update the file list after a create or delete operation, even the console shows an invalid object count if you check too soon after, so maybe that's related?

We will implement slider to allow providing the minimums. Our aim is to actually hide the S3 complexities and provide the "drive" experience for the general public. Our userbase is mixed. I take your point and will make a note to incorporate raw sharing. Speaking of Nextcloud, yes it's an option, but not anyone wants to set up their server, manage it, take care of updates and security... also if you want Nextcloud to not cost a lost (when e.g. hosted on some VPS)... then actually you need to connect it with some S3 provider to get reasonably priced storage. Our aim is to skip layers like Nextcloud and simply using some client-side magic provide similar experience (and less convoluted than Nextcloud - but that's my opinion) using raw S3. Speaking of caching issues. It's likely what we've experienced with MinIO and was little bit confused, because all other providers (except iDrive) update their listings immediately.

Xenthys®

While my messages are very opinionated, in the end it remains your application of course, I would never order an independent developer to do what I want, unless we have a paid contract haha

If we only got more such "opinionated" messages, it would've been much easier to build stuff... and working full-time on this for about a year now without a single paycheck, I wouldn't mind a paid contract

... but seriously, it's actually one of the coolest things I've been working on in my 10+ years career and seeing already such positive reception we've no intentions to stop. (edited)

I see, however you're going to have a hard time implementing a nice drive experience on top of bare S3 buckets, especially if you reduce yourself to the common denominator between the various providers. I guess some kind of "expert mode" would help settle the debate, where you expose raw S3 functions and let the user do whatever without interfering. Yes, Nextcloud was an example, I don't know many people with an S3 bucket who don't already have a way to share files, and most will probably just use OneDrive or similar if they want simplicity. With the single-file installer and compatibility with any shared hosting service, it's not the hardest to use

10:51 AM

I'd love to be rich enough to offer paid contracts by myself, the company I work at would unfortunately not be interested, our buckets are going to be heavily locked down so even accessing them directly from a company laptop on the internal network would be frowned upon. Welcome to healthcare!

Xenthys®

I'd love to be rich enough to offer paid contracts by myself, the company I work at would unfortunately not be interested, our buckets are going to be heavily locked down so even accessing them directly from a company laptop on the internal network would be frowned upon. Welcome to healthcare!

We're also planning some enterprise offerings. Basically S3Drive, locked down within the internal infrastructure with SSO, permission management, audit log and some management tools on top of MinIO (or other solution). Obviously the model would've to be slightly different. We would likely provide management tools as an open-source on some license. I am just thinking loud and this is something we will start work on 2023 Q4 / 2024 Q1 at earliest. Out of curiosity what your company uses for the S3? Is it MinIO, Ceph or some cloud S3? (edited)

We're moving from Azure Blob storage to AWS S3 directly, so pretty much going from the least to the most compliant S3 implementation :p (edited)

11:00 AM

As for our SSO, it's based on Microsoft so probably not what you would expect for a company fully using AWS as its sole cloud provider.

11:01 AM

I have to say, my main use of S3Drive is to have an S3 app on my phone, the others weren't as appealing, but your vision fully makes sense once we take all platforms into account.

11:08 AM

Regarding licensing, maybe you'd like the Sustainable Use created by n8n: https://github.com/n8n-io/n8n/blob/master/LICENSE.md It's based on the Fair-code software model (https://faircode.io/) which only restricts commercial applications of your software, it's interesting.

1

Xenthys®

I see, however you're going to have a hard time implementing a nice drive experience on top of bare S3 buckets, especially if you reduce yourself to the common denominator between the various providers. I guess some kind of "expert mode" would help settle the debate, where you expose raw S3 functions and let the user do whatever without interfering. Yes, Nextcloud was an example, I don't know many people with an S3 bucket who don't already have a way to share files, and most will probably just use OneDrive or similar if they want simplicity. With the single-file installer and compatibility with any shared hosting service, it's not the hardest to use

It's indeed a challenge to provide a nice drive experience on top of bare S3. There are multiple missing features, block read/writes, bulk operations, atomic operations. We're not discouraged though, as there are ways around. The beauty of such approach is that we're staying compliant with S3 at all times, so there is no vendor lock-in or proprietary back-end. With some optional tooling/layers we can improve the experience. The other advantages of S3Drive comparing to well-established players is price competitiveness (S3 is just cheap)... and ZK client-side encryption, so you can drop your files anywhere and secure them, so they are private. To be honest... every week/month as we build it we find new ideas that slightly affect our direction and it's just how things are developed.

Hey, I play Star Citizen, unless your app stays in alpha for a decade you're good doggokek

Xenthys®

Regarding licensing, maybe you'd like the Sustainable Use created by n8n: https://github.com/n8n-io/n8n/blob/master/LICENSE.md It's based on the Fair-code software model (https://faircode.io/) which only restricts commercial applications of your software, it's interesting.

Cool, thanks. It's really helpful. We've been studying different open-source licensing and having chance to read lots of horror stories (ElasticSearch, core-js recently ... and couple more) we were not exactly convinced what's the best path to take given that the S3 protocol itself is open and the fact that we don't have the proprietary back-end as a means to protect the IP by making the open-source client useless without back-end (not that we would want to take that route). (edited)

Well, the horror story for most users is seeing a project moving to another license, no matter from which to which. The model used by n8n has been criticized by pro-FOSS folks, but GitHub stars don't pay rent.

11:15 AM

I find it very interesting because self-hosted people won't have any barrier, even within companies, but they can't offer a SaaS version of your product. That seems to match your vision.

11:17 AM

Your "back-end" would be the layer you plan to use in order to transform an S3 bucket into a drive I assume, but licensing should only be validated with a lawyer anyway.

Xenthys®

Your "back-end" would be the layer you plan to use in order to transform an S3 bucket into a drive I assume, but licensing should only be validated with a lawyer anyway.

It's all interesting, thank you. I can imagine FOSS communities not approving such licences, but pragmatic open-source is just unsustainable and we're probably yet to see even more similar licences and more projects using these. (edited)

Xenthys®

I find it very interesting because self-hosted people won't have any barrier, even within companies, but they can't offer a SaaS version of your product. That seems to match your vision.

We'll get back to the licensing later this year. Next couple months are about delivering the major roadmap items. By that time we might have a clearer picture and more established userbase.