S3Drive - general

Hello guys and welcome to our Discord channel. In case you have any questions, problems, ideas or would like to simply talk about the app, S3, syncing and all that stuff... we would love to hear from you.

Legar joined the server. 1/30/2023 4:41 PM

Legar

Click to see original message

Hi there, how do you feel being the first to join?

It was a good idea to create a discord channel. Interactivity was very limited. This will improve the feedback loop

4:42 PM

Hahah feels good

I need to get my community management skills up to speed, in the meantime let's give a Discord channel a try and see how it goes. I hope actually to get more feedback here. Well, thanks for joining !

1

Today we've released support for video thumbnails and enlarged thumbnail, so it uses up the full square space. It's already available on Android, on iOS is pending release and will be available later today / tomorrow. How do you like it? (edited)

barrrrtek joined the server. 1/30/2023 7:35 PM

H10ISBFYAKE91 joined the server. 2/5/2023 7:52 PM

Welcome to our channel @barrrrtek and @H10ISBFYAKE91 !

myfrogger joined the server. 2/13/2023 7:57 AM

Hi, looks like a cool project. I can't seem to run it on windows because I get the error that files are missing. Any thoughts? The code execution cannot proceed because MSVCP140.dll was not found. repeated error for VCRUNTIME140.dll & VCRUNTIME140_1.dll

8:00 AM

Also, what is the recommended installation directory? It seems to be installing into %APPDATA% but I tried it in Program Files also. Both gave me the above errors.

Hi, @myfrogger thanks for letting us know about the issue. It seems that Visual C++ Redistributable package may be missing in your system: https://www.microsoft.com/en-gb/download/details.aspx?id=48145 What Windows version are you running exactly? https://support.microsoft.com/en-us/windows/which-version-of-windows-operating-system-am-i-running-628bec99-476a-2c13-5296-9dd081cdd808 There is no preference as where you install the package. We're using default settings of: "Inno Setup", but in principle you can install it wherever you want. Finally, sorry for this issue, we're Linux users ourselves and we're working on improving tests and setup instructions on different desktop platforms, Windows included. (edited)

Visual C++ Redistributable for Visual Studio 2015

The Visual C++ Redistributable Packages install run-time components that are required to run C++ applications built using Visual Studio 2015.

Which version of Windows operating system am I running? - Microsoft...

Learn how to find which version of Windows operating system your PC is running and device specifications.

9:26 AM

@myfrogger Wait, did you actually use our "exe" installer or installed via Microsoft Store?

Tom

Hi, @myfrogger thanks for letting us know about the issue. It seems that Visual C++ Redistributable package may be missing in your system: https://www.microsoft.com/en-gb/download/details.aspx?id=48145 What Windows version are you running exactly? https://support.microsoft.com/en-us/windows/which-version-of-windows-operating-system-am-i-running-628bec99-476a-2c13-5296-9dd081cdd808 There is no preference as where you install the package. We're using default settings of: "Inno Setup", but in principle you can install it wherever you want. Finally, sorry for this issue, we're Linux users ourselves and we're working on improving tests and setup instructions on different desktop platforms, Windows included. (edited)

I used the exe installer as I hate the microsoft store. I'll try downloading Visual C++ now. I'm using Windows 10 Pro 22H2 (edited)

No error messages now but when it loads I only get a black window

10:19 AM

I've uninstalled, reinstalled, and restarted

myfrogger

No error messages now but when it loads I only get a black window

Right, ~~I've tried it on a fresh Win install and I also get a black screen. We're working on the fix and will let you know hopefully later today.~~ Actually we've only reproduced it in a virtual environment and can't reproduce this on a physical environment. Black screen so far appears to be an issue with the GPU / hardware acceleration / DirectX version (at least 11 is required). (edited)

myfrogger

No error messages now but when it loads I only get a black window

May I ask what's your graphic card model? Is it dedicated or perhaps integrated within your CPU / motherboard?

k4ufj2 joined the server. 2/13/2023 12:16 PM

It's integrated graphics

myfrogger

It's integrated graphics

Thanks that helps, do you perhaps have DirectX (at least 11) installed?

12:22 PM

It can be checked typing: dxdiag in the search bar.

Actually I'm also testing your software in a virtual environment using Oracle VirtualBox so maybe that's the issue

Right, so in fact that's an issue with VirtualBox, we're using latest 7.0.4 and it doesn't work. It works fine on VMWare Workstation though. (edited)

12:25 PM

App does require 3D acceleration due to technology that we're using (Flutter), perhaps it's possible to start it in a software acceleration mode only, but we haven't explored that yet. (edited)

Gotcha. I'll try to test this soon on a physical computer. It doesn't seem to have any bloatware installed so I should be safe. It's incredibly difficult to find good backup and sync software if you want end-to-end encryption and using a S3 based cloud

12:26 PM

This idea is very, very cool though it wasn't what I was initially searching for. Those are the best finds though

12:27 PM

People upload FAR to much data to hosted clouds. I uploaded business "trade secret" data for work to dropbox for years...what a disaster that could have been! I was so ignorant

Thanks for the good words. It keeps us motivated

We're pending major release with multiple new features being added. It will also include photo backup from Android/iOS to S3. Different sync modes including two-way sync will be available ~April 2023 - https://s3drive.canny.io/feature-requests/p/syncbackup-options-two-way-one-way

That's an amazing idea. Actually I uploaded all my photos to the cloud up until recently out of ignorance also! Then I switched to Mega.io but recently scrapped the whole thing and decided to sync only to my laptop, which is then backed up in a more secure way. Photo backup is a major upgrade and very useful!!

12:31 PM

I don't know why but I've seen on other apps it's hard to make automatic backup triggers on iOS. It seems the main way people do it is based on GPS movement. I didn't want to give apps my location. The other way I've seen recently is auto sync is triggered upon the device connected to a charger. That's better but I've seen only used on one app.

12:32 PM

Maybe you know other ways but to my non-technical research brain, auto syncing on iOS seemed like a real obstacle for developers.

It's quite tricky to do it reliably on iOS and so far the backup mode will work for S3Drive when you start the app (comparing to fully supported background sync on Android)... but then we will be using all available means to improve that. Thanks for the hints with triggers.

So if you run into that problem, maybe try the sync when connected to a charger

In fact it's quite tricky to do it reliably on Android as well due to: https://dontkillmyapp.com/

Don’t kill my app!

Hey Android vendors, don’t kill my app!

1

myfrogger

So if you run into that problem, maybe try the sync when connected to a charger

It's possible that it needs to create an iOS shortcut in the "automation" section. Another iOS approved trigger might be to connect to a certain wifi

Given that iOS ecosystem is a pretty new thing to me and our team these hints are extremely useful !

I thought it might be. It's surprisingly hard to find solutions as a user if I don't want to trust the company also with my data. It seems most people want an all-in-one solution so who knows

12:38 PM

I mean, who knows what most people want. I don't think I'm the usual customer and there's not enough people like me to make lots of money

I was personally struggling to find such solution. That's actually how this project started.

1

12:40 PM

I actually don't like all-in-one. I prefer to choose best tools for specific case.

12:40 PM

Most people do prefer all-in-one though as you say.

12:40 PM

Great, I'd love to hear if you managed to install the app and whether it works all fine for you.

I'll keep you updated in a day or two when I have more time

Thanks for your help!

Thanks for your feedback !

shaike joined the server. 2/13/2023 7:08 PM

Hello

7:09 PM

Does a3 photosync supports syncing video files or just photos?

Hi @shaike, it supports both video and photos, however by default only photo albums upload is enabled, video albums needs to be explicitly marked for auto upload.

Where can i gind itcon photosync s3? It shows me DCIM and WA business folders

7:37 PM

Sorry for typos

That's how it looks on my end. There is Movies folder.

7:40 PM

Then I need to unhide it.

I am talking about photosync not s3 drive

This is PhotoSync screenshot.

7:41 PM

Depending on the Android version these folders are named differently, but in principle rule is the same.

Right, is that everything that you see? No more albums?

This is what i see

Got it

So no videos

I am not sure why videos are not displayed. ~~Within a week or maximum two, there will be~~ Early March we shall deliver photo & video module (with a backup and background mode) to S3Drive with more flexible album management. This is where our focus is currently. BTW. What's your phone model / Android version? (edited)

Fold 3 with android 13 oneui 5 December 2022

7:48 PM

That's will be grear

7:48 PM

Great

Thanks for letting me know. I need to say that we haven't tested PhotoSync with Android 13 enough, instead we worked hard on S3Drive. If you don't mind waiting little bit I would recommend trying it once the backup module is released. Sorry for the inconvenience.

That's fine i really appreciate your work

7:52 PM

So i should use s3 drive meanwhile?

It can't automatically backup your media, so it's not a replacement for PhotoSync just yet... but in general if you want to interact with S3 then S3Drive is the way to go.

Ok will do acctually i just want to clear some space. But since i had too many files on found that s3drive couldnt handle it

7:55 PM

By the way does s3 drive will supported multiple buckets available?

Given that we released first version in December, there is plenty of work ahead of us and lots of improvements deployed weekly.

7:56 PM

Multiple buckets support will come in a next earliest release, which is scheduled mid this week. (edited)

Great thanks for the great work

Nice

Thanks for the good words. It's been crazy past few months, but comments like this make my day and give me motivation to push hard on this

(edited)

shaike

Ok will do acctually i just want to clear some space. But since i had too many files on found that s3drive couldnt handle it

@shaike You did say that S3Drive couldn't handle lots of files. There is a known issue if too many files are being downloaded / uploaded simultaneously (there is no hard limit which might make app crash), but as such the front-end should handle buckets with lots of files. What problems you were facing in more detail if I can ask? (edited)

I have tried to upload about 6000 files and it just crashed

shaike

I have tried to upload about 6000 files and it just crashed

I see, right, it won't work just yet, but we're working on this as one of the high priority items with ETA end of February / early March. Once it's there we will then be able to support folder upload which will come next.

Good to hear

8:05 PM

Good luck

1

Any idea why Backblaze is refusing my connection?

12:55 AM

As an aside, it would be very helpful if there was a button to click to "list buckets" for me to select. Is my bucket formatted properly? This doesn't seem to be the issue though because even if I enter the wrong credentials, I still can't seem to connect to the server.

OOPS. It appears I need to enter https:// But what's interesting is that the computer was actually sending network traffic. Maybe it's defaulting to HTTP? So backblaze was rejecting all HTTP traffic but accepted HTTPS? (edited)

2:02 AM

A significant downside is that the file and folder names are not encrypted

2:03 AM

This is especially important to me because I like to use very descriptive file names.

2:06 AM

Next, when I try to create a share link, I cannot uncheck this box nor edit the date/time for link expiry. Do I need to make my S3 bucket public by chance? (edited)

Even with the bucket set to public the link doesn't work for me (edited)

Hi @myfrogger, you're right if no scheme (https/http) was provided it defaulted to HTTP. We're going to make more explicit. Likely the endpoint without scheme will default to https, where as IP addr will default to http. Filename encryption is on our Roadmap and we have a working prototype already. https://s3drive.canny.io/feature-requests/p/filenamefilepath-encryption (ETA ~April 2023). We're making further research to understand standards or well established implementation in that area, so we can stay compatible. The sharing functionality is based on S3 presign URLs, their limitation is that the signature can't be valid longer than 7 days, so every 7 days new link would have to be generated. We're researching how to overcome this limitation. For instance we could combine this with a link shortener, so there is single link that doesn't change, but under the hood we would regenerate the destination link as needed. The encrypted share link has the master key at the end after the # character and looks like this: https://s3.us-west-004.backblazeb2.com/my-s3drive/.aashare/hsnwye5bno3p/index.html?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=004060aad6064900000000044%2F20230214%2Fus-west-004%2Fs3%2Faws4_request&X-Amz-Date=20230214T095014Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=abdcd875e2106ee54c6a1d1851617c7e694e121464c5ca9023526ce2836be595#GKSGYX4HGNAd4nTcXb/GIA== What it does it tries to load the encrypted asset as usual, but it's not aware per se if an asset is encrypted. In the background JavaScript tries to fetch the asset and replaces the one on the screen with decrypted version. It looks like it has failed on your side. Can you go to the console (right-click -> inspect element) to see if there is anything abnormal (that is error in the Console or different than 200 status code in any of the network requests).

The current sharing functionality is pretty basic version initially introduced in PhotoSync. We actually plan to integrate it with our web client, so when you share a link, it would open normally inside the app: https://web.s3drive.app It's not on our roadmap yet, we need to build some spec first. Realistically we won't be able to build it earlier than June/July. Speaking of your file / image sharing issue. Does it work unencrypted? What size is it? Bucket doesn't need to be public (it shouldn't be). The idea of presigned sharing is that you choose which resource to share from a private bucket. I would be glad to assist you on any other issues. As you see, it's all pretty early stage, but we're catching up fast. Thanks for your feedback and an idea with listing buckets button. Noted and added to our improvement items. (edited)

You guys are awesome!! Thanks for all your hard work. I do see that you catch up very quickly

1. I'm happy to hear that folder name and file name is on the near roadmap for completion. Usability problem solved

2. The S3 presign URL concept is interesting. Can a link be done for a time period shorter than a week? I assume the user would need to have their program opened and some kind of timer to check if any links need to be signed again. Then the shortened link would need to be updated. It's a good idea actually but would want to make sure it's done in a trustless or trust minimized way. Also be careful deploying resources that you have to pay for as then you'd have ongoing costs. It depends what your monetization plan is long term I guess. 3. If the link is presigned, am I right to assume that there is no ability to revoke the sharing? 4. I'm actually not familiar. What is PhotoSync? 5. The link appears to work fine in Chrome but in Firefox it stops loading the page faster. If I click refresh, the image appears as it should. Strange but seems like all is fine. The things I would typically be sharing are not photos so it's not really a priority issue for me. 6. I'd love it if S3 drive had a different favicon so I can more easily distinguish it. I'd love it more if it was mounted as a drive letter in my file explorer.

Actually, I think what I'd personally use most is if I could right click, go to "send to" and have S3 drive as a destination. I'd have to think about if I'd want it in the main folder or prompt me where to put it. I'd probably choose to have a popup to prompt me where I want it saved within S3 drive. (edited)

Can I ask a general S3 question? Are there any implications of backing up say, 10000 files at 250kb each or a few larger zipped files? Before I came across S3Drive, I was actually looking for more permanent backup solutions. There are two different approaches that I see and I don't know if one is better than the other. One is storing each file individually and the other is first putting them into "chunks" of X MB/GB first.

myfrogger

You guys are awesome!! Thanks for all your hard work. I do see that you catch up very quickly

1. I'm happy to hear that folder name and file name is on the near roadmap for completion. Usability problem solved

2. The S3 presign URL concept is interesting. Can a link be done for a time period shorter than a week? I assume the user would need to have their program opened and some kind of timer to check if any links need to be signed again. Then the shortened link would need to be updated. It's a good idea actually but would want to make sure it's done in a trustless or trust minimized way. Also be careful deploying resources that you have to pay for as then you'd have ongoing costs. It depends what your monetization plan is long term I guess. 3. If the link is presigned, am I right to assume that there is no ability to revoke the sharing? 4. I'm actually not familiar. What is PhotoSync? 5. The link appears to work fine in Chrome but in Firefox it stops loading the page faster. If I click refresh, the image appears as it should. Strange but seems like all is fine. The things I would typically be sharing are not photos so it's not really a priority issue for me. 6. I'd love it if S3 drive had a different favicon so I can more easily distinguish it. I'd love it more if it was mounted as a drive letter in my file explorer.

Actually, I think what I'd personally use most is if I could right click, go to "send to" and have S3 drive as a destination. I'd have to think about if I'd want it in the main folder or prompt me where to put it. I'd probably choose to have a popup to prompt me where I want it saved within S3 drive. (edited)

2. Period can be shorter than 7 days. We just didn't implement it yet. 3. That's right, as usual there are ways around, but not all of them feasible for S3Drive. It's possible to revoke credentials and generate a new ones, that would invalidate the link, but that's not practical if user have to do it themselves. The other option is to change the resource / filename, that's also quite impractical. The most feasible option that we've researched so far is connected to E2E encryption. Basically revoking share would mean regenerating sharing key for that resource. That means that presigned link would still be valid for some time, but the adversary would only be able to get the encrypted blob. We're still researching this. 4. This project started in Summer 2022 just as an Android app to backup Photos to S3 - https://play.google.com/store/apps/details?id=com.photosync.s3 (actually month earlier it was just Backblaze - https://play.google.com/store/apps/details?id=com.syncsolutions.backblazesync). Lots of research was already done in order to implement them. S3Drive extends that on all file types and all platforms. 5. Yeah it's rough around the edges, but in principle works for the most persistent users. We'll improve that. 6. We didn't even go as far, but the right click context menu (and tray icon with options like auto-startup etc.) is definitely something we would be adding. Thanks for mentioning that. I will make sure it's added to our list. Speaking of favicon, logo and UI in general. That's something that received least amount attention, since we were focused on building all the blocks to validate the concept. Now we keep adding new features and deploying bugfixes / improvements. There will be a moment within a few months at least where we step back and focus on UI simplification and beautifying. Do you have any ideas what favicon would work better? Thanks ! (edited)

myfrogger

Can I ask a general S3 question? Are there any implications of backing up say, 10000 files at 250kb each or a few larger zipped files? Before I came across S3Drive, I was actually looking for more permanent backup solutions. There are two different approaches that I see and I don't know if one is better than the other. One is storing each file individually and the other is first putting them into "chunks" of X MB/GB first.

It's finding a sweet spot between usability, costs and performance. For instance minimum billable object size for AWS S3 is 128KB, that means that if you store smaller objects you're still paying for the full 128KB. Smaller files means also more requests which will likely degrade performance and increase API costs (e.g. https://www.backblaze.com/b2/b2-transactions-price.html). S3 protocol doesn't support bulk upload (it does support bulk delete though), so 10000 files means at least 10000 PUT requests. On the other hand if you compress files and store them as chunks of X MB/GB, then even if small 250kb file is being changed, you need to replace the whole chunk which also comes at some price. If it's a cold backup that you synchronize from time to time, then having some sort of block storage (https://github.com/archiecobbs/s3backer) / chunks might be preferable way. If you want to conveniently access your data and even have a working copy that you work with, then a standard file approach is preferable even if it comes at some cost. S3 has its limitation, but it's powerful protocol, soon we will be supporting Versioning (https://s3drive.canny.io/feature-requests/p/version-preview-restore-integration-with-trash). Perhaps over the time S3 protocol will further expand allowing new operations (including bulk ones). If not, we have some ideas how to overcome some of the S3 protocol limitations with a "convenience proxy" that we're also researching, but not everything at once

(edited)

1

myfrogger

You guys are awesome!! Thanks for all your hard work. I do see that you catch up very quickly

1. I'm happy to hear that folder name and file name is on the near roadmap for completion. Usability problem solved

2. The S3 presign URL concept is interesting. Can a link be done for a time period shorter than a week? I assume the user would need to have their program opened and some kind of timer to check if any links need to be signed again. Then the shortened link would need to be updated. It's a good idea actually but would want to make sure it's done in a trustless or trust minimized way. Also be careful deploying resources that you have to pay for as then you'd have ongoing costs. It depends what your monetization plan is long term I guess. 3. If the link is presigned, am I right to assume that there is no ability to revoke the sharing? 4. I'm actually not familiar. What is PhotoSync? 5. The link appears to work fine in Chrome but in Firefox it stops loading the page faster. If I click refresh, the image appears as it should. Strange but seems like all is fine. The things I would typically be sharing are not photos so it's not really a priority issue for me. 6. I'd love it if S3 drive had a different favicon so I can more easily distinguish it. I'd love it more if it was mounted as a drive letter in my file explorer.

Actually, I think what I'd personally use most is if I could right click, go to "send to" and have S3 drive as a destination. I'd have to think about if I'd want it in the main folder or prompt me where to put it. I'd probably choose to have a popup to prompt me where I want it saved within S3 drive. (edited)

The drive letter / mount is something we've already tried and had a working read-only prototype on Linux. https://s3drive.canny.io/feature-requests/p/file-mount-on-desktop-clients-with-write-support Mac and Linux will likely be supported end of Summer using FUSE driver that we're already tried. With Windows we haven't explored it yet, but there are some ways to do it (https://github.com/winfsp/winfsp). We'll definitely implement it, it's just at this stage it's hard to give us ETA. We might adjust priorities later on depending on the users percentage per platform, but we don't have a clear view / analytics to make that decision just yet. What platform are you using by the way?

Thank you for the detailed responses. I'm using I use Windows and iPhone. My initial thought with the favicon is the current folder with the letters S3 on it. I'd have to do a bit of thinking and asking around at identifying the target market for this product. I'm not even sure S3Drive is the best name for marketing purposes. I don't dislike it though; just rambling. As far as those coming from Dropbox, Onedrive, etc, etc.... I think that the E2EE/control or own your data is a significant selling point and also the "pay for what you need". I don't know why all these companies have different "packages" when it seems easier to onboard people to a pay-as-you-go model. Apple seems to be the only one that figured it out that millions of people don't care about a $0.99 fee but they do care about $5-10. I think that you could monetize by having your own built in cloud (simply putting everyone's data into your own BackBlaze or Wasabi account). At least then when someone downloads the app, there is a usable setting without any further configuration. Maybe offer X 50gb free storage like everyone else. I remember reading years ago that getting someone to use the app the moment they download it is critical to keeping them as a user. I think you should embrace accepting bitcoin, lightning, and maybe some other crypto payments. Also embrace the community that wants their data private. Don't ask for names, don't log their details, etc. For cards use stripe. Maybe this not a project that you'll monetize but I feel like this could be a full time job, and an opportunity to make a great product that people would pay for. Anyway---I agree with you that you can't do everything at once, and the priorities appear to be ordered in a logical way. Keep up the good work

I'm happy to be an early user and maybe able to contribute somehow along the way. (edited)