S3Drive - File Sync Filtering

Hi all, I'm trying to sync all .opus files but not .opus.crtemp files My current include/exclude filter is

+ *.opus
- *.crtemp
- *

however it still syncs files ending .opus.crtemp Could I please get some advice? Thank you. (edited)

Also tried these rules, I'm wondering if the file watcher is ignoring the rules? (edited)

10:58 AM

added bug report tag because I believe I've followed the documentation to the letter at this point, so this is unintended behaviour

10:59 AM

Sync logs

s3drive_log_20250324_105919301.csv

2.54 KB

11:04 AM

it's not the file watcher. I just had my callrecorder.key file autosync

Exitium

Also tried these rules, I'm wondering if the file watcher is ignoring the rules? (edited)

with these rules in place

Also tested with following rules:

- *
+ *.{{opus$}}

+ *.{{opus$}}
- *

Exitium

Hi all, I'm trying to sync all .opus files but not .opus.crtemp files My current include/exclude filter is

+ *.opus
- *.crtemp
- *

however it still syncs files ending .opus.crtemp Could I please get some advice? Thank you. (edited)

Hi, thanks for supporting S3Drive. I will play out with these rules once I am back in the office, however my first thought is that it should possibly be the other way round.

- *.crtemp
+ *.opus
- *

We will also review actually if file watcher follows these rules as well. I will try to get back to you later today, thanks! (edited)

Tom

Hi, thanks for supporting S3Drive. I will play out with these rules once I am back in the office, however my first thought is that it should possibly be the other way round.

- *.crtemp
+ *.opus
- *

We will also review actually if file watcher follows these rules as well. I will try to get back to you later today, thanks! (edited)

forgot to reply ^

1:15 PM

disregard previous message if seen I tried your rules

- *.crtemp
+ *.opus
- *

and they still continued to sync .crtemp files I look forward to your reply, thanks for your help! (edited)

I've did a brief check. When syncing files from folder containing:

file.opus
file.opus.crtemp
some.txt

and when setting the rules:

- *.crtemp
+ *.opus
- *

with Sync mode. On the destination mode only: file.opus was present, so it seems this worked. ... also initially we've tried without rules, so all were synced. After setting the rules the file.opus.crtemp and some.txt were removed, which is expected behavior of Sync mode. By any chance, are you using: Sync or Copy mode? We've tested that on Linux and S3 back-end though. We're going to repeat these tests on Android with Proton back-end. (edited)

Tom

I've did a brief check. When syncing files from folder containing:

file.opus
file.opus.crtemp
some.txt

and when setting the rules:

- *.crtemp
+ *.opus
- *

with Sync mode. On the destination mode only: file.opus was present, so it seems this worked. ... also initially we've tried without rules, so all were synced. After setting the rules the file.opus.crtemp and some.txt were removed, which is expected behavior of Sync mode. By any chance, are you using: Sync or Copy mode? We've tested that on Linux and S3 back-end though. We're going to repeat these tests on Android with Proton back-end. (edited)

I'm using Move mode

7:57 PM

Moving Local -> Proton

I see, let me include that in further tests.

Thank you

I am not exactly sure why this doesn't seem to be working on your end. 1. We've set up folder with this content. 2. We've configured Sync (Move) with rules:

- *.crtemp
+ *.opus
- *

3. We visit dst folder after sync, there are two files: .blank and file.opus. 4. We visit src folder, it seems only file.opus was removed.

8:26 PM

By any chance are you running latest S3Drive version? What's your Android version?

Tom

I am not exactly sure why this doesn't seem to be working on your end. 1. We've set up folder with this content. 2. We've configured Sync (Move) with rules:

- *.crtemp
+ *.opus
- *

3. We visit dst folder after sync, there are two files: .blank and file.opus. 4. We visit src folder, it seems only file.opus was removed.

I'm running stock pixel firmware patched with kernel SU and AVBRoot, bootloader locked with custom key Passing device integrity Will follow up with kernel SU modules shortly (edited)

8:47 PM

Tom

I am not exactly sure why this doesn't seem to be working on your end. 1. We've set up folder with this content. 2. We've configured Sync (Move) with rules:

- *.crtemp
+ *.opus
- *

3. We visit dst folder after sync, there are two files: .blank and file.opus. 4. We visit src folder, it seems only file.opus was removed.

Can you try creating the crtemp file after the sync has started with file watcher enabled? Because this closely replicates my setup in which sync is on, file watcher enabled, I get a call then the temp file is created and renamed when the call ends to .opus

8:54 PM

These are my sync settings

8:56 PM

Dw about the error in the background of the first image. It's because I changed my rclone config, it's resolved

Just so you know, we're on it and will get back to you once we find out the issue/fix.

Tom

Just so you know, we're on it and will get back to you once we find out the issue/fix.

No worries thanks for all your help

Exitium

No worries thanks for all your help

Thanks to your feedback we've managed to find the issue and develop a temporary fix. It's awaiting release on Android, other platforms will be addressed later this week. Bit of a background: Once we've implemented file watcher, we've also introduced concept of partial sync, to run sync only on the affected path. This logic was introduced before we've implemented include/exclude filters. Turns out that our concept of partial sync isn't compatible with include/exclude filters, as it was using these filters itself. To remediate this problem, once file watcher is enabled and file change is detected, we will run a full sync (instead of partial sync as currently). If there are multiple changes at the same time, only one sync instance will run at the time, to not collide and waste resources. In the long-run we could reinstantiate the partial sync feature that we've implemented, but every single file would have to be "checked" against these Rclone-like rules. We would need to build an interface to Rclone logic (or reimplement it from S3Drive), so we can query each file, check it against user-defined rules and either accept or reject partial sync. I hope this make sense. Thank you again for finding that out! (edited)

Tom

Thanks to your feedback we've managed to find the issue and develop a temporary fix. It's awaiting release on Android, other platforms will be addressed later this week. Bit of a background: Once we've implemented file watcher, we've also introduced concept of partial sync, to run sync only on the affected path. This logic was introduced before we've implemented include/exclude filters. Turns out that our concept of partial sync isn't compatible with include/exclude filters, as it was using these filters itself. To remediate this problem, once file watcher is enabled and file change is detected, we will run a full sync (instead of partial sync as currently). If there are multiple changes at the same time, only one sync instance will run at the time, to not collide and waste resources. In the long-run we could reinstantiate the partial sync feature that we've implemented, but every single file would have to be "checked" against these Rclone-like rules. We would need to build an interface to Rclone logic (or reimplement it from S3Drive), so we can query each file, check it against user-defined rules and either accept or reject partial sync. I hope this make sense. Thank you again for finding that out! (edited)

Hey yes that makes sense, glad you managed to identify the issue. I appreciate the extremely fast turnaround also! I'll keep an eye out on the play store for when it goes live. Thank you

@Tom updated the app and everything is working entirely as expected now, thanks once again

1

That's great to hear, if you do spot any oddities please let us know. Thanks again!

1