Include/Exclude mechanism

[memeplex]

As go regexps don't support negative lookahead is difficult to exclude some set defined by a barely selective pattern (like ^\. in a home directory) and yet include some small subset of it (say some dotfiles I want to sync).

[odeke-em]

Hello @memeplex, thank you for reporting this and welcome to drive!

Would you mind giving an example for your use case because I don't yet understand the deficiency?

[memeplex]

Sure. Say I want to sync the part of my home directory with contents generated by me. This will include some dotfiles and most directories and files that don't start with a dot. But there is a large number of dotfiles that I don't want to sync at all. So the rules will be something like:

-^\.*$
+^\.mydofile1$'
+^\.mydotfile2$'

I could do that with a negative lookahead as follows:

exclude=^\.(?!(mydotfile1|mydotfile2))$

But (?! is not supported by go.

[memeplex]

See for example rclone, which implements rule based filtering: http://rclone.org/filtering/

[odeke-em]

Alright, thanks for the example, am gonna take a look at this and make a couple of comparisons then get back to you.

[memeplex]

I thought drive push .mydotfile1 .mydotfile2 could be a workaround but it doesn't push anything. Am I doing something wrong?

[memeplex]

It only seems to work when .mydofile1 and .mydotfile2 are already in the server, then changes are pushed. Is it expected to work like that?

[odeke-em]

So I've ran your tests to compare with other languages e.g Python and none has matched, maybe a revision of the regex might change things. Please see https://gist.github.com/odeke-em/c29ae8bc475fd0fb9eda.

In regards to:

I thought drive push .mydotfile1 .mydotfile2 could be a workaround but it doesn't push anything. Am I doing something wrong?

drive won't push anything that you've already synced and needs no changes unless you request an explicit push in this case with a --force flag. Please see https://github.com/odeke-em/drive#pushing. Otherwise if the content changes then sure it will push it up or if modTime changes occur.

Would you mind giving me your drive version, ie after running

$ go get github.com/odeke-em/drive/drive-gen && drive-gen
$ driver version

that should give you information kind of similar to this

$ go get github.com/odeke-em/drive/drive-gen && drive-gen && drive version
drive version: 0.3.4
Commit Hash: '6992e27390b35ec51db59ae1d03e0cb439bbb38b'
Go Version: devel +337c5541 Sun Dec 20 20:08:59 2015 -0700
OS: darwin/amd64
BuildTime: 2015-12-27 20:28:07.005553354 -0700 MST

[odeke-em]

Oops, I forgot to mention that hidden files by default are ignored, please use the --hidden flag during a pull or push.

[odeke-em]

I think I can also reproduce the .dot files not being pushed. I'll file a bug about .dot files not being pushed, probably a regression. Thanks for that catch!

[memeplex]

Regarding the regexp: just remove the ending $.

Regarding the push thing: I'm not testing it with hidden files. What I'm doing is:

1. Create a new file or directory xxx.
2. Run drive push xxx.

Then nothing happens. Now if I push the parent directory of xxx, then xxx is added to the server. After that I:

1. Modify the contents of xxx.
2. Run drive push xxx.

Then xxx is updated in the server.

[odeke-em]

Roger that. I can confirm that I broke this functionality with PR #528 and here is a breakdown of what I screwed up:
Before a raw lookup was done for the first file, if none was returned, an appropriate error and nil file was passed back so a need to sync would be made.
Now a channel of files ie get all the matches is returned and if no match was made, that channel is closed and no files are registered as registered whether empty or not hence no syncing ever occurs since matches are within a loop ie

for rem := range remotesChan {
   ....
}

// Will never be run if the file didn't exist upstream.

Am patching this up and thank you very much for bringing this to my attention. Fix coming up soon.

[odeke-em]

@memeplex please get the latest from master and your work around should be working. The bug in your push/pull workaround that you found has been addressed by PR #537.

[odeke-em]

You can do so if you don't already know by

$ go get -u -v github.com/odeke-em/drive/drive-gen && drive-gen

[memeplex]

Thank you very much. Despite include/exclude support would be nice lacking regexp lookahead, I think I can workaround that with some shell glob matching and your fix.