code:
Direct mode is great because it removes symlinks. A must-have for directories like `~/Documents`. Unfortunately, it removes the possibility to use `git` commands other than `git-annex`. Also, it doen't preserve history of files.

I would be great to have a mode where:

- files are available in plain, not as sylinks
- the repository could still be trusted to hold version of some files from other repositories
- from a user point of view, the history of a file before the checkout would be preserved.

In feature rich file systems that have copy on write feature, it could be implemented by having the files in both places at the same time:

- the current version of a file would be in the working copy
- the file in the working copy would be a copy-on-write of the file in the annex repository
- when the file in the working copy changes, `git-annex` notices it and copy the file in the annex repository using copy-on-write semantic

If the file system do not support copy-on-write, it could be an option (do you want secure direct mode that takes twice the disk space or light direct mode that don't preserve the history of your files?)

This would make direct more much more robust.

copy on write is available using `cp --reflink=always`. It correspond to the following code ([coreutils src/copy.c line 224](http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/copy.c#n224)):

    /* Perform the O(1) btrfs clone operation, if possible.
       Upon success, return 0.  Otherwise, return -1 and set errno.  */
    static inline int
    clone_file (int dest_fd, int src_fd)
    {
    #ifdef __linux__
    # undef BTRFS_IOCTL_MAGIC
    # define BTRFS_IOCTL_MAGIC 0x94
    # undef BTRFS_IOC_CLONE
    # define BTRFS_IOC_CLONE _IOW (BTRFS_IOCTL_MAGIC, 9, int)
      return ioctl (dest_fd, BTRFS_IOC_CLONE, src_fd);
    #else
      (void) dest_fd;
      (void) src_fd;
      errno = ENOTSUP;
      return -1;
    #endif
    }

Looking at the code it would be preferable to exec directly to `cp`, see [copy_range() on LWN](http://lwn.net/Articles/550621/) and this [more recent article about splice() on LWN](http://lwn.net/Articles/567086/)

Also, `cp --reflink` fall back to copy when copy-on-write is not available while `cp --reflink=always` do not.


code: When using git-annex in a distributed fashion (lots of repos everywhere) It is easy to lose track of which remotes has a particular repo and enable it. Currently I have to run `git annex info` and see which remotes are available then add them through the webapp. Would it be possible to make webapp show all repos not just the ones it is syncing give an option to enable it.

repo_name: kakafarm/git

license: unknown


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.191"
 subject="comment 1"
 date="2014-04-17T20:54:18Z"
 content="""
I chose to make the webapp only show available remotes when it knows how to enable them. So it will show S3 special remotes, Box.com special remotes, etc. This avoids cluttering up the display of a shared repository with a list of all of your friend's removable drives, for example.

I would like to make the webapp smarter about handling repositories on remote ssh servers. As long as the server name is in the global DNS, the webapp could easily walk the user through setting up such a remote. The missing piece is that nothing is logged in remotes.log for these remotes, and so the assistant doesn't know the server name.
"""]]


code:
### Please describe the problem.
Instead of storing config for each remote in ~/.ssh/config, which mixes the user own config with that of git-annex-assistant, which is irritating if (like me) you store your ssh config in a vcs. Since the option -F allows the choice of the config file, it should be possible to move the config into ~/.ssh/git-annex/config. The only issue I see is according to the ssh man page on my system states that the system-wide config is ignored if a config file is specified on the command line.

### What version of git-annex are you using? On what operating system?
I'm using git-annex 4.20130601 on a Debian Testing/Unstable/Experimental mix.

[[!tag design/assistant]]

[[wontfix]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-06-11T14:44:47Z"
 content="""
The only interface git provides to do this is `GIT_SSH`, which would have to be set to a wrapper script that runs ssh with the desirned options.

And if that were used, `git pull` by itself would not work on the repositories set up by the assistant. I don't consider that very nice.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-06-11T14:48:01Z"
 content="""
Also, if you're going to set up something like local pairing, why would you *not* want to commit that config to git along with your other ssh configs? Config files in $HOME are quite frequently edited by helper programs to configure changes, and I personally commit those changes all the time.

Perhaps your real problem is that you have one `.ssh/config` that is shared between multiple hosts, and the git-annex settings are specific to a single host. Have you considered using [vcsh](https://github.com/RichiH/vcsh)?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawk3Wgg0XiqYFwM_Pw1RxZwlpNFi65g17sM"
 nickname="James"
 subject="comment 3"
 date="2013-06-12T01:12:24Z"
 content="""
Ah, ok, I presumed there was an option in git to set a per-repository ssh command. I've looked at vcsh, but I'm not that confident with git remotes, so I don't use it (I use hg). If a per-repository ssh command added to git, would you consider adding this?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 4"
 date="2013-06-12T19:23:50Z"
 content="""
If it were sane, I'd probably use it.

In the meantime, I'm moving this to [[todo]].
"""]]


code:
The [[special remotes]] available all do great things and enable a ton of different services to be integrated.

Strikingly, the one service I can't satisfactorily integrate with git-annex is a remote folder on a eg NAS (think: computer without git-annex installed) that I want to look like the original annex. As in, when I do a 'tree annexdir' it'd look the same on both locations (except, on the remote there would not be any symlinks, it'd be like it was in directmode, and there would not be a .git subdir).

## Why? Use Case?

I have a Synology NAS that I share access with with my wife. I want her to be able to access the files (photos/videos/music) in a sane manner (ie: not traversing sub-sub-sub 'randomly' named directories) but I also want to be able to manage them with git-annex on my machine (to gain the standard git-annex benefits, specifically the bob the archivist use case). The NAS has the ability to use ssh+rsync, so I'll assume those two tools can be used.

This special remote could be thought of as the 'least common denominator of special remotes'; almost any server with ssh+rsync could be a remote, no matter if you have install privs or if the architecture (eg: ARM) is supported by git-annex.

## Issues?

First and foremost, this can't be (really really shouldn't be) a trusted remote; my wife could accidentally delete all files on the NAS while I am away. So my local git-annex shouldn't assume the NAS counts towards numcopies (unless I'm a real masochist).

Secondly, what to do when files change/are added/removed on the special remote? Probably the same thing that the assistant does with everything. The only thing special is that new/modified files will need to be copied locally from this special remote before being added to the annex (to get hash and such).

> This is not feaisble given git-annex's design. If I wanted to
> make something completely unlike git-annex, I suppose it could be done,
> but it's off topic here. [[wontfix|done]].
>
> If you want to use git-annex on a Synology NAS, the arm standalone build
> will work, and then you can use the command-line, or the assistant
> to maintain a git repository that contains your files as desired. --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 10"
 date="2013-12-16T20:41:47Z"
 content="""
How is that better than `rsync -a gitrepo cleandirectory`?

Perhaps better related to the original problem of getting git-annex running on a Synology NAS, [[forum/new linux arm tarball build]].
"""]]


code:
[[!comment format=mdwn
 username="TroisSinges"
 ip="82.227.207.5"
 subject="comment 1"
 date="2013-11-27T14:59:11Z"
 content="""
You could mount this NAS as a network drive on your computer (via samba for example or, even better, nfs), couldn't you? In this case you could use a real annex repository, and not a remote one.
"""]]


code:
[[!comment format=mdwn
 username="http://grossmeier.net/"
 nickname="greg"
 subject="mounting over NFS/etc ..."
 date="2013-11-27T16:01:50Z"
 content="""
TroisSinges: Yeah, I could, but that'd limit me to being on the local network when I want to interact with the repo; not ideal.
"""]]


code:
[[!comment format=mdwn
 username="http://a-or-b.myopenid.com/"
 ip="203.45.2.230"
 subject="comment 3"
 date="2013-11-28T02:36:15Z"
 content="""
+1 on this suggestion.

Not all NASs are created equal so SAMBA/NFS isn't always the most recent version, upgrades aren't always possible (yes, I'm looking at you Drobo!) so sometimes you just can't do what g-a needs to do (locks are my big problem).

This would be a great option to have, even with the downsides of losing a lot of the awesome-sauce of git-annex.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmVICFY2CDP08xdsPr3cgmScomy9HA-1sk"
 nickname="Andrew"
 subject="comment 4"
 date="2013-11-29T04:11:24Z"
 content="""
+1 for this style of repository (again).

Presenting the content of a git-annex repository in a way that

1. is transparent (nobody but me need know or care it's a git-annex repository)
2. doesn't require installation (NAS or other device I can't install to)
3. works without thinking (doesn't required commits or checkouts or manual deferencing symlinks - see also point 1)

would fulfil the \"like dropbox but awesome\" idea from the Kickstarter.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 5"
 date="2013-12-05T19:52:52Z"
 content="""
Comment #1 was my initial reaction. A regular direct mode git-annex repository on the mounted NAS share.

I don't quite understand the response to it \"that'd limit me to being on the local network when I want to interact with the repo\". How could git-annex update this NAS if it's not on the local network?

So far, the only reason that has been brought up that makes sense to me is locking. git-annex does use locking in its repository to prevent 2 git-annex commands run at the same time doing conflicting operations. (Lack of that locking could in some cases cause data loss.) I think git-annex's locking is limited entirely to files within the .git directory. So, if a SMB mount for a NAS does not support locking, one approach could be to move the .git directory to local disk, and use `GIT_DIR` to make it be used when operating in that repository. This should get you exactly what was requested. Whether it makes sense or is sufficiently easy to use is another question.

I tend to think that a better use of a NAS in many cases is to put a directory or rsync (if it supports rsync) special remote on the NAS, and have git-annex on the individual client machines. The webapp could have a UI that makes setting up a NAS like that simple, and it would be easy to do.

The use case for having a mirrored directory tree on the NAS seems to be limited to when the clients using it are something that cannot run git-annex itself, but that still cares about filenames, as opposed to just operating on file contents.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 6"
 date="2013-12-05T19:57:32Z"
 content="""
The other problem with this idea is that it seems to want the assistant to watch the file tree in the NAS, and make commits etc when files there are changed.

But, any computer on the network that has the NAS mounted can change a file! AFAIK there is no interface to detect when such changes have happened; inotify does not help. It seems git-annex would have to expensively periodically scan the directory to find changes.
"""]]


code:
[[!comment format=mdwn
 username="http://grossmeier.net/"
 nickname="greg"
 subject="comment 7"
 date="2013-12-05T21:20:54Z"
 content="""
>  How could git-annex update this NAS if it's not on the local network?

ssh and port forwarding, like any home server.

> I tend to think that a better use of a NAS in many cases is to put a directory or rsync (if it supports rsync) special remote on the NAS, and have git-annex on the individual client machines.

I'm thinking of this like the public view of a git-annex repository. I want anyone to be able to view/download the files as they normally would.

> The use case for having a mirrored directory tree on the NAS seems to be limited to when the clients using it are something that cannot run git-annex itself, but that still cares about filenames, as opposed to just operating on file contents.

That's exactly my use case.

> So far, the only reason that has been brought up that makes sense to me is locking.

Locking is actually tangential to the issue/use case. Direct mode would solve that issue, but git-annex isn't built for the Synology NAS platform, so it isn't an option.

> The other problem with this idea is that it seems to want the assistant to watch the file tree in the NAS, and make commits etc when files there are changed.

I think you read too much into that part. :) I only want git-annex to do things when I 'sync' or similar. I'm actually not wanting *the assistant* to do much in this case. This might even be a non-assistant applicable use case. (I don't use the assistant in my large repositories, like photos and videos and music.) I only mentioned the assistant to point out that it has some of the logic (what to do with changed/deleted files) already.

And honestly, I'd be fine with an implementation that overwrote all changes on this special remote on each sync (as needed). The response to that suggestion is \"just use rsync by itself to put the files on the NAS\" but that assumes I have all of the files on my local checkout (I don't).
"""]]


code:
[[!comment format=mdwn
 username="gan"
 ip="85.224.8.104"
 subject="Hook implementation?"
 date="2013-12-07T01:15:46Z"
 content="""
1. I would propose the name \"plain directory\" for this special remote.  Clean is not a good description, IMHO.

2. When I read this a week ago I took the opportunity to play with the special remote hook, because as far as I can understand this could be a relatively trivial implementation that need not necessarily be in git-annex \"core\".   And I'd like to contribute back to the git-annex community and help a brother out...     I used the method of calling a single script from all hook points, and inside it look at what operation was requested.
I didn't get that far before giving up however.  I got the feeling that hook would benefit from some extension and a bit more complete API to really create the opportunity for third-party extensions to git-annex this way.   I would prefer that the hook script can handle several instances of remotes without creating custom hook scripts for each instance.  Specifically I think a hook is needed also for \"init\" of a new remote (to get a unique identifier, and other parameters given by the user such as in this particular case, the path to the \"clean directory\")   What do you think?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmicVKRM8vJX4wPuAwlLEoS2cjmFXQkjkE"
 nickname="Thomas"
 subject="pseudocode"
 date="2013-12-08T15:23:36Z"
 content="""
The implementation should really be nothing more than the following to be run from a regular git annex repository with a configures special plain directory remote:

    foreach(path : allAnnexedFiles) {
      if(remote.exists(path) && remote.filesize(path) === expectedFilesize) goto finally;

      if(fileAvailableLocally(path)) {
        copyFileToRemote(path);
        goto finally;
      }

      if(shouldCopyFromElsewhere && canCopyFileFromSomeOtherRemote(path)) {
        copyFileFromSomeOtherRemoteToRemote(path);
      }

      finally:
        logThatFileExistsOnRemote(path);
    }

    foreach(remotePath : filesInRemoteDir) {
      if(fileIsAnnexed(remotePath) || fileIsIgnored(remotePath) continue;

      delete(remotePath);
    }

The above pseude code assumes
- that the no other process is working on the remote at the same time.
- that the remote is not trusted.
- that nobody expects changes done at the remote to propagate to other clones.
"""]]


code:
[[!meta title="Android is only autobuilt for arm, not x86 or mips"]]

### Please describe the problem.

git-annex doesn't start on [Android-x86](http://www.android-x86.org) in VirtualBox (version 4.1.18-dfsg-2+deb7u1).

On Android 4.2.2 (android-x86-4.2-20130228.iso) it starts the terminal which prints nothing but `[Terminal session finished]`.
On Android 4.3 (android-x86-4.3-20130725.iso) it starts the terminal and prints:

    In mgmain JNI_OnLoad

    [Terminal session finished]

The browser/webapp is never started.

### What version of git-annex are you using? On what operating system?

Version 1.0.52 for Android. I made sure to install the correct APK files for each version of Android.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 1"
 date="2013-11-22T16:37:36Z"
 content="""
git-annex is a native binary, and is currently only being built for arm android.

Is Android on x86 a thing used on real-world hardware? I have only seen it in the context of a developer's test environment.

The instructions for building git-annex from source for Android should work on x86. The ghc-android build would need to be tweaked to build a cross compiler targeting that architecture. This can be done by editing settings near the top of ghc-android's `build.sh`.

I am going to move this from bugs/ to todo/ after posting this comment, because it is certiantly not a bug, but a wishlist item at best.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlYSw35UMlOf9rFlP5EUTBz7is5X09qVXI"
 nickname="Johannes"
 subject="Support on Motorola razr i"
 date="2013-11-28T16:10:38Z"
 content="""
Hi,
I'm using the Motorola razr i. So there is a usecase, where a x86 CPU is used in a mobile phone. If it's easy as you described, can you build a Version with x86 support?

Thanks
Jack
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.191"
 subject="comment 3"
 date="2014-04-20T21:01:50Z"
 content="""
Samsung Galaxy Tab 3 (GT-P5210)  also needs this.

Each new git-annex builder takes me days to set up, and is an ongoing drain to keep running. I would much rather walk someone else through setting one up, unless this becomes a very common architecture for android.
"""]]


code:
## Use case:

You have a few annexes that the assistant is watching for you. You're somewhere with poor wifi speed. You also just added a bunch of big files to a few annexes. Now all of a sudden your connection suffers and you want an easy way to pause all transfers until you're on a faster connection without losing the automatic 'add' and such of the assistant (iow: without having to shutdown the daemon).

## Proposal:

A "Pause all transfers" button in the webapp that pauses all transfers from all annexes the assistant is watching.

It should toggle to "Resume all transfers" when pushed so you can also easily start the transfers again when you get somewhere else.

This may or may not make more sense if the webapp showed all watched repos in a single view (instead of the separate pages/views as it is now).


code:
Instead of using `sync origin` for the first sync and a simple `sync` for the other syncs,

    # on pc1
    git annex init "pc1"
    git annex direct
    git annex add .
    git annex sync origin # remote specified on the first sync

    # add some files
    git annex add .
    git annex sync

I used `sync` first and only later I used `sync origin`

     # on pc1
    git annex init "pc1"
    git annex direct
    git annex add .
    git annex sync

    # add some files
    git annex add .
    git annex sync origin  # remote specified on a later sync

These sequences of commands create two completely different git histories.

More important, if one clones on pc2 the first repository, they will see both the pc1 remote and the pc2 remote. Instead, if one clones on pc2 the repository created by the second combination of commands, they will see only the pc2 remote.

What commands should I use on pc1 to fix the history so that when pc2 clones from the origin repository it will see both the pc1 remote and its own local remote?

> [[done]]; fixed per my comments. --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 1"
 date="2014-07-14T18:27:43Z"
 content="""
If this is a todo item at all, it may make sense to make `git annex sync` sync even with git remotes that have no annex-uuid.

As far as solving the problem, I think you just need to sync more in order to get the full list of remotes propigated around to all the remotes. Sync automatically merges disconnected git histories no matter how they got that way or how long they have been disconnected or diverged.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 2"
 date="2014-07-14T18:38:58Z"
 content="""
It seems that the person who filed this todo item also filed [[bugs/sync does not commit with alwasycommit = false]] and got pretty confused by that.

So, repurposing this todo item to be about perhaps syncing with remotes that have no annex-uuid by default.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 3"
 date="2014-07-15T18:23:50Z"
 content="""
Making `git annex sync` automatically sync with remotes with no annex-uuid is more complicated than I first thought.

In the case of a remote accessed over ssh, `git annex sync` already does sync with such a remote. Of course, it will set annex-ignore on it, since it has no annex-uuid. (Needed eg, for github, or just for preventing a repo from being used by git-annex if you don't want it to be.) Still, the git branches get synced, which is the behavior that we want.

So, only local remotes are affected. Note that `git annex assistant` automatically git-annex inits the local remote when it lacks a uuid, and syncs with it. That seems ok.

However `git annex sync` currently ignores the local remote when it has no uuid. Seems that this happens due to a bug, not intentionally. tryGitConfigRead tries to bootstrap up an annex state to read the repos's config, but this cannot be done in a repo that is not yet initialized. Result is the repo state is not read, and so it's treated as a local remote that is not currently available (ie, a disconnected disk).
"""]]


code:
## Use Case

I want to import a bunch of files that are hosted somewhere, they nicely sorted by year and such. Instead of addurl'ing each by hand (or writing a custom script each time this happens) I want to simply:

git-annex addurl --recursive http://somehost.tld/somedir/

For sanity, mimicking wget closely with default depth of 5, but customizable with the --level switch.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.244"
 subject="comment 1"
 date="2014-04-07T20:10:51Z"
 content="""
Recursively traversing websites is *hard*, so I would rather leave it out of git-annex.
"""]]


code: It would be helpful to show each repo's type in the list.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 1"
 date="2013-11-02T23:45:39Z"
 content="""
Currently if you go to the edit page for the repository, it shows some information about it, including its type and often its location, at the bottom of the page.

I tend to feel that putting anything else in the repo list would result in it being too cluttered.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmNu4V5fvpLlBhaCUfXXOB0MI5NXwh8SkU"
 nickname="Adam"
 subject="comment 2"
 date="2013-11-02T23:59:16Z"
 content="""
I understand.  Well, here's what I'm looking at right now.  :)

[[http://alphapapa.net/outbox/view.png]]

I just think it would be very useful to say \"Client\" or \"Transfer\" or \"Full Backup\" next to each one, and there's often plenty of room, depending on the size of the window.  Especially when one's trying to wrap one's head around git-annex, being reminded what each repo is for would help a lot.  Otherwise you basically have to put it in the name or description yourself...so why not just go ahead and show it?  :)

Another option might be to have an icon for each type.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 3"
 date="2013-11-03T00:19:34Z"
 content="""
I'd be happy to put in some icons if someone finds some good (and suitably licensed) ones that cover the different types of repositories.
"""]]


code:
What steps will reproduce the problem?
Sync a lot of small files.

What is the expected output? What do you see instead?
The expected output is hopefully a fast transfer.

But currently it seems like git-annex is only using one thread to transfer(per host or total?)

An option to select number of transfer threads to use(possibly per host) would be very nice.

> Opening a lot of connections to a single host is probably not desirable.
>
> I do want to do something to allow slow hosts to not hold up transfers to
> other hosts, which might involve running multiple queued transfers at
> once. The webapp already allows the user to force a given transfer to
> happen immediately. --[[Joey]]

And maybe also an option to limit how long a queue the browser should show, it can become quite resource intensive with a long queue.

> The queue is limited to 20 items for this reason. --[[Joey]]


code:
I'm currently in the process of gutting old (some broken) git-annex's and cleaning out download directories from before I started using git-annex.

To do this, I am running `git annex import --clean--duplicates $PATH` on the directories I want to clear out but sometimes, this takes a unnecessarily long time.

For example, git-annex will calculate the digest for a huge file (30GB+) in $TARGET, even though there are no files in the annex of that size.

It's a common shortcut to check for duplicate sizes first to eliminate definite non-matches really quickly. Can this be added to git-annex's `import` in some way or is this a no-go due to the constant memory constraint?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="24.159.78.125"
 subject="interesting idea"
 date="2014-07-30T15:03:46Z"
 content="""
This could be done in constant space using a bloom filter of known file sizes. Files with wrong sizes would sometimes match, but no problem, it would then just do the work it does now.

However, to build such a filter, git-annex would need to do a scan of all keys it knows about. This would take approximately as long to run as `git annex unused` does. It might make sense to only build the filter if it runs into a fairly large file. Alternatively, a bloom filter of file sizes could be cached and updated on the fly as things change (but this gets pretty complex).
"""]]


code:
[[!comment format=mdwn
 username="Xyem"
 ip="81.111.193.130"
 subject="comment 2"
 date="2014-08-01T09:05:45Z"
 content="""
Could be tested out with an additional flag `--with-size-bloom` on import?

It would then build a bloom (and use a cached one with --fast) and do the usual import.

So I could do this:

    # Bloom is created and the import is done using it
    git annex import --clean-duplicates --with-size-bloom $TARGET

    # Previously created bloom is used
    git annex import --clean-duplicates --with-size-bloom --fast $TARGET2
    git annex import --clean-duplicates --with-size-bloom --fast $TARGET3

I can implement this behaviour in Perl with Bloom::Filter and let you know how it performs if that would be useful to you..?
"""]]


code:
Currently events happening in the webapp (sync upload etc. on the right) has no time stamp thus user has no way to tell when was the last sync happened. Which is problematic when not using XMPP and repos lag behind.

> [[dup|done]] of <http://git-annex.branchable.com/todo/wishlist__91__minor__93__:_add_time_stamps_to_annex_log_popups_in_webapp/> --[[Joey]]


code:
Currently if photos or videos are copied into the Camera/DCIM directory on an Android device, or deleted the Gallery doesn't notice the changes.

It is necessary to call MediaScannerConnection - http://developer.android.com/reference/android/media/MediaScannerConnection.html - to notify the system of the change.

More info, and some sample Java code: http://stackoverflow.com/questions/13270789/how-to-run-media-scanner-in-android

It'd be awesome if the assistant did this on files it has changed.  Possibly just under Camera/DCIM, but perhaps it should be configurable. MediaScannerConnection is also used to notify and index new music files.


code:
There are times when it is handy to be able to upload a file to a web host somewhere and share a link for that file to a select few people.

It seems to be that the assistant could handle this scenario. It could generate a directory with a random name on the remote, and transfer the file there (using the existing filename) and the appropriate URL could be displayed in the assistant webapp to allow the user to copy the URL to send it to the appropriate people.

Note: Joey and I had a quick chat about this use case at LCA2013.

[[!tag design/assistant]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="99.54.57.201"
 subject="comment 1"
 date="2013-02-09T22:38:56Z"
 content="""
This would be an extremely cool feature to rip off from Dropbox. :)

"""]]


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="2001:1928:1:9::1"
 subject="comment 2"
 date="2014-04-07T04:35:39Z"
 content="""
i wonder if the assistant or the new daemon couldn't do this job as well... -- [[anarcat]]
"""]]


code:
Joey,

I've been thinking about leveraging git-annex for a workgroup document repository and I have just watched your views demo. The timing of the demo is great because I need to deploy a document repository with per-document metadata and your views concept seems like a great mechanism for associating metadata to documents and for displaying that metadata.

While I don't expect to use your views concept for my workgroup repostory, a later iteration might do.

The metadata in my use case begins with all the weird metadata seen on a book's copyright page. In addition, per-document provenance, like how one found the document and (if we're lucky) a URL where the latest version of the document may be found.  Metadata values may be simple strings or may be markdown text.

So, are you considering a metadata syntax that can support complex metadata? One example is multiple authors. Another issue is complex metadata values, like key=abstract and value="markdown text...".

FWIW,

Bob

> [[closing|done]]; requested feature was already present --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.172"
 subject="comment 1"
 date="2014-02-24T18:17:04Z"
 content="""
All that should work fine. All metadata fields are multivalued, and the value can be any arbitrary data.
"""]]


code:
It would be nice if a couple of additional environment variables to be set for hook uses.

In particular:

    GIT_ANNEX_DIRECT=`git config annex.direct`

and

    GIT_TOP_LEVEL=`git rev-parse --show-toplevel`

I've made some changes to flickrannex to allow the sub-directories above the uploaded image to be added as tags.  This change has been merged into trunk: [[https://github.com/TobiasTheViking/flickrannex]]

What I needed was both the environment variables mentioned above.  One is set as part of the annex-hook and the other I guestimate from the file path.  If it was set in git-annex it would be much cleaner (and accurate).  So...I think this info would be useful for other hook.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.154"
 subject="comment 1"
 date="2014-03-18T19:36:08Z"
 content="""
Is this still relevant?
"""]]


code:
Auto-repair takes up a lot of computing power, greatly slowing down the machine. It also still happens very often here, several times a month on different hardware, leading it to become an annoyance (and it would be easier and faster to perform manual repairs if that should be required).

Therefore I'd like a setting that makes the assistant not attempt to repair on its own. Either not do anything at all, or display a message about it instead.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.3"
 subject="comment 1"
 date="2014-05-21T18:29:16Z"
 content="""
What auto-repair are you referring to?

If you schedule a fsck of the repository in the webapp, then when that finds a problem with the git repository, it will be repaired. So if you don't want this, remove any scheduled fsck jobs.

The assistant also detects a few common problems at startup that prevent it from working, such as a corrupt index file, and automatically repairs those. These repairs only happen at startup. If the index file is corrupt, it has to delete it and re-add every file to the repository, which is expensive, but otherwise it would be completely non-functional.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.3"
 subject="comment 2"
 date="2014-05-21T18:35:42Z"
 content="""
Note that this is asking for a workaround for the bug [[bugs/Auto-repair_greatly_slows_down_the_machine]]. I would rather get to the bottom of that bug, but need more information to do so.
"""]]


code:
[[!comment format=mdwn
 username="EskildHustvedt"
 ip="80.202.103.55"
 subject="comment 3"
 date="2014-05-21T19:18:19Z"
 content="""
I'm referring to the large auto-repairs of the actual git repository, as mentioned in the bug report you are referring to. I was under the impression that the fscks also did the equivalent of 'git annex fsck' (which is greatly desired, even without the 'git fsck' checks). I realize it is a sort of workaround, but it is because the 'annex fsck' bit is desired, while the 'git annex repair' bit makes my computer unusable, the last time for over 24 hours, while it was running (at which point I needed to kill it and perform a manual repair).
"""]]


code:
In certain situations different client annexes might get the same remote repository added, but before being synced.

Once the two clients sync they will both have two remotes with the same name. But only one UUID will have any content(Assuming only one client pushed).

It would be nice to have some (automatic?) way to resolve this conflict.

Not sure if anything sane can be done if both clients have pushed?


code:
Please let me know if there already is a way of achieving this; I’ve googled around a lot, but could not find any information pertaining to my particular problem.

I am using direct mode in a bunch of repositories where I need quick write access to content and where I am not interested in preserving history. Some of these repositories do contain regular symlinks, however. Now, I suppose that in indirect repos, the way of adding symlinks would be to just «git add» them. However, since these are direct mode repos, I cannot do this.

Is there already a good way of adding symlinks in direct mode? If not, I would find it useful if there were one.

Best regards,
T.


code:
[[!comment format=mdwn
 username="zardoz"
 ip="134.147.14.84"
 subject="comment 1"
 date="2014-07-07T12:48:07Z"
 content="""
Can it be considered safe adding symlinks via «git -c core.bare=false add symlink; git -c core.bare=false commit -m Update»? If not, is there a better way?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 2"
 date="2014-07-10T19:54:30Z"
 content="""
That is safe, but you have to be very careful anytime you override with -c core.bare=false. For example, if you did a `git commit -a`, it would commit your large files directly into git, which you don't want.

"""]]


code:
My original use case was for using git-annex find from scripts, where I didn't want to depend on the branch
checked out at the time, but rather write something like "git annex find --branch=master $searchterms"


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.154"
 subject="comment 1"
 date="2014-03-17T19:48:57Z"
 content="""
The difficulty with adding a --branch is that if it causes git-annex to operate on a list of (file, key) from the branch, then commands that actually modify the working tree would modify it, instead of the branch. So the options seem to be only generating a list of keys, and so only letting commands that operate on keys work (which rules out the `git annex find` example), or carefully arranging for commands that actually affect the work tree to not be usable with this option.

I'm not sure how many commands are affected. The ones I can immediately think of are sync, lock, unlock. (Commands like get obviously affect the work tree in direct mode, but it's fine to have getting a file from a branch also update files in the work tree, if they pointed at the same key.)
"""]]


code:
right now it is unclear through the webapp how to unconfigure a jabber
account, which is especially critical considering the password needs to be
stored in the clear (where?). -- [[anarcat]]

> [[fixed|done]] --[[Joey]]


code:
[[!meta title="webapp does not accept ssh alias for ssh remote"]]

What steps will reproduce the problem?

Using the assistant, create an SSH remote.  Try to use an alias as the name
of the remote (e.g. I have a server which I have aliased to "homeworld" in
my .ssh/config.  When I'm at home, that is an alias for 192.168.1.253.
When I'm not at home, I edit .ssh/config so that "homeworld" becomes an
alias for a hostname at no-ip.com.)  Despite the fact that "homeworld" is a
viable ssh target because of the alias, the assistant doesn't recognize it
as a valid host to ssh to.

I had trouble with an ip address the first time I tried it but just tried
it again and it worked fine, so please disregard that part of the title of
this bug report.

What is the expected output? What do you see instead?

expected output = move to the "create a repository -- rsync or regular" page.
observed output = "cannot resolve host name"

What version of git-annex are you using? On what operating system?

 Version: 3.20130102 OS X Lion

Please provide any additional information below.

I realize this is kind of a power user whine.  Using an ssh alias which
does not correspond to an actual resolvable hostname (and cannot, because
it's supposed to be a layer of indirection over the hostname) is not an
everyday problem for an average user.

> The assistant tries to resolve the hostname explicitly
> to catch user's typos, and also expands it to a FQDN, to make
> it more likely to be able to reach the host when roaming to other
> networks.
>
> Also, the assistant sets up it *own* .ssh/config hostname alias,
> in order to make it use the special ssh key that it generates for the host.
> So that is not compatable with using a ssh host alias you've set up.
> Even if it knew about your alias, it would set up a new hostname alias, and
> whatever machinery you have to update the alias would not work.
>
> You can, of course, add git remotes using any ssh alias you like, by
> hand, and restart the assistant and it will use them. --[[Joey]]

[[!tag /design/assistant]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
 nickname="Justin"
 subject="comment 1"
 date="2014-05-22T20:05:54Z"
 content="""
A temporary workaround is to add an entry to /etc/hosts until the remote is added.
"""]]


code:
Hi and thank you for an incredible piece of software and great work!

I've noticed that when I add new files to a repository and I have my USB drive connected, the assistant alternate it's transfers of files. And only transfers one queued file at the time.

file1 -->> Internet offsite computer
file1 -->> USB drive
file2 -->> Internet offsite computer
file2 -->> USB drive

I would prefer a logic where the assistant transfer files in parallel to my different repositories. I know that it might not be a good thing doing that with network accessed repositories, but when I have "low cost", locally attached USB drives it would be great if the transfers could be done in parallel.

Is there a configuration option for this already?


code:
When maintaining several replica of the same git-annex repo "git annex sync" is quite handy.
But it would be even handier if "git annex sync" would also perform automatic "git merge synced/*" actions on all remotes.

Clearly, this is beneficial when the user wants to keep all working copies synchronized.
This is likely the case in git annex assistant like scenarios. And it's always the case in my day to day scenarios :-)
I'm not sure about other use cases that I've hard time imagining...

As just discussed on IRC (#vcs-home/OFTC), this could be implemented in various ways:

1) By doing ssh on each remote and running the appropriate "git merge ..." commands there.
   The drawback of this is that quite often it won't be permitted to ssh on the remote and run arbitrary commands there.

2) Having a default post-receive hook, created at the time of "git annex init" that automatically does the merges when contacted by other remotes as a consequence of "git annex sync".

Thanks for git-annex!


code:
there's a lot of good documentation on this wiki, but it's hard to find sometimes. it's also unclear if we should look in the [[git-annex]] manpage or elsewhere in the wiki or where. this is a typical problem with the use of wikis for documentation: it's there, but hard to find. it doesn't mean a wiki shouldn't be used but, as with any user manual, special care needs to be taken about structure, organisation and making sure the manual is exhaustive.

a good example of this problem is [[todo/document_standard_groups_more_extensively_in_the_UI]]. --[[anarcat]]


code:
Most of git-annex is designed to be fast no matter how many other files are
in the annex. Things like add/get/drop/move/fsck have good locality;
they will only operate on as many files as you need them to.

(git commit can get a little slow with a great deal of files,
but that's out of scope -- and recent git-annex versions use queuing
to save git add from piling up too much in the index.)

But currently two git-annex commands are quite slow when annexes become large
in quantity of files. These are unused and status.
(Both have --fast versions that don't do as much).
> (Update: status has become acceptably fast; most of its slowdown was due to using a bad data structure; scanning the tree is not particularly slow and it no longer looks at the git-annex branch.)

unused is slow because it needs two pieces of information that are not
quick to look up, and require examining the whole repo, very seekily:

1. The keys present in the annex. Found by looking thru .git/annex/objects
2. The keys referenced by files in git. Found by finding every file
   in git, and looking at its symlink.

Of these, the first is less expensive (typically, an annex does not have every
key in it). It could be optimized fairly simply, by adding a database
of keys present in the annex that is optimised to list them all. The
database would be updated by the few functions that move content in and
out.

The second is harder to optimise, because the user can delete, revert,
copy, add, etc files in git at will, and git-annex does not have a good way
to watch that and maintain a database of what keys are being referenced.

It could use a post-commit hook and examine files changed by commits, etc.
But then staged files would be left out. It might be sufficient to
make --fast trust the database... except unused will suggest *deleting*
data if nothing references it. Or maybe it could be required to have a
clean tree with nothing staged before running git-annex unused.

Anyway, this is a semi-longterm item for me. --[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 1"
 date="2011-05-17T07:27:02Z"
 content="""
Sounds like a good idea.

* git annex fsck (or similar) should check/rebuild the caches
* I would simply require a clean tree with a verbose error. 80/20 rule and defaulting to save actions.
"""]]


code:
Since most file transfers, particularly to/from encrypted special remotes involve git-annex streaming through the contents of the file anyway, it should be possible to add a verification of the checksum nearly for free. The main thing needed is probably a faster haskell checksum library than Data.Digest.Pure.Sha, which is probably slow enough to be annoying.

I have not verified if an upload could be aborted before sending the data to the remote if a checksum failure is detected. It may be dependent on the individual special remote implementations. Some probably stream the encrypted data directly out the wire, while others need to set up a temp file to run a command on. It would certainly be possible to at least make the upload abort and fail if a bad checksum was detected.

Doing the same for downloads is less useful, because the data is there locally to be fscked. The real advantage would be doing the check for uploads, to ensure that hard-to-detect corrupted files don't reach special remotes.

--[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 1"
 date="2013-09-09T11:50:05Z"
 content="""
Doing this during downloads would still be nice.

While the files are easier to fsck, users will need to actually do this. If it happenend automatically, it would increase safety and reduce disk i/o.

Of course, this will not detect degradation during/after writing.

If you don't make it the default, please at least make it optional for us bordering on OCD when it comes to data storage.

Richard
"""]]


code:
To properly use amazon AWS S3 for CDN, we need to publish videos to S3. Ideally, we would like to do this via git-annex as the back-end of video.debian.net is being migrated to git-annex by me, atm.

Obviously, we will need clear text names and proper directory structure, not SHA512E file names. This would need to be supported by the S3 special remote.

I talked to TobiasTheViking in the past and he hinted at a reasonably clean way to do this, but that a clean solution would need support from git-annex. I will link him to this page and ask him to supply whatever info is needed.

Thanks,
Richard

> This is not feaisble given git-annex's design. If I wanted to
>  make something completely unlike git-annex, I suppose it could be done,
>  but it's off topic here. [[wontfix|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.41"
 subject="comment 1"
 date="2014-03-26T17:26:37Z"
 content="""
I don't see how this can possibly be done. A single git-annex object can have any number of file names, which can change at any time.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 2"
 date="2014-03-26T22:32:18Z"
 content="""
In that case, we would need to export the same file name several times, just like direct mode does.

Could files be tracked via metadata? And yes, fsck would be... interesting...
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.41"
 subject="comment 3"
 date="2014-03-27T17:44:46Z"
 content="""
Sounds like \"I want a pony to me\".
"""]]


code:
As of right now, the only way to commit changes to direct repositories is `git annex sync [remote]`.
There is no way to specify what directory to operate on.
When moving around files on a larger scale, the ability to commit specific subsets of changes would be rather nice.

`git annex commit [path]` or `git annex sync [remote] -- [path]` would probably make sense.

Thanks,
Richard


code:
Sometimes I start off a large file transfer to a new remote (a la "git-annex copy . --to glacier").

I believe all of the special remotes transfer the files one at a time, which is good, and provides a sensible place to interrupt a copy/move operation.

Wish: When I press ctrl+c in the terminal, git-annex will catch that and finish it's current transfer and then exit cleanly (ie: no odd backtraces in the special remote code). For the case where the file currently being transfered also needs to be killed (ie: it's a big .iso) then subsequent ctrl+c's can do that.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.172"
 subject="comment 1"
 date="2014-02-21T21:36:14Z"
 content="""
This really depends on the remote, some can resume where they were interrupted, such as rsync, and some cannot, such as glacier (and, er, encrypted rsync).
"""]]


code:
[[!comment format=mdwn
 username="http://grossmeier.net/"
 nickname="greg"
 subject="very remote specific"
 date="2014-02-21T22:11:16Z"
 content="""
Yeah, this is very remote specific and probably means adding the functionality there as well (eg: in the glacier.py code, not only in git-annex haskell). Maybe I should file bugs there accordingly :)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.172"
 subject="comment 3"
 date="2014-02-21T22:34:14Z"
 content="""
Hmm, I forget if it's possible for git-annex to mask SIGINT when it runs glacier or rsync, so that the child process does not receive it, but the parent git-annex does.
"""]]


code:
It would be great to have a custom f-droid repo "alla guardianproject.info" (before getting git-annex into the main f-droid repo).

See <https://github.com/guardianproject/fdroid-repo> (<https://guardianproject.info/repo/>).


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.146"
 subject="comment 1"
 date="2014-03-05T20:45:32Z"
 content="""
The f-droid developers are willing to add git-annex in principle, their build environment just needs to have its build dependencies added to it. I've completely automated (except for occasional breakage due to changes on hackage) to setup of the necessary cross build environment. I have not had time to look into how to integrate that into the f-droid build system, and it would be great if someone with interest could do so.

I suspect that setting up a custom f-droid repo would be just as much work, plus more work on an ongoing basis.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnR6E5iUghMWdUGlbA9CCs8DKaoigMjJXw"
 nickname="Efraim"
 subject="comment 2"
 date="2014-03-06T10:06:02Z"
 content="""
I've noticed that fdroidserver is available in the main debian repos, and not having really looked at it too much yet, if it shares similar code to wannabuild then it should be possible to continue everything as you're already doing it, just in addition rsync or git push the code base to a local fdroidserver instance and also copy the binary over once it's set up.
(if I understood wannabuild correctly, if you upload new source and a prebuilt binary it will mark that architecture as built.)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmLB39PC89rfGaA8SwrsnB6tbumezj-aC0"
 nickname="Tobias"
 subject="comment 3"
 date="2014-03-06T22:49:22Z"
 content="""
F-droid compiles all the binaries themselves internally.
"""]]


code:
[[!comment format=mdwn
 username="http://lj.rossia.org/users/imz/"
 ip="79.165.57.121"
 subject="this topic in their Submission Queue"
 date="2014-05-03T14:59:11Z"
 content="""
I opened [a request to add git-annex](https://f-droid.org/forums/topic/git-annex/) in the F-Droid's submission queue about a year ago.

Just posting here the link, so that all the discussion (not much) can be found by those interested.

"""]]


code:
[[!comment format=mdwn
 username="http://lj.rossia.org/users/imz/"
 ip="79.165.57.121"
 subject=" clarification: a custom F-Droid server just to post your updated apps"
 date="2014-05-03T15:57:55Z"
 content="""
If I've understood Efraim correctly, he means that someone (perhaps, Joey) can run a custom F-Droid server with Joey's builds of the app **copied to the repo as is (without rebuilding there)**, so that those using an F-Droid client would get the updates through this standard path (for them) instead of manually looking for updates here.

Like from a custom APT repository run not by Debian (in this case -- F-Droid team), but by an independent developer who is providing package builds not avaialble in Debian by building the packages himself, signing them, and putting them into his own APT repo (indexing them).

"""]]


code:
[[!meta title="--notify-finish operates on a per-file, not per-process basis"]]

so this is another UX pickyness, but it seems important to me.

i like the new [[desktop notifications|tips/file_manager_integration]], but they are little too verbose. when i choose "git annex get" on the folder, if there's a lot of files, it will flood me with all the files being transfered in a mostly incomprehensible list of files being transfered.

what i would expect is more: "starting transfer of folder X", "transfer of folder X finished!", only two message per item i chose. this is especially a problem with DVD backups, which have a bunch of small files (screenshots, .nfos and so on) and large video files - so it seems the thing has finished transfering, while it's only partly done. --[[anarcat]]


code:
i have been using git-annex for a while now, yet I still can't quite wrap my head around [[preferred_content/standard groups]], especially how they are documented in about/repogroups in the assistant web interface. i have repeatedly synced files where they shouldn't have been synced (usually by setting the repo as "client" or "transfer") and also destroyed files I wanted to keep by setting it to "unwanted" (actually, that was by pressing the "delete" button on the repo, which i didn't expect to drop the files on the remote...)

i have been able to understand a lot of what's going on by trial and error and by decrypting the [[preferred_content]] expressions on the wiki.

it seems to me the [[preferred_content/standard groups]] wiki page and the `about/repogroups` URL in the assistant should be merged:

 1. the assistant should be more explicit: maybe it should have examples of what will happen in some cases to give an idea. maybe "stories" like "a transfer repo is for when you have two client repos that can't talk to each other, so you use a transfer repo, e.g. a portable hard drive, to transfer files between them". having the actual, current [[preferred_content]] expressions from the [[preferred_content/standard groups]] groups page would also help, maybe in a smaller font to not scare people of
 2. the [[preferred_content/standard groups]] wiki page should be expanded to include narratives like the ones that are in the `about/repogroups` page of the assistant. that way people looking at the software from the outside can understand the mechanics better

ideally, that documentation would be the one and the same so that a change on one side would reflect on the other.

in fact, having an inline manual in the assistant would be a must: we want this thing to work offline, so it should be able to access this wiki, or whatever of it is shipped with git-annex.

that way we wouldn't have this kind of inconsistencies... more generally, maybe we could even [[build a user guide]]! -- [[anarcat]]


code:
recently fixed [[todo]] items.

[[!inline pages="./* and link(./done) and !*/Discussion" sort=mtime show=10
archive=yes]]


code: While trying so establish a jabber link via a server that runs a nonstandard port I came across the problem that git-annex always appends :5222 to the jabber server address. This effectively stops the connection procedure at a point where it does not need to.


code:
Moved this comment to todo list --[[Joey]]

[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnfM7ZF0Q5U9k1LljyDXH37cuXU5Gx6gtM"
 nickname="A"
 subject="fast migrate"
 date="2014-07-05T08:21:20Z"
 content="""
dear Joey and everybody else,
 some time ago I used \"git annex migrate\" to bring all my repositories up-to-date; after that I found (to my dismay) that some keys are SHA256, some others are SHA256E, so my data is not really deduplicated  ; now, it would possible to migrate from SHAnnnE to SHAnnn (and vice versa) very fast... but currently AFAICS git-annex recomputes the whole checksum, and this (on my USB2.0 old disks) takes forever; may somebody please implement a fast migration?
"""]]

> Certianly doable, for $hashE to $hash. Probably about an hour's work.
> --[[Joey]]

>> [[done]] --[[Joey]]


code:
Should be possible to configure an annex.diskreserve setting for local
special remotes, such as a directory special remote or possibly a bup
special remote. (Although bup's deltas will make storing some versions of
files take less space than git-annex would have to assume it would take.)


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.235"
 subject="comment 1"
 date="2013-07-11T16:19:26Z"
 content="""
Directory special remotes do already honor annex.diskreserve, it turns out. Let's repurpose this bug to be about adding a per-remote configuration, in case the main annex.diskreserve is not appropriate.
"""]]


code:
### Please describe the problem.

git uses environment variable GIT_SSH to determine SSH client.

I set it to plink.exe because I extensively use pageant infrastructure and do NOT want to have 2 systems lying around.

Unfortunately git-annex seems to ignore that.

Even worse, it results in unpredicted behavior because the git part works (e.g. clone) whereas annex/rsync does not resulting in half-ok repositories without meaningful error messages.

It only becomes evident when ssh.exe in the git repository is deleted.

### What steps will reproduce the problem?

Set %GIT_SSH% and remove ssh.exe

You will get

     git-annex: ssh: createProcess: does not exist (No such file or directory)
     failed
     git-annex: drop: 1 failed

### What version of git-annex are you using? On what operating system?

Windows 8,

     $ git annex version
     git-annex version: 5.20140411-gda795e0
     build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV DNS Feeds Quvi TDFA CryptoHash
     key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
     remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier hook external
     local repository version: 5
     supported repository version: 5
     upgrade supported from repository versions: 2 3 4

### Please provide any additional information below.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 1"
 date="2014-05-16T16:42:41Z"
 content="""
This is good spotting of a git configuration that git-annex does not support.

However, I doubt that even if I made it use `GIT_SSH`, it would be useful. git-annex uses several features that are probably unique to openssh, including connection caching. While you could disable annex.sshcaching and perhaps get a different ssh to work, it would be much slower.
"""]]


code:
[[!comment format=mdwn
 username="divB"
 ip="171.67.172.81"
 subject="comment 2"
 date="2014-05-17T23:58:19Z"
 content="""
Hi Joey,

Thanks for your answer. In my opinion, this would be an important requirement for various reasons:

1.) It is very confusing and results in unpredictable errors. I spent days in finding out what caused all the weird stuff that happened. Even if it is not supported, an error message or at least warning should be issued.

2.) At least in Windows, plink.exe is the quasi-standard SSH client. All SW I am aware of supports at least plink.exe as alternative to openssh (SVN, git, unison, ...). Even within cygwin I often use plink for X11 forwarding etc. If features like SSH caching do not work with that it's totally fine.

3.) Even for a unix environment, it is critical to be able to use a wrapper (or at least to configure SSH parameters). In my opinion, this should and must work consistently (git, git-annex and rsync). For example what if I have a dedicated public key for a repository and to not want to use %HOME%\.ssh\id_rsa ?
For unison, I use a wrapper my_ssh.cmd which wraps specialized parameters (in particular SSH key, port) with plink.exe to ssh.exe's interface. Similarly, I might be interested in disabling agent functionality and use GSSAPI etc. etc.

A little bit OT now: I already wondered if and how inefficient git-annex is in this regard. For example, if I sync content, it seems that ssh opens a new connection for each file! (at least each file results in a signing request in my agent). This happens even if I use ssh.exe. Is there anything wrong?

Thanks

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5WyknJirJJridJjiPNgrlYxGG9xrZBvA"
 nickname="Daniel"
 subject="comment 3"
 date="2014-06-06T10:49:27Z"
 content="""
Agreed. I'd rather lose connection caching than have to maintain both putty keys and openssh keys on my windows machines.
"""]]


code:
### Please describe the problem.
git annex get fileName-  should perform a hash check on the file content before adding to the local repository

### What steps will reproduce the problem?
Two scenarios:
1) Malicious user and owner of repository being pulled from can edit his/her local .git/annex/objects directory
to alter the file content. For src code, this could be to insert a bug, insert a backdoor, or for example
to replace an image file artifact for a website, with a pornographic image.
The user pulling the file content with "git annex get fileName"  might not be aware of the file contents
until they actually examine the file or perform an fsck or commit locally.
In the meantime a kiddy porn image could be sitting in their repository or a src code backdoor can get incorporated and deployed etc.
2) a file could also simply get corrupted during download. An inherent hash check during the 'annex get' would
point out the problem immediately.
To reproduce:  create repoNum1, and clone it to create repoNum2. manual edit/replace content in repoNum1/.git/annex/objects/...
then perform a 'git annex get <fileName>' from repoNum2 on the file that has been manipulated

### What version of git-annex are you using? On what operating system?
3.2012112ubuntu2 on running linux mint

### Please provide any additional information below.

Aside: Thanks Joey - this is fantastic work you are doing. You have really improved git. The ability to checkout
an entire tree - but selectively get only the content actually needed is a real killer feature.
Kudos and again many many thanks
M.

# End of transcript or log.
"""]]

> [[duplicate|done]] of [[checksum_verification_on_transfer]] --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 1"
 date="2013-12-15T19:38:48Z"
 content="""
If you don't trust a remote repository, then you should either

a) Not use that repository at all, because its malicious owner could put any evil file he wants in it with an entirely correct hash.

b) Make it a gcrypt remote so all content stored on it is encrypted. Decrypting it will include validating that you get out what you originally put in.

So these scenarios are not good arguments for validating every file after it's downloaded.

If it were possible to do a rolling checksum as part of the download, rather than needing to pull the entire file back off disk and checksum it, I'd do so. But it's generally not; for example when git-annex is downloading a file using rsync it may resume part way through a previous interrupted download, and rsync is storing the file to disk, not streaming it to git-annex.
"""]]


code:
Add a `git annex hide $file` that behaves like drop, checking counter info
and updating location log to say the current repo no longer has a file --
but does not actually remove the content.

Then `git annex unused` can be used to clean it up later. And in the
meantime, it's still locally accessible. This can be useful if you're
planning to need to free up space later, but want to hold onto the content
for a while. Possibly you'll be disconnected later, so it's easier to push
out that intent now.

--

TODO:

* Make 100% sure this is safe. Drop, etc should never check content files
  are present on other repos if the location log doesn't say the repo
  has the content.

* What will `git annex get` do if it's asked to get a file that has been
  hidden?

> Unless I am missing something: Make sure the data is correct (for SHA1 or other tracking) and restore locally. If that's not the case, delete and restore from remote. -- RichiH

----

Is 'unused' a good name? 'clean' and 'autoclean' would make more sense, imo. 'clean' deletes everything, whereas an optional 'autoclean' could try to be smart based on disk usage and/or SHA1, etc. -- RichiH

> Nah, `git annex unused/dropunused` already exist.  --[[Joey]]

>> OK, in that case forget what I said. No idea about your internal policy, but feel free to delete this part of the page, then. -- RichiH


code:
It would be great to be able to use the pubDate of the entries with the --template option of importfeed.

Text.Feed.Query has a getItemPublishDate (and a getFeedPubDate, if we want some kind of ${feeddate}).

The best would be to allow a reformating of the date(s) with (for example) %Y-%m-%D


code:
[[!comment format=mdwn
 username="gueux"
 ip="2a01:240:fe6d:0:7986:3659:a8bd:64f1"
 subject="syntax"
 date="2013-09-12T14:05:16Z"
 content="""
use \"itemdate\" and \"feeddate\" as names?

use ${itemdate=%Y-%m-%D} syntax option?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.2.134"
 subject="comment 2"
 date="2013-09-13T19:53:52Z"
 content="""
getItemPublishDate returns a String, which can contain any of several date formats. Deferred until the feed library has something more sane.
Upstream bug: <https://github.com/sof/feed/issues/6>

As for how to format the date in the feed, I would be ok with having itemdate (YYYYMMDD), itemyear (YYYY), itemmonth (MM) and itemday (DD). Full date formatting seems like overkill here.
"""]]


code:
SpiderOak is a great backup service and many of us have unlimited accounts with them since World Backup Day.  That makes SpiderOak a very interesting candidate for use as an archive or backup node.  I can think of only two ways to go about this:

1. Designate one of your computers as an archive/backup and use spideroak independantly to sync that archive.  This is very unattractive, since it makes the spideroak backup completely unknown to git-annex.

2. Integrate the SpiderOak CLI tool somehow as a remote.  I don't know to what extent this would be possible, but if it were, that'd be awesome.  And a lot of work, presumably.

Bonus option:

3. Can the SpiderOak API be useful? https://spideroak.com/faq/questions/37/how_do_i_use_the_spideroak_web_api/


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="216.145.95.162"
 subject="comment 1"
 date="2014-05-19T15:05:49Z"
 content="""
If spideroak has a CLI tool that can get/put/delete individual files, it should be quite easy to use [[the_external_special_remote|special_remotes/external]] to support it. The demo shell script could be used as a starting place.

I built that so that others can easily write special remotes, and so unless spideroak's CLI is free software, I don't anticipate working on this myself.
"""]]


code:
I don't want files that I dropped to immediately disappear from my local or all of my remotes repos on the next sync. Especially in situations where changes to the git-annex repo get automatically and immediately replicated to remote repos, I want a configurable "grace" period before files in .git/annex/objects get really deleted.

This has similarities to the "trash" on a desktop. It might also be nice to

* configure a maximum amount of space of the "trash"
* have a way to see the contents of the trash to easily recover deleted files

Maybe it would make sense to just move dropped files to the desktops trash? "git annex trash" as an alternative to drop?


code:
Just started today with git-annex and it looks great replacement for proprietary syncing solutions (well in allot of aspects it's much better than proprietary solutions) but I do believe desktop and email notifications are must have features.

I think these services would be nice to have: growl, libnotify, email, twitter (publicly sharing with a group or repository stored on public server for users to download).


code:
hi

recently i have installed openwrt on my mikrotik routerboard. i am verry suprised how well it works. it lacks git-annex package. openwrt has git and i can install it.

how can i build one on a mips arch ?
is it possible to build multiple architecture standalone binaries ?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="206.74.132.139"
 subject="comment 1"
 date="2014-02-06T17:26:58Z"
 content="""
I would be quite happy if someone took care of adding git-annex to openwrt.

I don't have time to personally handle packaging for different linux distributions myself. What I could do is add mips builds of git-annex to the existing standalone linux builds. These would need to be built the same way the arm builds are done, using a Debian chroot and qemu to run tools from it. This is rather a lot of work for me to set up, and I don't know if I'd have to do it for both little and big endian mips.

Also, it seems that Debian does not currently have a working haskell toolchain for mips. Which may well mean that ghc is not in a working state on mips at all.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkNE-H4vEcbcGndxq5daT8qUb7yIf7r1OE"
 nickname="Łukasz"
 subject="comment 2"
 date="2014-02-11T21:05:00Z"
 content="""
if debian does not have working toolchain for mips then there is no way to port it into mips machines.
am i correct ?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.172"
 subject="comment 3"
 date="2014-02-11T21:39:34Z"
 content="""
Actually, debian stable does still have ghc building for mips/mipsel. It's just newer versions that have failed to build and nobody has fixed it yet.
"""]]


code:
Typically `git-annex merge` is fast, but it could still be sped up.

`git-annex merge` runs `git-hash-object` once per file that needs to be
merged. Elsewhere in git-annex, `git-hash-object` is used in a faster mode,
reading files from disk via `--stdin-paths`. But here, the data is not
in raw files on disk, and I doubt writing them is the best approach.
Instead, I'd like a way to stream multiple objects into git using stdin.
Sometime, should look at either extending git-hash-object to support that,
or possibly look at using git-fast-import instead.

---

`git-annex merge` also runs `git show` once per file that needs to be
merged. This could be reduced to a single call to `git-cat-file --batch`,
There is already a Git.CatFile library that can do this easily. --[[Joey]]

> This is now done, part above remains todo. --[[Joey]]

---

Merging used to use memory proportional to the size of the diff. It now
streams data, running in constant space. This probably sped it up a lot,
as there's much less allocation and GC action. --[[Joey]]


code:
One of my reasons for using haskell was that it provides the possibility of
some parallell processing. Although since git-annex hits the filesystem
heavily and mostly runs other git commands, maybe not a whole lot.

Anyway, each git-annex command is broken down into a series of independant
actions, which has some potential for parallelism.

Each action has 3 distinct phases, basically "check", "perform", and
"cleanup". The perform actions are probably parellizable; the cleanup may be
(but not if it has to run git commands to stage state; it can queue
commands though); the check should be easily parallelizable, although they
may access the disk or run minor git query commands, so would probably not
want to run too many of them at once.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkptNW1PzrVjYlJWP_9e499uH0mjnBV6GQ"
 nickname="Christian"
 subject="comment 1"
 date="2011-04-08T12:41:43Z"
 content="""
I also think, that fetching keys via rsync can be done by one rsync process, when the keys are fetched from one host. This would avoid establishing a new TCP connection for every file.
"""]]


code:
[[!comment format=mdwn
 username="http://ertai.myopenid.com/"
 nickname="npouillard"
 subject="comment 2"
 date="2011-05-20T20:14:15Z"
 content="""
I agree with Christian.

One should first make a better use of connections to remotes before exploring parallel possibilities. One should pipeline the requests and answers.

Of course this could be implemented using parallel&concurrency features of Haskell to do this.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.90"
 subject="comment 3"
 date="2013-07-17T19:59:50Z"
 content="""
Note that git-annex now uses locks to communicate among multiple processes, so it's now possible to eg run two `git annex get` processes, and one will skip over the file the other is downloading and go on to the next file, and so on.

This is an especially nice speedup when downloading encrypted data, since the decryption of one file will tend to happen while the other process is downloading the next file (assuming files of approximately the same size, and that decryption takes approxiately as long as downloading).

The only thing preventing this being done by threads in one process, enabled by a -jN option, is that the output would be a jumbled mess.
"""]]


code:
Here's yet another weird use case. I have a ext3 external drive for storing media that i brought to the office. Naturally, all files on the drive are owned by my desktop user (uid = 1000). In the office, my userid is different (say 1001), so git-annex doesn't see the removable drive at all (which is a little confusing). But even if i try to add the repo on the drive as an external repo, it says it can't write to it (which is true).

I would expect it to at least be able to leech the files off of it.

Otherwise, I would welcome advice on how to fix this problem without doing a `sudo chown -R` every time i plug this drive somewhere ... --[[anarcat]]

> Workaround: `sudo setfacl -R -m u:anarcat:rwx /media/foo/annex`


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 1"
 date="2014-05-16T18:10:28Z"
 content="""
The assistant ignores drives that it cannot write to. This is a very good heuristic to avoid listing every special device in linux, OSX, etc as possible removable drives. I don't think it makes sense for the assistant to try to handle this use case.

Is there some problem using git-annex at the command line with a remote that is read-only? I don't see any problem in my tests.

<pre>
joey@darkstar:~/tmp/r>git annex drop
drop foo ok
(Recording state in git...)
joey@darkstar:~/tmp/r>ls -ld ../readonly
drwxr-xr-x 3 root root 4096 May 16 14:08 ../readonly/
joey@darkstar:~/tmp/r>git annex get --from readonly
get foo (from readonly...) ok
(Recording state in git...)
</pre>
"""]]


code:
Currently, `git annex status` only shows the size of 1 copy of each file.
If numcopies is being used for redundancy, much more disk can actually be
in use than status shows.

One idea:

	known annex size: 2 terabytes (plus 4 terabytes of redundant copies)

But, to get that number, it would have to walk every location log,
counting how many copies currently exist of each file. That would make
status a lot slower than it is.

One option is to just put it at the end of the status:

	redundancy: 300% (4 terabytes of copies)

And ctrl-c if it's taking too long.

Hmm, fsck looks at that same info. Maybe it could cache the redundancy
level it discovers? Since fsck can be run incrementally, it would be tricky
to get an overall number. And the number would tend to be stale, but
then again it might also be nice if status shows how long ago the last fsck
was.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 1"
 date="2012-10-30T08:09:13Z"
 content="""
I like the idea of using fsck as a pre-run for status.

Basically, it's the same as `updatedb` and `locate`; locate will warn the user if it considers its cache to be too old, as well.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.220"
 subject="comment 2"
 date="2013-09-25T18:03:55Z"
 content="""
`git annex status .` or otherwise running it with a directory has recently started walking all the location logs for all files in the directory in order to display variance from configured numcopies. It would be easy to add a redundancy counter to that.

It would slow down the global status when not passed a directory to add redundancy info there. Maybe local is enough?
"""]]


code:
We have preferred content, which is advisory, and numcopies, which is
enforced (except by `git annex move`). What is missing is an expression
like preferred content, which is enforced. So, required content.

For example, I might want a repository that is required to contain
`*.jpeg`. This would make get --auto get it (it's implicitly part of the
preferred content), and would make drop refuse to drop it.

> I've implemented the basic required content. Currently only configurable
> via `vicfg`, because I don't think a lot of people are going to want to
> use it.
>
> Note that I did not yet add the active verification discussed below.
> So if required content is set to `not inallgroup=backup`, or
> `not copies=10`, trying to drop a file will not go off and prove
> that there are 10 copies or that the file is in every repository in
> the backup group. It will assume that the location log is accurate
> and go by that.
>
> I think this is enough to cover Richard's case, at least.
> In his example, A B and C are in group anchor and have required
> content set to `include=*`, and D E F have it set to
> `not inallgroup=anchor`. --[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 1"
 date="2014-03-10T17:15:54Z"
 content="""
To give a specific example of what I want to do:

I have a bunch of repos

* A, Machine1 -- anchor repository and sometimes used to add new content
* B, Machine2 -- anchor repository, offsite
* C, External1 -- anchor repository, stored off-site
* D, Laptop -- transient repo to carry data around and to add new content
* E, USB1 -- transient, travel backups
* F, USB2 -- transient, travel backups

A-C should get _all_ data. A-F trust A-C to always retain all data, online checks for availability are not needed because A-C run frequent fsck. Ideally, `git annex drop` should be no-op in A-C.

D-F should retain data as long as it's not been saved in _all_ of A-C. As soon as A-C have copies, `git annex drop` in D-F should drop that content; before that, they _must not_ drop said content.

Richard
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.154"
 subject="partial implementation plan"
 date="2014-03-19T16:37:05Z"
 content="""
* When there is a required content expression, OR it with the preferred content expression. This will make the assistant, --auto etc want to get required content.
* When checking if something can be dropped, check the required content expression. Only Command.Drop does this so nicely centralized in one place.
* When checking required content expression for drop, must do active verification of terminals that relate to other copies, for the same reason the numcopies check when dropping does active validation. This includes `copies=`, `lackingcopies=`, `approxlackingcopies=`, `inallgroup=`.

The last is where the complication comes in. Seems to need a cache of places the key was just now verified to be present, which can be used to avoid unnecessary redundant active verification (for example \"not (copies=2 and inallgroup=bar)\" would otherwise redundantly check some remotes). The numcopies checking code should use the same cache too.

* Finally, if the required content cannot be satisfied, it would be nice to know which terminal failed in order to show the user a useful error message. The current Matcher does not provide a way to do that though. Or could just say, \"unable to satisfy required content: $expression\"
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 3"
 date="2014-05-18T06:49:35Z"
 content="""
    22:34:10 < joeyh> required content is implemented, though active verification is not
"""]]


code:
Resuming interrupted uploads to encrypted special remotes is not currently
possible, because gpg does not produce consistent output. Special remotes
that could support resuming include rsync and glacier.

Without consistent output, git-annex would need to locally cache the encrypted
file, and reuse that cache when resuming an upload. This would make
encrypted uploads more expensive in terms of both file IO and disk space
used.

[It would be possible to write to the cache at the same time the special
remote is being fed data, and if the special remote upload fails, continue
writing the rest of the file. That would avoid half the overhead, since
the file would not need to be read from, just written to. (Although OS
caching may accomplish the same thing.)]

Also, `git annex unused` would need to show temp files for uploads,
the same as it currently shows temp files for downloads, and users would
sometimes need to manually dropunused old uploads, that never completed.

The question, then, is whether resuming uploads is useful enough to add
this overhead and user-visible complexity.
--[[Joey]]


code:
[[!comment format=mdwn
 username="http://phil.0x539.de/"
 nickname="Philipp Kern"
 subject="comment 1"
 date="2012-12-28T23:23:29Z"
 content="""
Doesn't the encryption part already write an encrypted version completely to disk before starting to upload? (At least with the rsync remote?) So the disk space and I/O usage is already there. (Except that it's probably a temporary file that's deleted after it was created, so that it's not kept around by the filesystem when certain destructive events strike.)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="comment 2"
 date="2012-12-29T08:00:56Z"
 content="""
Being able to resume transfers of encrypted files would absolutely be useful!  Disk space is cheap, but bandwidth is not.
"""]]


code:
git's core.SharedRepository is supported by git-annex, but only
with the group/all/world/everybody settings. core.SharedRepository=0644
etc is not supported.

There's no insormountable reason why not, Joey just hates umask mode math
stuff and nobody has sent a patch. Note that Annex.Content.freezeContent
should remove the write bit from files, no matter what.


code:
git-annex should use smudge/clean filters.

----

Update: Currently, this does not look likely to work. In particular,
the clean filter needs to consume all stdin from git, which consists of the
entire content of the file. It cannot optimise by directly accessing
the file in the repository, because git may be cleaning a different
version of the file during a merge.

So every `git status` would need to read the entire content of all
available files, and checksum them, which is too expensive.

> Update from GitTogether: Peff thinks a new interface could be added to
> git to handle this sort of case in an efficient way.. just needs someone
> to do the work. --[[Joey]]

----

The clean filter is run when files are staged for commit. So a user could copy
any file into the annex, git add it, and git-annex's clean filter causes
the file's key to be staged, while its value is added to the annex.

The smudge filter is run when files are checked out. Since git annex
repos have partial content, this would not git annex get the file content.
Instead, if the content is not currently available, it would need to do
something like return empty file content. (Sadly, it cannot create a
symlink, as git still wants to write the file afterwards.)

So the nice current behavior of unavailable files being clearly missing due
to dangling symlinks, would be lost when using smudge/clean filters.
(Contact git developers to get an interface to do this?)

Instead, we get the nice behavior of not having to remeber to `git annex
add` files, and just being able to use `git add` or `git commit -a`,
and have it use git-annex when .gitattributes says to. Also, annexed
files can be directly modified without having to `git annex unlock`.

### design

In .gitattributes, the user would put something like "* filter=git-annex".
This way they could control which files are annexed vs added normally.

(git-annex could have further controls to allow eg, passing small files
through to regular processing. At least .gitattributes is a special case,
it should never be annexed...)

For files not configured this way, git-annex could continue to use
its symlink method -- this would preserve backwards compatability,
and even allow mixing the two methods in a repo as desired.

To find files in the repository that are annexed, git-annex would do
`ls-files` as now, but would check if found files have the appropriate
filter, rather than the current symlink checks. To determine the key
of a file, rather than reading its symlink, git-annex would need to
look up the git blob associated with the file -- this can be done
efficiently using the existing code in `Branch.catFile`.

The clean filter would inject the file's content into the annex, and hard
link from the annex to the file. Avoiding duplication of data.

The smudge filter can't do that, so to avoid duplication of data, it
might always create an empty file. To get the content, `git annex get`
could be used (which would hard link it). A `post-checkout` hook might
be used to set up hard links for all currently available content.

#### clean

The trick is doing it efficiently. Since git a2b665d, v1.7.4.1,
something like this works to provide a filename to the clean script:

	git config --global filter.huge.clean huge-clean %f

This could avoid it needing to read all the current file content from stdin
when doing eg, a git status or git commit. Instead it is passed the
filename that git is operating on, in the working directory.
(Update: No, doesn't work; git may be cleaning a different file content
than is currently on disk, and git requires all stdin be consumed too.)

So, WORM could just look at that file and easily tell if it is one
it already knows (same mtime and size). If so, it can short-circuit and
do nothing, file content is already cached.

SHA1 has a harder job. Would not want to re-sha1 the file every time,
probably. So it'd need a local cache of file stat info, mapped to known
objects.

But: Even with %f, git actually passes the full file content to the clean
filter, and if it fails to consume it all, it will crash (may only happen
if the file is larger than some chunk size; tried with 500 mb file and
saw a SIGPIPE.) This means unnecessary works needs to be done,
and it slows down *everything*, from `git status` to `git commit`.
**showstopper** I have sent a patch to the git mailing list to address
this. <http://marc.info/?l=git&m=131465033512157&w=2> (Update: apparently
can't be fixed.)

#### smudge

The smudge script can also be provided a filename with %f, but it
cannot directly write to the file or git gets unhappy.

### dealing with partial content availability

The smudge filter cannot be allowed to fail, that leaves the tree and
index in a weird state. So if a file's content is requested by calling
the smudge filter, the trick is to instead provide dummy content,
indicating it is not available (and perhaps saying to run "git-annex get").

Then, in the clean filter, it has to detect that it's cleaning a file
with that dummy content, and make sure to provide the same identifier as
it would if the file content was there.

I've a demo implementation of this technique in the scripts below.

----

### test files

huge-smudge:

#!/bin/sh
read f
file="$1"
echo "smudging $f" >&2
if [ -e ~/$f ]; then
	cat ~/$f # possibly expensive copy here
else
	echo "$f not available"
fi

huge-clean:

#!/bin/sh
file="$1"
cat >/tmp/file
# in real life, this should be done more efficiently, not trying to read
# the whole file content!
if grep -q 'not available' /tmp/file; then
	awk '{print $1}' /tmp/file # provide what we would if the content were avail!
	exit 0
fi
echo "cleaning $file" >&2
# XXX store file content here
echo $file

.gitattributes:

*.huge filter=huge

in .git/config:

[filter "huge"]
        clean = huge-clean %f
        smudge = huge-smudge %f


code:
[[!comment format=mdwn
 username="http://christian.amsuess.com/chrysn"
 nickname="chrysn"
 subject="git-add instead of git-annex-add"
 date="2011-02-26T21:43:21Z"
 content="""
would, with these modifications in place, there still be a way to *really* git-add a file? (my main repository contains both normal git and git-annex files.)
"""]]


code:
[[!comment format=mdwn
 username="http://dieter-be.myopenid.com/"
 nickname="dieter"
 subject="symlinks"
 date="2011-04-03T20:30:21Z"
 content="""
> (Sadly, it cannot create a symlink, as git still wants to write the file afterwards.
> So the nice current behavior of unavailable files being clearly missing due to dangling symlinks, would be lost when using smudge/clean filters. (Contact git developers to get an interface to do this?)

Have you checked what the smudge filter sees when the input is a symlink? Because git supports tracking symlinks, so it should also support pushing symlinks through a smudge filter, right?
Either way: yes, contact the git devs, one can only ask and hope.  And if you can demonstrate the awesomeness of git-annex they might get more 1interested :)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn1QhtPvsRBV7pfaDW_ZTPFv_ZIxSzQ8Rg"
 nickname="Paul Léo"
 subject="comment 3"
 date="2013-11-13T20:41:52Z"
 content="""
> SHA1 has a harder job. Would not want to re-sha1 the file every time, probably. So it'd need a local cache of file stat info, mapped to known objects.

I think that is not true? If gits wants the file to be cleaned, it thinks that the file was changed. So you would have to SHA1 it anyway if you don't want rely on WORM (which git already does in the first step anyway).
"""]]


code:
ssh:// remotes are not special remotes. Perhaps it would be useful to have
a special remote that wraps a ssh:// remote? This would allow setting up a
ssh:// remote that can be enabled using the webapp's normal UI for enabling
special remotes.

Enabling such a special remote would just make a regular git remote, so
there would be no need to implement the methods to get/put data. (Although
it might need to provide stubs to appease the compiler.)

> Above is done. The command line interface in initremote and enableremote
> is not too easy or perhaps useful, but it works great in the webapp.
> --[[Joey]]

It could optionally embed the ssh private key into the git-annex branch as
a credential, for when you want anyone who has access to the git repo to be
able to use the (locked-down) git-annex-shell on that server.

> Leaving this todo open for this ssh private key embedcreds part.
> I think it makes sense to do, but it it probably not too easy.
> ([[webapp_ssh_setup_should_work_with_locked_down_git-annex-shell_account]]
> needs to be fixed first). --[[Joey]]

[[!meta title="remember ssh remote including optionally ssh key"]]

----

I am on the fence about whether this would be useful, and would appreciate
use cases.

One use case I was thinking about was a LAN with a central server, with a
shared account with a git-annex repository on it. But then I realized this
wouldn't really help set up git-annex in that situation, most of the time,
because new clients would have the central server added as their first
remote.

(It would help if one client paired with another new client, but
that is unncessarily round-about most of the time.)

It might help in a more complex situation, where the LAN is not the whole
network an a client might come onto the LAN already knowing about the
central server there. --[[Joey]]

A very compelling use case is switching from XMPP to a ssh server,
and wanting to make it easy for users. --[[Joey]]


code:
I am just asking myself, is it stupid to think that streaming in git annex would be a good idea and wouldnt it be totaly easy to implement?

Ok just tried to link to files over ssh, it creates a link but you cant open with it its content ^^

But at least the files you have access over some filesystem as example samba/sshfs or just a other directory or usb-drive you could stream instead of "get"

you could add another mode like direct and indirect, like symbolic-links or something like that?

Sadly linux is to stupid to allow direct ssh links ( thats maybe one of the biggest features hurd has over linux  ) but you could mount with sshfs readonly (checking first if sshfs is installed) to mount the files there and then map the links there.

ok I am not so shure how hard it would be and how much bug potentials it creates, but it would be great I guess.

git annex is a bit like a telephone book, where you get a list of where the targets are. So using it to call the persons so that they drive to you to talk with you is nice. But I think the better feature would be if you just talk with the guy over the telephone directly bevore he comes to you (streaming)

I mean you did one great thing, you did make cloudy thing peer to peer, like git is targeted too but for smaller files, yes there are may use cases without this feature, but I would be really glad if it could do that too, if I give annex 5 locations on other pcs usb-sticks etc, I find it stupid to additionaly do setup all this sources again a second time for streaming, and then I have maybe even 2 different file names because you map stuff in git.

So sorry its late here, I am a bit tired so I maybe dont know what I am talking right now, my english isnt the best, too, but I think it would be a great feature.

I mean on your setup, with slow internet, you maybe always make a get command, but here, if I link to youtube, I have no problem to stream it, or even on internal network between my pcs I have gb-lan, I start directly movies streaming, I would only use get, in rare cases where I need them on a train, the normal thing is to stream stuff.

So I have to go sleep now

bye


code:
Did not know of this when I wrote S3 support. Ability to resume large
uploads would be good.

<http://aws.typepad.com/aws/2010/11/amazon-s3-multipart-upload.html>

Also allows supporting files > 5 gb, a S3 limit I was not aware of.

NB: It would work just as well to split the object and upload the N parts
to S3, but not bother with S3's paperwork to rejoin them into one object.
Only reasons not to do that are a) backwards compatability with
the existing S3 remote and b) this would not allow accessing the content
in S3 w/o using git-annex, which could be useful in some scenarios.

--[[Joey]]


code:
As discussed on debconf, I have the following use case:

* I have a dump remote, a folder on my webserver where files are uploaded through the web app. I don't have git on the webserver, just a plain folder.
* I have git-annex repo on a development server. The development server polls the webserver (ssh/ftp) once in an hour and synchronizes the state of the local git-annex repo with the state found on the webserver and commits that.
* This is not meant to be backup facility. I just want to be able to have a state on my development machine that is very likely to the state on the webserver.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 1"
 date="2013-08-13T21:44:17Z"
 content="""
We had a conversation about this IRL. At the time, I thought I understood what you wanted. Reading the above, I am not so sure.

What I thought you wanted was something like `git annex mirror --from remote`, which would, for each object known to git-annex that the location log said was present on the remote, make sure that the local repo had the object too, and for each object that the location log said was not present on the remote, drop it from the local repo (if numcopies etc allowed).

`git annex mirror --to remote` could also be used as the complement of the above.

If that's not the sort of thing you meant, let me know.
"""]]


code:
[[!comment format=mdwn
 username="http://thkoch2001.myopenid.com/"
 nickname="thkoch"
 subject="pseudocode"
 date="2013-08-14T04:58:22Z"
 content="""
lets say my local annex is in direct mode, then the following might already do what I want:

cd $LOCAL_ANNEX
rsync --recursive --delete $REMOTE .
git annex add && git commit

It would however be nice if I could do the same with an annex in indirect mode or even a bare annex.
"""]]