code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlutNrg7ajiNAfi5gqJfD8crv0aimZa27k"
 nickname="Chuck"
 subject="I'm having this problem too"
 date="2013-07-21T19:14:21Z"
 content="""
I'm a git-annex newbie looking for Dropbox-like functionality.  I've got git-annex and the webapp running on my desktop and laptop, and I believe I've got a Jabber account working since Kopete seems happy with my account info.  But when I enter the correct Gmail UID and password I get the above error.

I'm running the version of git-annex that's current in the standard Ubuntu repos (I'm actually running Kubuntu if that matters), and that version is 20121112ubuntu2.  If this is an issue that was fixed in the 20121126 release, how would I get that?  Doing the standard install gets me the version I have now...
"""]]


code:
Hi,

Some time ago I asked [[here|git_annex_copy_--fast_--to_blah_much_slower_than_--from_blah]]
about possible improvements in git `copy --fast --to`, since it was painfully slow
on moderately large repos.

Now I found a way to make it much faster for my particular use case, by
accessing some annex internals. And I realized that maybe commands like `git
annex find --in=repo` do not batch queries to the location log. This is based on
the following timings, on a new repo (only a few commits) and about 30k files.

    > time git annex find --in=skynet > /dev/null

    real	0m55.838s
    user	0m30.000s
    sys	0m1.583s

    > time git ls-tree -r git-annex | cut -d ' ' -f 3 | cut -f 1 | git cat-file --batch > /dev/null

    real	0m0.334s
    user	0m0.517s
    sys	0m0.030s

Those numbers are on linux (with an already warm file cache) and an ext4 filesystem on a SSD.

The second command above is feeding a list of objects to a single `git cat-file`
process that cats them all to stdout, preceeding every file dump by the object
being cat-ed. It is a trivial matter to parse this output and use it for
whatever annex needs.

Above I wrote a `git ls-tree` on the git-annex branch for simplicity, but we could
just as well do a `ls-tree | ... | git cat-file` on HEAD to get the keys for the
annexed files matching some path and then feed those keys to a cat-file on
the git-annex branch. And this still would be an order of magnitude faster than
what currently annex seems to do.

I'm assuming the bottleneck is in that annex does not batch the `cat-file`, as the rest of logic needed for a find will be fast. Is that right?

Now, if the queries to the location log for `copy --to` and `find` could be batched this way, the
performance of several useful things to do, like checking how many annexed files
we are missing, would be bastly improved. Hell, I could even put that number on
the command line prompt!

I'm not yet very fluent in Haskell, but I'm willing to help if this is something that makes sense and can be done.

repo_name: kakafarm/git

license: unknown


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 1"
 date="2013-11-06T22:21:33Z"
 content="""
git-annex uses cat-file --batch, yes. You can verify this with --debug. Or you can read Annex/CatFile.hs and Git/CatFile.hs

git-annex has to ensure that the git-annex branch is up-to-date and that any info synced into the repository is merged into it. This can require several calls to git log. Your command does not do that. git-annex find also runs `git ls-files --cached`, which has to examine the state of the index and of files on disk, in order to only show files that are in the working tree. Your command also omits that.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkkyBDsfOB7JZvPZ4a8F3rwv0wk6Nb9n48"
 nickname="Abdó"
 subject="comment 2"
 date="2013-11-06T23:14:03Z"
 content="""
Ok, then I don't understand where annex spends its time. git annex takes 55
seconds! vs less than a second for a batched query on all the keys in the
location log. Checking that branches are in sync, or traversing the working dir
shouldn't amount the extra 54 seconds! At least not on a recently synced repo
with up to date index and clean working dir.

> git-annex has to ensure that the git-annex branch is up-to-date and that any info synced into the repository is merged into it. This can require several calls to git log

Ok, I understand that. Checking that should be typically fast though, isn't it? On a repo that has just been synced, it doesn't need to go very far on the log.

> git-annex find also runs git ls-files --cached, which has to examine the state of the index and of files on disk, in order to only show files that are in the working tree

I understand that too. For my particular use case, I know I do the `git copy` when the
repo is in sync and the working dir has no uncommited changes. So I use HEAD to retrieve the keys for
the files in the working tree. I do something like that:

    time git ls-tree -r HEAD | grep -e '^120000' | cut -d ' ' -f 3 | cut -f 1 | git cat-file --batch > /dev/null

    real	0m0.178s
    user	0m0.277s
    sys 	0m0.037s

That plus some fast parsing of the output gets the list of keys for the files in HEAD in less than a second. Where do the 54 extra seconds hide, then?

Mm... how does annex retrieve the keys for files in the working tree? Does it follow
the actual symlinks on the filesystem? I can believe that following 30k symlinks may be slow (although not 55 second slow).

Sorry for being so insistent on this... It is just that I do think the same can be done much faster, and such an improvement in performance would be very interesting, not only for me.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 3"
 date="2013-11-07T18:12:09Z"
 content="""
It's hard to say until some profiling has actually been done. Comparing apples and oranges, or perhaps better to say, orange blossoms and oranges, is not very useful.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkkyBDsfOB7JZvPZ4a8F3rwv0wk6Nb9n48"
 nickname="Abdó"
 subject="comment 4"
 date="2013-11-07T18:40:27Z"
 content="""
Ok, thanks.

I understand that annex is more sophisticated than what I'm proposing. I'm also
not comparing apples to oranges, what I'm saying is this: All the
logic and checking and git calls that I thought annex needs to do, take
much much less time than those 55 seconds. So either I'm missing something big (about 98% of it in execution time), or
annex is doing something very inefficient. That's all I'm saying, and I was
hoping that you could clarify that, since you already know the code.

Also I'm sorry for being so annoying. If annex were writen in C, I would either have understood those 55 seconds or sent you a patch making annex faster. Being in
haskell means that it will take me considerably more time and effort. But if I manage to
improve things and if it is ok with you I'll send you patches... eventually.
"""]]


code:
I’m currently trying to set up a sync which involves an SQLite3 database file that should always be in the same state on all systems at all times (since there is no readily available way of merging the data). Basically, I’m looking for a practical way that gives me some help in making sure the files never drift apart between my remotes. Since I’m forgetful and might forget syncing the repo before going home from the office, I was wondering whether there might be a good way to assist me in this. Has anyone had some good ideas in this direction, or is there a canonical solution?

It occurred to me one way would be to instate a hook somewhere that links the database to /dev/null in all other remotes (so the software will fail to work if I start it without having synced), but it seems tricky. I guess would have to involve per-remote branches, which will be hard to do, since I need to use direct mode.

A less complex method might be forcing a sync before shutting down the system resp. when booting up.

Maybe other folks have had ideas for a practical, yet robust solution.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.55"
 subject="comment 1"
 date="2014-07-08T17:31:53Z"
 content="""
This is something git's distributed nature preludes. You might frankly be better with one of the centralized VCSs that supports locking in this specific case. Or, you can do some kind of \"lock server\" that the clients contact, layering a centralized approach on top of git.
"""]]


code:
[[!comment format=mdwn
 username="zardoz"
 ip="134.147.14.84"
 subject="comment 2"
 date="2014-07-09T07:07:27Z"
 content="""
Yea, it’s basically just the isolated one file that needs «locking», so one could just try a different route. The idea with keeping state on a server sounds like a pretty good idea. I could just «echo» via SSH to my webserver some info on who has the file locked; then I could wrap the program that uses the SQLite3 DB in a script that checks whether it’s safe.
"""]]


code:
I need some help understanding here.

We use AeroFS at work to sync the normal user files across computers. I'll quite likely be replacing that with git annex as soon as Windows port is stable enough.

How it works is that you create a repo, and share it with one or more users. Then AeroFS discovers what other repos are online and if they're on the local network, and syncs from wherever is most convenient.

This sounds a little like pairing, but with pairing you need to arrange more than 2 devices in a "star" or "chain". This is fine for my own devices, but it becomes brittle if you have lots of devices syncing from each other, which may or may not be online at any given time. The only way around it seems to be to pair each device with each other device, with the labor rising exponentially.

Is this correct? Best compromise for my use case seems to be to just do a star setup and pair everything with an always-on machine. Do you agree?

Thanks! Carlo


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="64.134.31.139"
 subject="comment 1"
 date="2013-10-22T17:41:58Z"
 content="""
You're basically correct. Typically if you have a central machine, it's a server, so at that point you're not pairing, but are just adding a ssh server, or some other type of remote.

I think that systems like AeroFS are pretty neat, but my goal is not really to build another one of those. I'd rather make git-annex be able to use such things as [[special_remotes]].
"""]]


code:
I have a remote repository that I have yet to add as a remote into git-annex, but which already contains data that I want git-annex to manage. I already know what the SHA256 hashes and sizes of all the files are, and I can arrange them to match what git-annex will expect.

Is there a way that I can tell git-annex about the presence of the data, to save me having to download and re-upload everything, in a way that is safe? What I want seems to be similar to "git annex reinject" but for special remotes (and I'll take care of the renaming), but I don't see anything in the manpage that looks likely.

I can quite easily create and commit the symlinks with correctly predicted names in my master branch. Will git-annex will treat these correctly?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.3"
 subject="comment 1"
 date="2014-05-21T17:41:22Z"
 content="""
If you also have the files present locally, you can simply do `git annex copy --fast --to remote`. git-annex copy will first check to see if the remote has the file; seeing that it does it will update the location log.

Another option, if you have shell access on the remote is to simply set up a git repository there, move the files into it and `git annex add` them, and merge that into your local repository.

There is not currently any way to set the [[location_tracking]] information to tell git-annex that a file has appeared on a remote. Of course, you can modify the git-annex branch manually to do so. See [[internals]].
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkexhIpGcYa22aPQtLm-StpHiF-MHYPh5w"
 nickname="Robie"
 subject="Solved"
 date="2014-06-01T13:55:58Z"
 content="""
Thanks to Joey for pointing me in the right direction. I got this working now.

There are approximately three steps:

1. Obtain a mapping of git-annex key to friendly name, and rename all entries
in the special remote to their git-annex keys.

2. Create and commit symlinks in the `master` branch (or wherever you want them).

3. Add location tracking entries to the `git-annex` branch for all entries.

First, I created an \"index\" file describing the contents of my special remote,
in the form \"KEY NAME\" where KEY is the git-annex key (I used SHA256) and NAME
is the name I want to use for each file.

## Step 1: Map and rename

In my case I was \"importing\" a ddar remote, so I wrote a quick script
(https://github.com/basak/ddar/blob/master/contrib/git-annex-convert.py) to
generate this index as well as rename all ddar archive members to their
git-annex keys instead.

## Step 2: Create and commit symlinks

Then, I created symlinks in my master branch using:

    exec 3<index
    while read key name <&3; do ln -s `git annex examinekey --format='.git/annex/objects/${hashdirmixed}${key}/${key}' \"$key\"` \"$name\"; git add \"$name\"; done
    git commit -m'Import ddar repository contents'
    exec 3<&-

## Step 3: Add to location tracking

First I set three variables:

    remote_name=ddar  # set to the name of the ddar git-annex special remote

    uuid=`git config remote.\"$remote_name\".annex-uuid`
    now=`printf %6fs $(date +%s)`

Then I added the entries from `index` into location tracking as follows:

    git checkout git-annex
    exec 3<index
    while read key name <&3; do log=`git annex examinekey --format='${hashdirlower}${key}.log' \"$key\"`; mkdir -p `dirname \"$log\"`; echo \"$now 1 $uuid\" >> \"$log\";git add \"$log\";done
    exec 3<&-
    git commit -m'Import knowledge of ddar repository contents'

## Verifying

`git-annex fsck --from ddar --fast` checks that the keys expected in the
special remote can be found (replace the special remote name as needed).

Skipping `--fast` will download all data to verify it. I didn't do this -
instead I just sampled one entry which seemed to be OK.

"""]]


code:
One reason to use git-annex is to save disk space by tossing files you don't use that often.

I can find big files in the repository with git annex find --largerthan=100M, but is there a way to find large *directories*?  In an ordinary filesystem I'd use "du -h" with a maxdepth to get an idea of what parts of a directory are taking up my disk space, but obviously that won't work with git annex because all the content is in .git/annex.  Any ideas?

(I can get a listing of file sizes in a directory with the handy -L flag of ls -- "ls -lL" shows me the sizes of the link targets -- but that won't summarize all the sizes of subdirectories.  Unless my ls-fu is just weak.)


code:
[[!comment format=mdwn
 username="http://sunny256.sunbase.org/"
 nickname="sunny256"
 subject="comment 1"
 date="2012-11-28T23:21:11Z"
 content="""
du(1) also accepts the -L option, so if you for example want to find what directories occupies most storage:

    $ du -L | sort -n
"""]]


code:
[[!comment format=mdwn
 username="http://sunny256.sunbase.org/"
 nickname="sunny256"
 subject="comment 2"
 date="2012-11-28T23:24:11Z"
 content="""
And if you want to find the biggest files in a directory tree:

    $ find -type l -print0 | xargs -0 du -L | sort -n | tail -500
"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 3"
 date="2012-11-29T02:03:24Z"
 content="""
Sweet!  I should have RTFM a bit more.  Thanks. :)
"""]]


code:
[[!comment format=mdwn
 username="Steve"
 ip="92.104.175.136"
 subject="comment 4"
 date="2012-11-29T23:51:21Z"
 content="""
I've been thinking about writing a sort of git-annex du.  I'm surprised to find someone else looking for such a thing.  While \"du -L\" will tell you how much space is used by files you actually have, I was interested in knowing (approximately) how much space would be used if you were to git-annex get everything you don't yet have.

There are many options and variations to think about, such as:

* do you want to count duplicate files once or as many times as they appear (as if you 'git-annex lock'd them all)
* maybe you want to know how much space is used by files that reside only on a certain remote or set of remotes
* you might want to know how much space would be used by all the files you don't yet have, but not count the files you already have

All of the backends so so far seem to store the size of the files in the filename, so my plan was to read it out of the links.  If anybody has a better idea about how to get the sizes of annexed files or options that would be handy for a git-annex du, let me know.  I'll see if I can get the start of something useful this weekend.  I'll post here when I have something to share.

I'm also open to suggestions for the executable name.  Right now I'm thinking \"gadu\" for git-annex disk usage.
"""]]


code:
[[!comment format=mdwn
 username="http://sunny256.sunbase.org/"
 nickname="sunny256"
 subject="comment 5"
 date="2012-11-30T00:29:44Z"
 content="""
Steve, that would be a very useful utility. I've been thinking of such a tool, but haven't gotten around to write it yet. It would be practical to have before copying big/many files from another drive. If I've been short of free space, I've executed `du -L` in the source directory, but that's a bit cumbersome.

And \"gadu\" is a fine name, yes. Goes well along with my \"ga\" shortcut for \"git annex\", which I created two hours after I started using git-annex. I've probably saved thousands of keystrokes because of that. ☺
"""]]


code:
[[!comment format=mdwn
 username="Steve"
 ip="92.104.175.136"
 subject="gadu 0.01 is up"
 date="2012-12-08T06:22:50Z"
 content="""
I've got an initial try at gadu up over at <http://git-annex.mysteryvortex.com/git-annex-utils.html>  I created a separate thread for it: <http://git-annex.branchable.com/forum/gadu_-_git-annex_disk_usage/>
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/2grhJvAC049fJnvALDXek.6MRZMTlg--#eec89"
 nickname="John"
 subject="Also try sizes"
 date="2013-01-02T22:05:39Z"
 content="""
The \"sizes\" tool on Hackage is a git-annex aware du-like utility.  Just give it the \"-A\" flag to have it interpret annex symlinks as if they were normal files, and to also ignore files inside a .git/annex.
"""]]


code:
I tried to "git annex copy --to SSH-REMOTE" it says it sends it:
copy Test (checking SSH-REMOTE...) (to SSH-REMOTE...)
SHA256E-s6--8283daccd02d2b2797da53446d367e39dc0fb1615bcae4274460f44455bd9822
           6 100%    0.00kB/s    0:00:00 (xfer#1, to-check=0/1)

sent 146 bytes  received 31 bytes  354.00 bytes/sec
total size is 6  speedup is 0.03
ok
(Recording state in git...)

But I can't find it in the working directory. Can anyone help me?


code:
[[!comment format=mdwn
 username="http://www.rfc1149.net/"
 nickname="Sam"
 subject="comment 1"
 date="2013-08-05T09:49:56Z"
 content="""
`git annex copy` will only push objects (content) into git annex private directory. You have to issue a `git annex merge` (or `git annex sync`) on the receiving end, or run `git annex assistant` there to update the working directory.

Note that you can run `git annex merge` as a post-update hook if you want this to be done automatically.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmqaNwDQ367zpW6cIRviLz6zJZZFODgoEI"
 nickname="Zack"
 subject="It worked!"
 date="2013-08-05T09:57:25Z"
 content="""
Thanks for the help, it worked.
"""]]


code:
I found the command "git annex lock" very slow (much slower than the initial "git annex add" with SHA1), for a not so big directory, when run in a big repo.
It seems that each underlying git command is not fast, so I thought it would be better to run them once with all files as arguments.
I had to stop the lock command, and ran "git checkout ." (I did not change any file), is this a correct alternative?

Thanks a LOT for this software, one that I missed since a long time (but wasn't able to write)!

Rafaël


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="fixed"
 date="2011-05-31T18:51:13Z"
 content="""
Running `git checkout` by hand is fine, of course.

Underlying problem is that git has some O(N) scalability of operations on the index with regards to the number of files in the repo. So a repo with a whole lot of files will have a big index, and any operation that changes the index, like the `git reset` this needs to do, has to read in the entire index, and write out a new, modified version. It seems that git could be much smarter about its index data structures here, but I confess I don't understand the index's data structures at all. I hope someone takes it on, as git's scalability to number of files in the repo is becoming a new pain point, now that scalability to large files is \"solved\". ;)

Still, it is possible to speed this up at git-annex's level. Rather than doing a `git reset` followed by a git checkout, it can just `git checkout HEAD -- file`, and since that's one command, it can then be fed into the queueing machinery in git-annex (that exists mostly to work around this git malfescence), and so only a single git command will need to be run to lock multiple files.

I've just implemented the above. In my music repo, this changed an lock of a CD's worth of files from taking ctrl-c long to 1.75 seconds. Enjoy!

(Hey, this even speeds up the one file case greatly, since `git reset -- file` is slooooow -- it seems to scan the *entire* repository tree. Yipes.)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnpdM9F8VbtQ_H5PaPMpGSxPe_d5L1eJ6w"
 nickname="Rafaël"
 subject="comment 2"
 date="2011-05-31T21:43:22Z"
 content="""
Nice!
So if I understand correctly, 'git reset -- file' was there to discard staged (but not commited) changes made to 'file', before checking out, so that it is equivalent to directly 'git checkout HEAD -- file' ?
I'm curious about the \"queueing machinery in git-annex\": does it end up calling the one git command with multiple files as arguments? does it correspond to the message \"(Recording state in git...)\" ?
Thanks!

"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 3"
 date="2011-05-31T21:54:23Z"
 content="""
@Rafaël , you're correct on all counts.
"""]]


code:
[[!comment format=mdwn
 username="josch"
 ip="2001:638:709:5:2ad2:44ff:fe4b:56aa"
 subject="thanks a lot!"
 date="2014-06-23T15:13:34Z"
 content="""
Thank you for the answer to this problem!

I put 150k emails in maildir format in git annex. Adding them took a couple of hours and so did the first full sync. But after that things didnt noticibly slow down - that was a surprise!

What but then I wanted to do a one-time sync using rsync. I didnt find out how to tell rsync to follow symlinks at the destination instead of replacing symlinks with normal files. So I first unlocked the 150k emails (which was moderately quick and probably only I/O bound) and then did the rsync using the --checksum option which worked well as well. The problems started when I wanted to lock the whole thing again. This took ages and even after two days it did not output that it was adding files at all. So I used `find -type f | split -a3 -l100` and then `for f in x*; do echo $f; git annex add `echo \`cat $f\``; rm $f; done`. This started off well and I could finally see files being added. Unfortunately when I came back after a couple of hours, the progress slowed down to a crawl of only one file every few seconds.

The solution was to just `git checkout -- Mail` everything. This finished in a matter of seconds and left the new mail which was copied over by rsync intact.

Thanks a lot for the tip!

Let me hereby also report that git annex seems to work without problems when using it with offlineimap and notmuch. At least I did not run into any problems with that setup yet.
"""]]


code:
git-annex is seriously cool, however I havent figured out an important detail:
in the short presentation Richard Hartmann mentions the Nomad use case and says "... next time Im online I would like to have file x y z ... "
How can this be achieved exactly? If I do a git annex copy/get then it will want to do it instantly, not queuing it.
thanks a lot!
Aron


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
 nickname="Justin"
 subject="comment 1"
 date="2012-07-19T14:41:39Z"
 content="""
What I do for this is maintain a `todo` directory and `cp -a` + `git annex add` files I want to download into this directory.
the `cp -a` maintains the symlinks and `git annex add` fixes them if needed.

Also this works the same way if the situation is reversed and the machine you want to download the files onto is not online.  You can add files to the `todo` directory on the server, then once the client machine is online do a `git annex sync` + `git annex get todo`.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnbBRfl5F8gKRr1ko8Ai6FbEZStXXNF1S4"
 nickname="Áron"
 subject="comment 2"
 date="2012-07-19T22:41:53Z"
 content="""
hmm not bad workaround but still not very comfortable, thanks though.
"""]]


code:
i'm getting errors in ``git annex fsck`` on a shared bare git repo with git-annex 3.20120418 local repo version 3:

``git-annex: ${PATH}/${MYREPO}.git/annex/objects/${HA}/${SH}/SHA1-${HASH}/SHA1-${HASH}: setFileMode: permission denied (Operation not permitted)``

the repository is shared among several users in a common group, and the repo is set up with sticky group, and with appropriate umasks, everything should work.

however, even with the file having permissions -rw-rw-r-- in the directory with permissions drwxrwsr-x, owned by someone else but by a group i'm currently in (as verified by issuing `groups`), i get said error message.

a strace reveals that the failing syscall is:

``[pid 17626] chmod("${FILENAME}", 0100555) = -1 EPERM (Operation not permitted)``

(maybe related: git annex looks for the file in another ${HA}/${SH} combination (of three digits instead of two digits each) before, i take it this is just a new feature not used by the data in my repo? also, i should add that the repository dates back to git-annex 0.13.)

as a workaround, i'm currently ``sudo chown``ing all files to me before the check.

why does fsck try to set permissions even if they are ok? is this a bug in my setup, and if yes, how is a shared repository set up correctly?


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2012-04-21T16:09:19Z"
 content="""
Well, the modes you show are wrong. Nothing in the annex should be writable. fsck needs to fix those. (It's true that it also always chmods even correct mode files/directories.. I've made a change avoiding that.)

I have not thought or tried shared git annex repos with multiple unix users writing to them. ([[tips/Using_gitolite_with_git-annex]] would be an alternative.) Seems to me that removing content from the annex would also be a problem, since the directory will need to be chmodded to allow deleting the content from it, and that will fail if it's owned by someone else.  Perhaps git-annex needs to honor core.sharedRepository and avoid these nice safeguards on file modes then.
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 2"
 date="2012-04-21T23:46:42Z"
 content="""
All right, I've made all the changes so it supports `core.sharedRepository`.
"""]]


code:
[[!comment format=mdwn
 username="http://christian.amsuess.com/chrysn"
 nickname="chrysn"
 subject="comment 3"
 date="2012-04-23T14:14:28Z"
 content="""
thanks, that's great. will there be a way to have sharedRepository work for shared remotes (rsync, directory) too, or is that better taken care of by acls?

@not thought of shared repos: we're having our family photo archive spread over our laptops, and backed up on our home storage server and on an rsync+encryption off-site server, with everyone naturally having their own accounts on all systems -- just if you need a use case.
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 4"
 date="2012-04-23T14:35:39Z"
 content="""
I'm not currently planning to support sharedRepository perms on special remotes. I suppose I could be convinced otherwise, it's perhaps doable for the ones you mention (rsync might be tricky). (bup special remote already supports it of course.)

thanks for the use case!
"""]]


code:
After a `git annex copy --auto -t m3` I got this error:

    (Recording state in git...)
    error: unable to resolve reference refs/heads/git-annex: No such file or directory
    fatal: Cannot lock the ref 'refs/heads/git-annex'.
    git-annex: git [Param "update-ref",Param "refs/heads/git-annex",Param "d768b20f76ce40157214a713fb0ccb9cfc9134c2"] failed

I did a `git annex sync m3` and got

    [...]
    push m3
    Counting objects: 2976, done.
    Delta compression using up to 2 threads.
    Compressing objects: 100% (2052/2052), done.
    Writing objects: 100% (2379/2379), 392.33 KiB, done.
    Total 2379 (delta 1652), reused 460 (delta 316)
    Auto packing the repository for optimum performance.
    error: bad ref for refs/heads/git-annex
    error: bad ref for refs/heads/git-annex
    error: bad ref for refs/heads/git-annex
    To /media/m3/annex
    87c82c5..06219eb  git-annex -> synced/git-annex
    be7ff5e..6625634  master -> synced/master
    ok

And then I ran the command git-annex was trying to run:

    $ git update-ref refs/heads/git-annex d768b20f76ce40157214a713fb0ccb9cfc9134c2
    error: Trying to write ref refs/heads/git-annex with nonexistent object d768b20f76ce40157214a713fb0ccb9cfc9134c2
    fatal: Cannot update the ref 'refs/heads/git-annex'.

`git fsck --full` gives no errors.

What does this error mean?  Should I be worried?  Thanks.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-04-01T18:33:37Z"
 content="""
I'm afraid that this looks like a partially corrupted git repository to me. Something seems to have gone wrong with `.git/refs/heads/git-annex`. Also, git object d768b20f76ce40157214a713fb0ccb9cfc9134c2 seems to have been present before, and be gone now.

I would run `git fsck`, then look at what's happened in `.git/refs/heads`, and if the repository does seem to be damanged, re-clone it. (You can copy over `.git/config` and `.git/annex/` to keep your annexed data in the new clone.) You should probably also run `git annex fsck`, as it could have lost some location log changes.
"""]]


code:
[[!comment format=mdwn
 username="spwhitton"
 ip="163.1.166.255"
 subject="comment 2"
 date="2013-04-06T19:06:24Z"
 content="""
Thanks for taking the time to reply.

1) The corruption spread between repositories so I resorted to restoring from a backup from some days ago, and then I restored newly added files by manually copying the symlinks from the old repo into the new one.  Of course I can't update the git-annex branch by hand.  I've run `git annex fsck`: does that re-create the location log information for the symlinks that I re-added, at least for the local repository even if it doesn't know about copies elsewhere?

2) I initially tried to git clone from my restored backup, rather than just moving the restore into place.  But then while this clone showed no errors on a `git fsck --full`, any git-annex command, such as `init` (following [this](http://git-annex.branchable.com/tips/what_to_do_when_a_repository_is_corrupted/)), gives the following sort of error:

    error: invalid object 100644 <long-SHA-string> for '<path to one of git annex's log files>'
(unfortunately I lost my xterm so don't have one of my actual errors in full).  Can you think of any reason why doing a clone of the backup copy would cause this?  Instead I have just mv'd the backup copy into place and it works fine.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 3"
 date="2013-04-08T17:23:24Z"
 content="""
1. I don't understand how corruption could spread between git repositories. git is supposed to use checksums to prevent bad data propigating. I would be running a lot of `git fsck` if I were you. Yes, `git annex fsck` will update location log information for files it finds in the local repository.

2. Again this looks like something is corrupting your repository when git writes it. Perhaps bad memory? I think you need to get to the root of the problem with the corrupting git repositories.
"""]]


code:
[[!comment format=mdwn
 username="spwhitton"
 ip="163.1.167.50"
 subject="comment 4"
 date="2013-04-17T17:02:11Z"
 content="""
I didn't think that git would let corruption spread either, but it did: the repository m3 which I showed a push to in my original post showed a broken refs/heads/git-annex too.  You can see it happening when git tries to repack m3 after pushing to it in my original post.

I am pretty sure that my HDD on the machine where this first occurred needs replacing; I've marked the remote as untrusted.
"""]]


code:
Hi, another installation issue on Ubuntu Lucid:

I started with a clean `~/.cabal` directory and did the following:

    andreas@antares:~$ sudo aptitude install cabal-install
    andreas@antares:~$ cabal update
    andreas@antares:~$ cabal install git-annex -v --bindir=$HOME/

However, I got some dpendancy error:

    cabal: dependencies conflict: base-3.0.3.2 requires syb ==0.1.0.2 however
    syb-0.1.0.2 was excluded because json-0.5 requires syb >=0.3.3

Any ideas?

Thanks for your help!


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2012-01-15T19:53:35Z"
 content="""
This is now about different build failure than the bug you reported, which was already fixed. Conflating the two is just confusing.

The error message about `syb` is because by using cabal-install on an Ubuntu system from 2010, you're mixing the very old versions of some haskell libraries in Ubuntu with the new versions cabal wants to install. The solution is to stop mixing two package management systems --

* Either install git-annex without using cabal, and use apt-get to install all its dependencies from Ubuntu, assuming your distribution has all the necessary haskell libraries packaged.
* Or `apt-get remove ghc`, and manually install a current version of [The Haskell Platform](http://hackage.haskell.org/platform/) and use cabal.
"""]]


code:
[[!comment format=mdwn
 username="https://openid.stackexchange.com/user/fd55c6e3-966a-4626-865f-5d0f73e1eb88"
 nickname="Andreas H."
 subject="Thanks and sorry"
 date="2012-01-15T20:54:55Z"
 content="""
Joey, thanks for you quick help! I'll try the manual haskell-platform install once I have quicker internet again, i.e. tomorrow.

And sorry for the mess-up; I splitted the post into two. Hope it's clearer now.
"""]]


code:
So much time gone by after the kickstarter and now I wanted to give it a kick finally.
I set up everything on my linux desktop with no problems at all.

I configured a ssh remote with shared encryption key.

Then I tried to use this to share files with a windows Desktop. This is where problems started.

1. Since there is no jabber sync I need to exchange keys manually. But although I read and googled for hours I just found that it would be there in the git repo somewhere in plain text. I didnt find a command to actually display it on the command line.
2. I need to setup the (existing) ssh remote on windows manually, because I could not make the webapp recognize the ssh-keys from pageant. Also, unfortunately, all examples refer to s3. I didnt even find out which type I had to specify. There seems to be no comprehensive list of repository types for the cmdline and their required arguments respectively at all. The man page says "type=..." for the three-line description of shared encryption setups which appeared like a mockery to me. Since the remote does not have git metadata I also can not use git clone.
3. The docs say initremote would only be used on fresh remotes, not existing ones. But enableremote only enables remotes that are already configured. So what am I to do here?

Could someone tell me please

1. How to find the shared key
2. How to add the ssh remote properly
3. Where 1 and 2 are documented

Thank you!


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="71.80.94.56"
 subject="comment 1"
 date="2014-02-07T19:28:00Z"
 content="""
Sorry for the delay getting to this post.

The lack of jabber support on Windows makes this a bit hard to set up. You instead need to make a git repository some place that both the Windows and Linux machines can both connect to. Once git-annex on both systems is syncing with that common git repository, the Windows system will learn about the encrypted remotes you have set up, and then `git annex enableremote` will be able to use them with no problem.

Of course, if you don't trust your ssh server where you already made an encrypted remote, you may not want to store a un-encrypted git repository on it, and since Windows also doesn't support encrypted git repositories yet, you'd be sort of out of luck. (For now; Windows support is being improved.)

OTOH, if you just set up that encrypted rsync remote on the ssh server because the ssh server didn't have git-annex installed on it, you can easily also put a git repository on the ssh server, and the combination will be enough to let you sync between the 2 machines.
"""]]


code:
Let me announce 'metatag', a simple metadata extraction utility.

The Design Idea is to make it completely event driven. There are string matching rules over added metadata,
who invoke engines when matched, which in turn add more metadata and so on.
Thus the whole metadata extraction process is controlled by those easily configurable rules. Processing a file or
directory just starts by adding "/=filename" to the metadata, everything else bootstraps from that. After metadata
got extracted there are exporters which implement different backends for storing this metadata (currently only a
'print' and a 'gitannex' exporter are implemented)

While still in a infancy state it already works for me. It now needs more rules and engines for metadata extraction
and some more efforts to 'standardize' generated metadata. I'd like to welcome comments and contributions.

A README about it can be found at

 <http://git.pipapo.org/?p=metatag;a=blob_plain;f=README>

The code is available under git from

    git clone git://git.pipapo.org/metatag

To make the contribution barrier as low as possible there is a public pushable 'mob' repository where everyone can
send changes too at `git://git.pipapo.org/mob/metatag`

after installing it, using it on a annexed directory is like

    metatag -r -O gitannex,gitexclude -o gitannex:-stat ./

There is a mailinglist for the project, you can subscribe at

 <http://lists.pipapo.org/cgi-bin/mailman/listinfo/metatag>


code: Is there any kind of "lastpresent" in the preferred-content expression? If set, git-annex would see if "git log --follow $path -n 1" (or some configurable -n) was present.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.246"
 subject="comment 1"
 date="2013-11-15T20:39:43Z"
 content="""
The idea seems to be to match files that have been deleted.

I don't see how that could work; preferred content expressions are always matched against files in the working tree.

Even if it were doable, it would be quite expensive if every preferred content query involved a `git log`...

What are you trying to achieve with this?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn1QhtPvsRBV7pfaDW_ZTPFv_ZIxSzQ8Rg"
 nickname="Paul Léo"
 subject="comment 2"
 date="2013-11-16T13:12:35Z"
 content="""
The aim is to be able to have some semi-automatic mode (using \"git-annex wanted . (lastpresent)\"), where file paths which you \"git-annex get\" one time always get updated to the latest version.

Or, you could also use some variant of it the other way round (using \"git-annex wanted . (notlastpresent)\"): all files would be present by default, but paths which you \"git-annex drop\"ed once wouldn't become present again.
"""]]


code:
Thanks for creating git-annex.

I am confused about the advantages of the SHA* backends over WORM.  The "backends" page in this wiki says that with WORM, files "can be moved around, but should never be added to or changed".  But I don't see any difference to SHA* files as long as the premise of WORM that "any file with the same basename, size, and modification time has the same content" is true.  Using "git annex unlock", WORM files can be modified in the same way as SHA* files.

If the storage I use is dependable (i.e. I don't need SHA checksums for detection of corruption), and I don't need to optimize for the case that the modification date of a file is changed but the contents stay the same, and if it is unlikely that several files will be identical, is there actually any advantage in using SHA*?


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2011-08-29T16:10:38Z"
 content="""
You're right -- as long as nothing changes a file without letting the modification time update, editing WORM files is safe.
"""]]


code:
When I went to alternativeto.net I noticed that SpiderOak is a featured application. I decided to search git-annex and see how "Like"-ed it is in comparison... there were 0 "Like"-s.

I suggest going to <http://alternativeto.net/software/git-annex/> and "Like" git-annex.


code:
It would be really neat if there was a way to get just the `git-annex` binary on Android, without getting the entire APK. This can be useful when one is already using their own shell and rootfs, for example I work with this.

This way, one can just use parts of git-annex, without having to have a whole APK and its dependencies, and the space it takes up, etc.

So would you be able to add a download link for this?

Thanks for the wonderful project!


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.193"
 subject="comment 1"
 date="2013-07-02T16:57:43Z"
 content="""
This seems a pretty unusual use case. Can't you just unpack the apk (it's a regular zip file) and get out the parts you want?
"""]]


code:
First off, thanks so much for your hard work, git-annex is amazing.

I just started using the [web as a special remote](http://git-annex.branchable.com/tips/using_the_web_as_a_special_remote/) feature with the SHA256E backend, and I noticed that although the annexed file has the correct backend prefix (SHA256E) it does not have the extension of the file in the URL. The URL is `https://...IMG_1234.JPG` but the annexed file is `SHA256E-...832c99` with no extension.

This is fine for most use cases, but I actually access an S3 remote directly from another app (independent of git-annex) to render photos, and in that app I'm using the extensions to figure out file types, so not having that info is slightly inconvenient.

Is there any way to either:

1. tell git-annex to preserve the extension of a file on the web in the annexed file, or
2. alternatively, change the annexed filename (add the extension manually) without screwing anything up?

Any help would be much appreciated, thanks!


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-05-06T15:43:06Z"
 content="""
You don't say what version of git-annex you're using. I tested with the current version, 4.20130405:

[[!format sh \"\"\"
joey@gnu:~/tmp/rr>git annex addurl http://localhost/~joey/header_background.png
addurl localhost_~joey_header_background.png (downloading http://localhost/~joey/header_background.png ...) --2013-05-06 11:36:41--  http://localhost/~joey/header_background.png
Resolving localhost (localhost)... ::1, 127.0.0.1, 127.0.1.1
Connecting to localhost (localhost)|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 53693 (52K) [image/png]
Saving to: ‘/home/joey/tmp/rr/.git/annex/tmp/URL--http&c%%localhost%~joey%header_background.png’

100%[======================================>] 53,693      --.-K/s   in 0.001s

2013-05-06 11:36:41 (58.9 MB/s) - '/home/joey/tmp/rr/.git/annex/tmp/URL--http&c%%localhost%~joey%header_background.png' saved [53693/53693]

(checksum...) ok
(Recording state in git...)
joey@gnu:~/tmp/rr>dir
lrwxrwxrwx   1 joey joey  194 Sep 11  2010 localhost_~joey_header_background.png -> .git/annex/objects/M1/p0/SHA256E-s53693--3f065e8e2db1248765d0753cf483e40ae0eaf9bffe34b0cf738158815d0884e3.png/SHA256E-s53693--3f065e8e2db1248765d0753cf483e40ae0eaf9bffe34b0cf738158815d0884e3.png
\"\"\"]]

Looks like it's doing the right thing with the extension!
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnkBYpLu_NOj7Uq0-acvLgWhxF8AUEIJbo"
 nickname="Chris"
 subject="comment 2"
 date="2013-05-07T03:37:24Z"
 content="""
Thanks very much! That's great news. I'm using 3.20120629, I'll just update then.
"""]]


code:
Earlier this week, I somehow lost a ton of files from my annex -- by switching on the command line from indirect to direct mode while the assistant was running, I think. I'm not sure.

Anyway, by "lost" I mean "lost the symlinks to," because git-annex defaults to keeping content around till you tell it otherwise.  So I still had the content in the repos on my two backup drives.  All I needed was the symlinks back.

But how to figure out exactly what I lost and get it back?

I found that out here:

http://stackoverflow.com/questions/953481/restore-a-deleted-file-in-a-git-repo

Here's a magical formula you can use to find every single file deletion in the history of your repo:

    git log --diff-filter=D --summary

That will give you every commit that deleted things, and what was deleted.

To bring back all the files deleted in a given commit, where COMMITHASH is the commit hash, use this command:

    git checkout COMMITHASH^1 -- .

to bring back only a specific file:

    git checkout COMMITHASH^1 -- path/to/file.txt

to bring back only a subdirectory:

    git checkout COMMITHASH^1 -- sub/directory

that will bring them back into the staging area.  You can see which ones just reappeared by typing:

    git status

then you can actually make the restore permanent by typing:

    git commit -m "I just resurrected some files"


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="99.54.57.201"
 subject="comment 1"
 date="2013-01-25T14:07:36Z"
 content="""
seems I inadvertently got caught by Markdown -- I tried to write COMMITHASH-hat-1 and it turned into COMMITHASH superscript 1.

Tilde (~) would also have worked instead of hat (^)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
 nickname="Justin"
 subject="comment 2"
 date="2013-01-25T14:27:47Z"
 content="""
I fixed it for you
"""]]


code:
The assistant archival walk-through describes how files get removed from local repo if moved into archive repos.
What if the files in question aren't needed immediately (and can be archived) for one user of a shared repo, but needed by someone else on their own machine?
git-annex directory arrangement propagation wouldn't work for this case, it seems.  Could assistant handle this case?

It looks like having a browser of the directory tree in git-annex assistant webpage with say right-click menu with "get" and "drop" would be the most flexible.

Or maybe it could be more about integration with Linux etc file browsers (but then it's system specific, unlike with webapp).


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 1"
 date="2013-07-24T15:30:51Z"
 content="""
A feature I would love to see is a file manager integrated with the webapp, so that you could browse your files and retrieve or drop them manually as if you were using the command line.  Then you could have your repo set on \"manual\" and not worry about those archive directories.  Just manually drop content if you got low on space.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 2"
 date="2013-07-25T18:42:59Z"
 content="""
I believe that the world has enough (or too many) file managers, and do not want to build another one just for git-annex. In any case \"git annex get\" or any GUI wrapping around it is not going to help with the described use case, because the problem is that the files are archived away in my offline drive, while you want to access them in your repository, which does not have access to that drive.

The way to handle this situation is to copy or move the files you want out of the archive directory, and back to the parent directory. This move of the file will be synced from your repository to all the other repositories. When the repository that does have access to the archive notices this, it will get the files that have been moved out of the archive. Normal assistant syncing will then arrange to get them transferred back to the repository of the user who had wanted them.

Of course, if the archive is on an offline drive, this won't happen until that drive is plugged back in.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="comment 3"
 date="2013-07-25T19:16:20Z"
 content="""
> The way to handle this situation is to copy or move the files you want out of the archive directory, and back to the parent directory. This move of the file will be synced from your repository to all the other repositories. When the repository that does have access to the archive notices this, it will get the files that have been moved out of the archive. Normal assistant syncing will then arrange to get them transferred back to the repository of the user who had wanted them.

But won't it also transfer the same files into repos of all other users (who didn't want the files)?
On a related note, allowing moving files for everyone could jeopardise integrity of the master archive.  Is there a way to limit what kind of ops the users can do (e.g. copy but not move)?

Re offline drives, the case I was thinking of is actually where each user has access to e.g. special remotes directly.  So a manual get/put would solve the problem at hand.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 4"
 date="2013-07-25T19:23:30Z"
 content="""
Moving files shouldn't jepardize anything. If a user makes a change they shouldn't have made, it can be reverted.

If you have the archive drive available locally you can set your repository to manual mode and just use `git annex get` on content you want. Or you could set it to be a backup repository and the assistant would get all content.
Or write a custom [[preferred_content]] expression for your repository to make it want files in `archive/stillusing/`

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8"
 nickname="develop"
 subject="comment 5"
 date="2013-07-26T08:20:21Z"
 content="""
I think i agree that this should be implemented.

Not only to be able to \"get/drop\" from the assistant without the use of archive folders.

But mainly because i think this will be needed anyways, to do a \"recover previous version of file\" option. Which should definitely be in the web assistant.

We can't expect family and friends to figure out all command line stuff, thus they need to use the assistant. And i really think it would be a shame to not have an easy way to recover old versions, when the old versions are actually stored.

Always think about having a good WAF(Wife acceptance factor).

Just my 2 bits (and a haircut).

"""]]


code:
Hi,

I am trying to understand git-annex, and the walkthroughs on the page did help a lot. I *feel* that git-annex is the solution for my use case: having a large archive, several computers, NAS both at work and at home, being offline during commuting, …

Everything seems to be clear for me when using the command line version. Using annex assistant looks promising, and it is easy to create repositories that match my usage. But now I see that these repositories are in direct mode by default; and I am not sure how this impacts my ability to work with large files. I do not want to end up with git (without annex) handling huge files.

So - can somebody explain how direct mode affects this? Can I switch to indirect mode with the assistant? Am I asking the wrong questions?

thanks,
Ulli


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.246"
 subject="comment 1"
 date="2013-11-09T17:36:01Z"
 content="""
Files in direct mode are stored in the git annex the same way as when using indirect mode. But you're perhaps right to be wary of using direct mode, it's been quite easy to mess up while using it at the command line until very recently when the [[todo/direct_mode_guard]] was implemented.

You can switch repositories created using the assistant from direct to indirect mode if you like. The assistant works in either mode.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmkXtBdMgE1d9nCz2iBc4f85xh4izZ_auU"
 nickname="Ulrich"
 subject="Thanks"
 date="2013-11-12T08:57:36Z"
 content="""
Ok, thanks for the explanation. I still have to figure out what happens when I annex drop files on a direct repository, and how well all this plays with the Finder on Mac…
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmkXtBdMgE1d9nCz2iBc4f85xh4izZ_auU"
 nickname="Ulrich"
 subject="So what does "git annex drop" do in direct mode?"
 date="2013-11-15T15:07:57Z"
 content="""
I just tried a git annex drop on a file in a direct mode repository, and that just did not change anything at all, as far as I can see. Actually I don't know what to expect, but doesn't that mean that I should have to switch to indirect mode in order to be able to drop large files to save space?

Funny is that git annex drop did not complain at all but just reported \"ok\" after quite a while. And when I tried to drop a file that was not available in any other repository, it failed (which is expected behavior). So what is it what was \"ok\"?

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.246"
 subject="comment 4"
 date="2013-11-15T20:33:47Z"
 content="""
Drop in direct mode does the same thing as in indirect mode.

If you are using git-annex on a crippled filesystem without symlinks, git-annex cannot represent a file whose content has been dropped using a broken symlink, so it instead represents it with a nearly empty file.
"""]]


code:
Hi,

I have a question about encryption keys.
Basically I do not know how to use the data which was copied in case my local
machine dies?

## I have done the following:
---
1. copied my ssh public key to a remote server
1. created a directory and started git annex assistant

   mkdir ~/test_annex

   cd ~/test_annex

   git annex webapp

1. clicked Repository->Add another local repository
1. Assistant: Where do you want to put this new repository? Me: ~/test_annex "Make Repository"
1. Assistant "Combine repositories?" Me: "Keep repositories separate"
1. clicked "Add another repository" clicked "remote server"
1. filled out "Adding a remote server using ssh" form and clicked "Check this server"
1. choose "Use an encrypted rsync repository on the server"
1. Assistant "Repository created" I select "Full backup" as repository group.

So far so good. If I now add a file on my local machine into ~/test_annex something not human readable is created on the remote machine.

What I do not understand is how I use this backup later. I am expecting something along the "hybrid encryption keys" scheme
explained in this wiki. However I was not able to determine which of my gpg keys was used or how the data was encrypted.

So my question is: "How do I use the encrypted backup remote on a second machine?"


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 1"
 date="2013-10-27T21:19:45Z"
 content="""
When you create an encrypted rsync repository using the webapp like that, its encryption key is stored in your git repository, using the [[shared encryption scheme|encryption#index2h2]]. No gpg key needs to be used to decrypt files from the rsync repository; anyone with a clone of your git repository can do so.  This has its plusses and its minuses; the webapp picks that type of encryption because it's easy to use.

So, the answer is to just make a clone of your repository on the other machine, and then you can use it. There are lots of ways to do that; if you stay in the webapp, go to Add Another Repisitory and any of the \"Share with your other devices\", \"Share with a friend\", or \"Local computer\" options are easy ways to do it.

----

Now, if you had, manually, set up a rsync repository encrypted with the [[hybrid encryption key scheme|encryption#index1h2]], to access it from another computer with clone of the repository you would need to have a gpg key that has been given access to the repository. So you would either copy your gpg secret key to the other computer, or if you don't want to trust that other computer with the your main gpg key, you could make another gpg secret key for that computer, and add that key as one of the keys that can access the encrypted repository. (Or, if the computer belonged to a friend, you could just get their gpg key, and add it.)
"""]]


code:
[[!comment format=mdwn
 username="Frank"
 ip="31.19.114.93"
 subject="Comment 2"
 date="2013-11-07T19:45:16Z"
 content="""
Thank you very much for the explanation! Have fun!

"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/W6vhceUml87q0GB9MGH1qEeteW8-#ee26a"
 nickname="Leandro"
 subject="comment 3"
 date="2014-05-17T17:39:04Z"
 content="""
So, just to clarify things a little bit: Suppose that you have a remote at box.com (shared encrypted), then the CEO at box.com would eventually be able to decrypt your files easily since the keys are store in the git repository without encryption. Is that right?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="216.145.95.162"
 subject="comment 4"
 date="2014-05-19T15:38:14Z"
 content="""
No, box.com cannot decrypt your files, because the git repository is not stored on box.com.
"""]]


code:
Debian Squeeze, git version 1.7.10.4, git-annex version 3.20121211

The machine has a clone of the annex with preferred content string: `present or include=calibre/* or include=img/* or include=mail/* or include=music/* or include=sounds/*`

This, I believe, should mean the assistant should never drop anything.  However for the past two days it's been moving files to an encrypted rsync remote so that there are two copies (there is another copy on an external HDD) and then it's dropping them from the current annex.  I want to keep the files here; can anyone think of any reason why they would be moved away?


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 1"
 date="2012-12-19T17:19:28Z"
 content="""
I had similar issues when I tried to use \"present\" settings a while back, except it was with adding files.  I would add a file and then it would be immediately dropped (because it was not yet \"present\" while it was being added?).  I gave up and just started using \"archive\" subdirectories for things I didn't want to be present.  Even so I'm having some confusing issues with the assistant.  It keeps dropping certain files which are in \"backup\" repositories and therefore should never have anything dropped.  I haven't isolated what's going on to the degree I can get a good bug report together.  But I'm having a hard time trusting assistant and its content settings to do what I want.  I have a couple scripts that do things like:

git annex copy --not --in=usbdrive --to=usbdrive

to populate my backup repos, and these seem much more reliable than the assistant.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.8.117"
 subject="comment 2"
 date="2012-12-19T18:20:50Z"
 content="""
I've found and fixed a bug in transfer log parsing that could cause the assistant to get confused about the name of the file it had just transferred, and think it was not preferred content and drop it. I think this was causing both problems reported above.
"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 3"
 date="2012-12-20T00:05:08Z"
 content="""
Thanks, I'll give it another shot!

"""]]


code:
"watch" is described as

> With this running as a daemon in the background, you no longer need to manually run git commands  when  manipulating your files.

and "assistant" is described as

> Like watch, but also automatically syncs changes to other remotes.

I would like the behaviour of assistant, without the watch part:

I have an archive of large files, which I think would be useful to manage manually using git.  But I don't want to manually enforce the "numcopies=" requirement, playing with the assistant and webapp it seems really nice to have it take care of that.

Is there currently an "assistant-without-watch" option?   If not, is it planned?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.253.75"
 subject="comment 1"
 date="2013-02-22T22:31:29Z"
 content="""
Not quite.

Recent versions allow you to disable the automatic committing while the assistant is running by setting annex.autocommit=false .. or just by going into the webapp repository list and pausing syncing for the \"here\" repository.

But, with automatic committing disabled, the assistant doesn't know when new files are added, so will not transfer them.  The rest of it still works, so for example if another clone of the repository makes changes
the assistant will merge them in and download the new files.

To transfer new files you created, you could use `git annex copy --auto --to someremote`, which will only copy them if needed to satisfy numcopies.
"""]]


code:
To connect to a remote server I have to tunnel it through a server I have access to. My .ssh/config for this connection looks like this:

    Host server-at-home
        HostName foo.bar.de
        User foo
        Port 222
        ProxyCommand /usr/bin/ssh me@server-i-have-access-to /bin/nc -w 3700 %h %p

Is it possible to create a connection using a similar configuration via the git-annex assistant?


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/FHnTlSBo1eCGJRwueeKeB6.RCaPbGMPr5jxx8A--#ce0d8"
 nickname="Hamza"
 subject="comment 1"
 date="2013-11-28T22:08:04Z"
 content="""
You can add/clone from the remote from the command line and webapp will use it.
"""]]


code:
This is a continuation of the conversation from [[the comments|design/assistant/#comment-77e54e7ebfbd944c370173014b535c91]] section in the design of git-assistant. In summary, I've setup an auto builder which should help [[Joey]] have an easier time developing on git-annex on non-linux/debian platforms. This builder is currently running on OSX 10.7 with the 64bit version of Haskell Platform.

The builder output can be found at <http://www.sgenomics.org/~jtang/gitbuilder-git-annex-x00-x86_64-apple-darwin10.8.0/>, the CGI on this site does not work as my OSX workstation is pushing the output from another location.

The builder currently tries to build all branches except

* debian-stable
* pristine-tar
* setup

It also does not build any of the tags as well, Joey had suggested to ignore the bpo named tags, but for now it's easier for me to not build any tags. To continue on this discussion, if anyone wants to setup a gitbuilder instance, here is the build.sh script that I am using.

<pre>
#!/bin/bash -x

# Macports
export PATH=/opt/local/bin:$PATH

# Haskell userland
export PATH=$PATH:$HOME/.cabal/bin

# Macports gnu
export PATH=/opt/local/libexec/gnubin:$PATH

make || exit 3

make -q test
if [ "$?" = 1 ]; then
        # run "make test", but give it a time limit in case a test gets stuck
        ../maxtime 1800 make test || exit 4
fi
</pre>

It's also using the branches-local script for sorting and prioritising the branches to build, this branches-local script can be found at the [autobuild-ceph](https://github.com/ceph/autobuild-ceph/blob/master/branches-local) repository. If there are other people interested in setting up their own instances of gitbuilder for git-annex, please let me know and I will setup an aggregator page to collect status of the builds. The builder runs and updates on a very regular basis.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 1"
 date="2012-07-02T16:25:55Z"
 content="""
I've some binaries for OSX which can be found at <http://www.sgenomics.org/~jtang/gitbuilder-git-annex-x00-x86_64-apple-darwin10.8.0-binary/dist/> its just the master branch, and it's built on a system that runs macports. Binaries are built and updated whenever there are changes made to the master branch of git-annex.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 2"
 date="2012-09-29T09:01:11Z"
 content="""
I'm currently playing with the autobuilder to get it to do more sensible things so it may be broken for a day or two
"""]]


code:
I have two copies of a large collection of photos on two computers. I have been using rsync to keep them synchronized, but I would like to use git annex instead. I finished initializing one repo, and now its time to initialize the other repo before I can sync them up. But it seems redundant that the other repo will compute its own hashes although I know that the two contents are identical. Can't I just copy the keys and tell git annex to assume that any files seen have already been hashed?

As a start, I cloned the .git from the first machine to the second. But that wasn't enough ... I now have an empty git-annex repo in the second machine. What else is missing?


code:
Wouldn't it make sense to offer

    git annex pull

which would basically do

    git pull
    git annex get

and

    git annex push

which would do

    git annex commit .
    git annex put # (the proposed "send to default annex" command)
    git commit -a -m "$HOST $(date +%F-%H-%M-%S)" # or similar
    git push

Resulting in commands that are totally analogous to git push & pull: Sync all data from/to a remote.

> Update:

This is useful:

    git config [--global] alias.annex-push '!git pull && git annex add . && git annex copy . --to $REMOTE --fast --quiet && git commit -a -m "$HOST $(date +%F--%H-%M-%S-%Z)" && git push'


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2011-04-05T18:05:00Z"
 content="""
Maybe, otoh, part of the point of git-annex is that the data may be too large to pull down all of it.

I find mr useful as a policy layer over top of git-annex, so \"mr update\" can pull down appropriate quantities of data from
appropriate locations.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 2"
 date="2011-04-05T20:52:52Z"
 content="""
No-so-subtle sarcasm taken and acknowledged :)

Arguably, git-annex should know about any local limits and not have them implemented via mr from the outside. I guess my concern boils down to having git-annex do the right thing all by itself with minimal user interaction. And while I really do appreciate the flexibility of chaining commands, I am a firm believer in exposing the common use cases as easily as possible.

And yes, I am fully aware that not all annexes are created equal. Point in case, I would never use git annex pull on my laptop, but I would git annex push extensively.

"""]]


code: I wrote more than 2 years ago a bash completion file for git-annex (did I miss an official version?) I just updated it today to match current version of git-annex, it is far from perfect, still incomplete, and probably buggy, but works for me. I tried to attach the file in case it could be useful to someone, but it seems that I can't; is it ok to paste the code directly?


code:
[[!comment format=mdwn
 username="ringprince"
 ip="134.76.140.110"
 subject="comment 1"
 date="2013-10-10T07:08:21Z"
 content="""
I hope it is.  I'd love to use it.
"""]]


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="72.0.72.144"
 subject="comment 2"
 date="2013-10-10T09:22:00Z"
 content="""
it is, please publish.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkUwqII7LhbatqAQY1T5ZZOdPEFcQJKG6I"
 nickname="Rafael"
 subject="comment 3"
 date="2013-10-13T16:14:34Z"
 content="""
This code needs the bash-completion file for git by Shawn O. Pearce, I think it is distributed by default on debian. Any feed-back or improvement is welcome! I'm far from expert, use at your own risks.

    #!bash
    #
    # bash completion support for git-annex

    # depends on git completion file (by Shawn O. Pearce):
    [ -n \"$__git_whitespacelist\" ] || . $BASH_COMPLETION_DIR/git

    # almost copy of __git_aliased_command
    # requires 2 arguments: alias and main command (after git)
    __git_aliased_subcommand ()
    {
        local word cmdline=$(git --git-dir=\"$(__gitdir)\" \
            config --get \"alias.$1\")
        for word in $cmdline; do
            case \"$word\" in
                \!*)	: shell command alias ;;
                -*)	: option ;;
                *=*)	: setting env ;;
                git)	: git itself ;;
                \"$2\")   : main command
                    local found=1 ;;
                *)
                    [ -n \"${found-}\" ] && echo \"$word\"
                    return
            esac
        done
    }

    _git_annex ()
    {
        _get_comp_words_by_ref -n =: cur words
        # $ git annex 2>&1 |sed '1,6d '|grep -v '^$'|grep -v 'commands:$'|cut -c1-12
        local subcommands=\"
            add addurl assistant copy drop edit get import importfeed lock
            mirror move rmurl sync unlock watch webapp content dead describe
            direct enableremote group indirect init initremote semitrust
            trust ungroup untrust vicfg addunused dropunused fix forget fsck
            merge unused upgrade find help list log map status version
            whereis migrate reinject unannex uninit dropkey
            \"
        # plumbing (to complete?): fromkey fuzztest pre-commit rekey test
        # transferkey transferkeys xmppgit
        local subcommand=\"$(__git_find_on_cmdline \"$subcommands\")\"
        if [ -z \"$subcommand\" ]; then
            ## search for aliased subcommand
            ## works only if the alias is invoked just after git
            ## (simpler, cf _git function)
            local maybesubcom maybealias=${words[1]}
            [ -n \"$maybealias\" ] && # false with a bash alias='git annex'
            [ \"${maybealias:0:1}\" != '-' ] &&
            maybesubcom=$(__git_aliased_subcommand \"$maybealias\" annex)
            for subcommand in $subcommands \"\"; do
                [ \"$maybesubcom\" = \"$subcommand\" ] && break
            done
            [ -z \"$subcommand\" ] &&
            __gitcomp \"$subcommands\" &&
            return
        fi
        case \"$cur\" in
            --from=*|--to=*|--trust=*|--semitrust=*|--untrust=*|--in=*)
                __gitcomp \"here $(__git_remotes)\" \"\" \"${cur##*=}\"
                ;;
            --*)
                __gitcomp \"
                    --force --fast --auto --all --unused --quiet --verbose --json --debug
                    --no-debug --from= --to= --numcopies= --time-limit= --trust= --semitrust=
                    --untrust= --trust-glacier-inventory --backend= --format= --user-agent=
                    --exclude= --include= --in= --copies= --inbackend= --inallgroup=
                    --smallerthan= --largerthan= --not --and --or
                    \"
                ;;
            # -*)
            # __gitcomp \"-( -) -c\"
            # ;;
            *)
                case \"$subcommand\" in
                    # subcommands with PATH (currently outputs wrongly \"uninit\",
                    # and not \"reinject\")
                    # $ tr ' ' '|' <<< $(git annex 2>&1 |sed '1,6d '|grep -v '^$' | \
                    # grep -v 'commands:$'|grep PATH |cut -c1-12)
                    add|copy|drop|edit|get|import|lock|mirror|move|unlock|fix| \
                        fsck|find|list|log|status|whereis|migrate|unannex| \
                        fromkey|pre-commit|rekey|reinject)
                        COMPREPLY=() # complete with paths
                        ;;
                    addurl|importfeed|rmurl) # URL commands
                        COMPREPLY=() # at least works with file://...
                        ;;
                    init|uninit|unused|merge|status|map|upgrade|version|assistant| \
                        watch|webapp|direct|indirect|vicfg|forget|help) # | plumbing...
                        COMPREPLY= # no more args expected
                        # is 'COMPREPLY=' correct? (seems not standard practice...)
                        ;;
                    describe|trust|untrust|semitrust|sync|content|dead|group|ungroup)
                        __gitcomp \"here $(__git_remotes)\"
                        # TODO: handle git-annex special remotes
                        ;;
                    initremote|enableremote)
                        case \"$cur\" in
                            type=*)
                                __gitcomp \"S3 bup directory rsync gcrypt webdav\" \"\" \"${cur##type=}\"
                                ;;
                            encryption=*)
                                __gitcomp \"none hybrid shared pubkey\" \"\" \"${cur##encryption=}\"
                                ;;

                            buprepo=|directory=|rsyncurl=)
                                COMPREPLY=()
                                # use COMPREPLY (and compgen below) directly because
                                #  __gitcomp does not handle '*=*' pattern, only '--*=*'
                                # Writing a __gitannexcomp function may be worth it.
                                ;;
                            *)
                                local initremoteopts=\"
                                    type= encryption= buprepo= directory= rsyncurl= \"
                                COMPREPLY=($(compgen -W \"$initremoteopts \" -- \"$cur\"))
                                ;;
                        esac
                        ;;
                esac
                ;;
        esac
    }

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="64.134.31.139"
 subject="comment 4"
 date="2013-10-15T17:53:21Z"
 content="""
@Rafael, you should put a license on that code..
"""]]


code:
When I copy my local repository with SHA* to a remote repo with SHA*, every single file is checked by itself which seems rather inefficient. When my remote is accessed via ssh, git-annex opens a new connections for every check. If you are not using a ssh key or key agent, this gets tedious...

For all locked files, either git's built-in mechanisms should be used or, if that's not possible, a few hundred checksums (assuming SHA* backend) should be transfered at once and then checked locally before deciding that to transfer.

Once all checks are done, one single transfer session should be started. Creating new sessions and waiting for TCP's slowstart to get going is a lot less than efficient.

-- RichiH

> (Use of SHA is irrelevant here, copy does not checksum anything.)
>
> I think what you're seeing is
> that `git annex copy --to remote` is slow, going to the remote repository
> every time to see if it has the file, while `git annex copy --from remote`
> is fast, since it looks at what files are locally present.
>
> That is something I mean to improve. At least `git annex copy --fast --to remote`
> could easily do a fast copy of all files that are known to be missing from
> the remote repository. When local and remote git repos are not 100% in sync,
> relying on that data could miss some files that the remote doesn't have anymore,
> but local doesn't know it dropped. That's why it's a candidate for `--fast`.
>
> I've just implemented that.
>
> While I do hope to improve ssh usage so that it sshs once, and feeds
> `git-annex-shell` a series of commands to run, that is a much longer-term
> thing. --[[Joey]]

>> FYI, in a repo with 1228 files, all small, repos _completely in sync_.

    % git annex copy . --to foo # 1200 seconds
    % git annex copy . --to foo --fast # 20 seconds

>> RichiH


code:
I am evaluating the best strategy to use git-annex to manage my media library. It consists of about 300.000 files totaling 1 TB of data.

My question is, wheither it would be of advantage to split the repo into several smaller ones (Like Photos, Videos, Musik, Books, ...)?

Would this affect performance of certain operations? I.e. Operations that have superlinear (O(n^a) with a > 1) complexity?

I am thinking about "git annex unused", which takes 22 minutes on my machine performed on the full repo.

Do you have more interesting information on using git-annex in this scale?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.6.49"
 subject="comment 1"
 date="2012-11-28T18:16:10Z"
 content="""
`git-annex unused` needs to scan the entire repository. But it uses a bloom filter, so its complexity is O(n) to the number of keys.

`git annex fsck` scans the entire repository and also reads all available file content. But we have incremental fsck support now.

The rest of git-annex is designed to have good locality.

The main problem you are likely to run into is innefficiencies with git's index file. This file records the status of every file in the repository, and commands like `git add` rewrite the whole file. git-annex uses a journal to minimise operations that need to rewrite the git index file, but this won't help you when you're using raw git commands in the repository.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlgyVag95OnpvSzQofjyX0WjW__MOMKsl0"
 nickname="Sehr"
 subject="comment 2"
 date="2012-12-02T19:24:59Z"
 content="""
The biggest problem I am facing actually is the git pack-objects command, which takes forever.

I am not starting it manually, but \"git annex sync\" is.

Any advice on how to make this faster?
Does it make sense at all to pack a bunch of uncompressible simlinks?

I already tried the delta=false attrigute, without any (major) effect.
"""]]


code:
What are everyone's tips for importing photos to make best use of metadata and views?

Let's assume there's no need to be compatible with a photo manager app, but we may be importing lots of duplicates, and while content deduplication is great, I'd like to avoid naming problems too.

Do you bother to rename your photos?

Do you use EXIF metadata as git-annex metadata? Selectively or wholesale, with all the redundant tags in EXIF?

If you do use a photo manager app, do you need to do anything special to make that work?

Thanks for your responses everyone!

-mike


code:
[[!comment format=mdwn
 username="Xyem"
 ip="178.79.137.64"
 subject="comment 1"
 date="2014-04-15T14:25:04Z"
 content="""
git-annex's metadata and views made me stop hopping between programs (digikam, tagsistant etc.) to organise my photos (I had even just started working on my own FUSE tagging filesystem which was effectively going to be tagsistant, but with a git-annex'y backend).

As usual, my method is probably a little odd :)

Photos are 'git import'ed into a $(uuidgen) directory (so no worries about filename collisions) and tagged with media=Photograph and tag=untagged. Then I go through them and add relevant tags (one of which is \"xbmc\", no prizes for guessing how that works with the views :]) and move them into a more appropriate directory structure, using gqview and its \"sort manager\". This is really nice and fast, due to it only copying/moving symlinks!

One thing I'm considering doing it putting a shim between git-annex and gqview, so that it generates entries in the sort manager which are appropriate for the current view. So, for example, if the view is location=*, the sort manager would have:

    location=Malta
    location=York

While it wouldn't get updated if I create new tags (by creating directories in the view), it would save a lot of time creating them every time the view changes.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.191"
 subject="comment 2"
 date="2014-04-17T20:11:59Z"
 content="""
It seems to me that there is a lot of cruft in EXIF that I would not want to bloat my git-annex branch with storing. That's why [[tips/automatically_adding_metadata]] imports only the listed fields. It's easy to add fields later and re-run the metadata importer on your existing files.

I have not gotten as far as having any best practices to share. :)

"""]]


code:
I am using bit torrent sync to sync folders and files between my server, my laptop and my smartphone.
How similar, or different, is git-annex?

Btsync allows me to sync files with no centralized server, encrypted, with some revision history (but I believe it is basic).

/glattering


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="2001:1928:1:9::1"
 subject="comment 1"
 date="2014-01-16T15:55:05Z"
 content="""
Maybe [[telehash|devblog/day_97__exciting_telehash_possiblities/]] is the free and open source answer to btsync?
"""]]


code:
Hey, so I am trying to compile git-annex through cabal but it fails on building uuid. It gives me this strange error:

    Resolving dependencies...
    Configuring uuid-1.3.1...
    Building uuid-1.3.1...
    Preprocessing library uuid-1.3.1...
    [ 1 of 10] Compiling Data.UUID.Builder ( Data/UUID/Builder.hs, dist/build/Data/UUID/Builder.o )
    [ 2 of 10] Compiling Data.Word.Util   ( Data/Word/Util.hs, dist/build/Data/Word/Util.o )
    [ 3 of 10] Compiling Data.UUID.Internal ( Data/UUID/Internal.hs, dist/build/Data/UUID/Internal.o )
    Data/UUID/Internal.hs:394:20: Not in scope: `BL.fromStrict'
    Data/UUID/Internal.hs:399:48: Not in scope: `BL.toStrict'
    cabal: Error: some packages failed to install:
    git-annex-4.20131002 depends on uuid-1.3.1 which failed to install.
    uuid-1.3.1 failed during the building phase. The exception was:
    ExitFailure 1

Any advice on how to proceed?

System is: Debian Wheezy (CrunchBang Waldorf) 64bit

versions are:
    uuid-dev: 2.20.1-5.3
    cabal: 1.14.0


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="64.134.31.139"
 subject="comment 1"
 date="2013-10-22T17:52:10Z"
 content="""
That version of the haskell uuid library installs ok on my Debian sid system. Seems to me you have a too old version of the haskell bytestring library, which added `fromStrict` in version 0.10.0.0. Probably `cabal install bytestring` will fix it up for you, but building git-annex from source using cabal is a complicated thing that can go wrong in many ways. Have you considered one of the prebuilt binaries?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkj7tMEJKcZpNXIFkHAAcNi5qJFSFyjn6o"
 nickname="thissideup"
 subject="comment 2"
 date="2013-10-23T11:15:20Z"
 content="""
I contacted the maintainer of the uuid-package and he fixed the issue in the current release, so everythings fine now. and: everything compiled beautifully.

But to answer your question: I tend to rather compile things myself if I don't find them (or a current enough version) in the repos - but that's purely personal. ;)
"""]]


code:
Hi,

[ sorry for the wrong bugreport in the last version of this entry ]

It seems as if I can only add my own file to the annex, even if I have group write permissions.  Is that correct?  Can that be circumvented other than copying/deleting the large file?

Here is some demonstration:

    > git init
    > git config user.name dtr
    > git config user.email dtrn@dtrn.com

    > dd if/dev/zero of=test2.bin count=1000

    > ll
    -rw-rw-r-- 1 dtr dtr 512000 Sep 16  2013 test2.bin

    > sudo chown someone test2.bin
    > ll
    -rw-rw-r-- 1 someone dtr 512000 Sep 16  2013 test2.bin

    > git annex init
    > git annex add test2.bin
    add test2.bin (checksum...)
    git-annex: /test-git-annex/.git/annex/objects/Fq/f6/SHA256-s512000--2d4da04b861bb9dbe77c871415931785a18138d6db035f1bbcd0cf8277c6fc23/SHA256-s512000--2d4da04b861bb9dbe77c871415931785a18138d6db035f1bbcd0cf8277c6fc23: setFileMode: permission denied (Operation not permitted)
    failed
    git-annex: add: 1 failed


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.220"
 subject="comment 1"
 date="2013-09-25T18:42:00Z"
 content="""
git-annex needs to be able to lock down files to ensure that nobody can write to them, and to do this it needs to remove the write bit, and you can't remove the write bit from a file you don't own.

Note that if you configure git's core.sharedRepository when making a repository (git init --shared), then all files in both git and git-annex will be group writable. Put you and the other person you wanted to be able to write to the file in a group, and you can both access the repository. So that's the right way to do it.
"""]]


code:
[[!comment format=mdwn
 username="ringprince"
 ip="134.76.140.110"
 subject="comment 2"
 date="2013-09-26T06:32:48Z"
 content="""
Thanks a lot.  The solution with --shared was really helpful.
"""]]


code:
Hi,
a few years ago I wrote a tool called 'ddm'.  The code is overengineered and the script is more complicated then it should be,
but I think it demonstrates some good use cases, and I wonder how well git-annex can fulfill the requirements for those use cases - maybe I should remove ddm and start hacking with git-annex instead.

To answer this question, you should read the section about the possible dataset types on http://dieter.plaetinck.be/ddm_a_distributed_data_manager.html, and the example at the bottom of that page. it demonstrates the idea behind the "selection" dataset to always try to keep a subset (the most appropriate, based on the output of some script) of files "checked out".
the introduction section on https://github.com/Dieterbe/ddm/raw/358f7cf92c0ba7b336dc97638351d4e324461afa/MANUAL should further clarify things, as well as give some more good use cases (as you can see it's a bit more about [semi-]automated workflows then purely tracking what's where)

So I'm not sure, maybe the way to go for me is to make git-annex my "housekeeping about which data is where" backend and make ddm into a set of policies and tools on top of git-annex.

Any input?

Thanks,
Dieter


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2011-02-14T22:08:54Z"
 content="""
Yes, there is value in layering something over git-annex to use a policy to choose what goes where.

I use [mr](http://kitenet.net/~joey/code/mr/) to update and manage all my repositories, and since mr can be made to run arbitrary commands when doing eg, an update, I use its config file as such a policy layer. For example, my podcasts are pulled into my sound repository in a subdirectory; boxes that consume podcasts run \"git pull; git annex get podcasts --exclude=\"*/out/*\"; git annex drop podcasts/*/out\". I move podcasts to \"out\" directories once done with them (I have yet to teach mpd to do that for me..), and the next time I run \"mr update\" to update everything, it pulls down new ones and removes old ones.

I don't see any obstacle to doing what you want. May be that you'd need better querying facilities in git-annex (so the policy layer can know what is available where), or finer control (--exclude is a good enough hammer for me, but maybe not for you).
"""]]


code:
[[!comment format=mdwn
 username="http://dieter-be.myopenid.com/"
 nickname="dieter"
 subject="comment 2"
 date="2011-02-16T21:32:04Z"
 content="""
thanks Joey,

is it possible to run some git annex command that tells me, for a specific directory, which files are available in an other remote? (and which remote, and which filenames?)
I guess I could run that, do my own policy thingie, and run `git annex get` for the files I want.

For your podcast use case (and some of my use cases) don't you think git [annex] might actually be overkill?  For example your podcasts use case, what value does git annex give over a simple rsync/rm script?
such a script wouldn't even need a data store to store its state, unlike git. it seems simpler and cleaner to me.

for the mpd thing, check http://alip.github.com/mpdcron/ (bad project name, it's a plugin based \"event handler\")
you should be able to write a simple plugin for mpdcron that does what you want (or even interface with mpd yourself from perl/python/.. to use its idle mode to get events)

Dieter
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 3"
 date="2011-03-16T03:01:17Z"
 content="""
Whups, the comment above got stuck in moderation queue for 27 days. I will try to check that more frequently.

In the meantime, I've implemented \"git annex whereis\" -- enjoy!

I find keeping my podcasts in the annex useful because it allows me to download individual episodes or poscasts easily when low bandwidth is available (ie, dialup), or over sneakernet. And generally keeps everything organised.
"""]]


code:
I have a remote rsync with gpg encryption and can restore without problems on my thinkpad (FreeBSD) - but not on my MacOSX:

$ git annex whereis DSC_7615.JPG
whereis DSC_7615.JPG (2 copies)
  	6855de17-c8fb-11e1-9948-f0def1c18073 -- thinkpad
   	e388bcf6-c8fc-11e1-a96f-6ffcbceb4af4 -- backup (rsync xxxx)
ok

$ git annex get --from backup DSC_7615.JPG
fatal: Could not switch to '../../../.git/annex/objects/Pw/XP/SHA256-s1028494--SHA256': No such file or directory

git-annex: <file descriptor: 6>: hGetLine: end of file
failed
git-annex: get: 1 failed

$ git annex version
git-annex version: 3.20120629
local repository version: 3


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
 nickname="Justin"
 subject="comment 1"
 date="2012-08-07T13:11:48Z"
 content="""
Do the remotes in `.git/config` look the same?
"""]]


code:
[[!comment format=mdwn
 username="hannes"
 ip="130.226.133.37"
 subject="they do"
 date="2012-08-07T14:06:55Z"
 content="""
I was also able to copy from the macosx to the rsync backup...
"""]]


code:
[[!comment format=mdwn
 username="hannes"
 ip="130.226.133.37"
 subject="also, fsck works"
 date="2012-08-07T14:12:20Z"
 content="""
git annex fsck -fbackup some-local-and-remote-file works fine.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 subject="comment 4"
 date="2012-08-07T17:53:35Z"
 content="""
I recently encountered this problem on OSX, and I'm pretty sure I fixed it. But I don't remember the details of how. I'm sorry I don't have more detail, but I recommend trying a newer version of git-annex.

And if the problem persists, --debug will show the git commands and should help identify the one that's being passed bad parameters.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 subject="remember now"
 date="2012-08-07T19:37:22Z"
 content="""
This was a bug in determining the git version at compile time. Fixed in commit bafc50e05e098234d2d22886085d9844fc763e0e which was included in version 3.20120721.
"""]]


code:
[[!comment format=mdwn
 username="hannes"
 ip="130.226.133.37"
 subject="awesome, thanks"
 date="2012-08-08T12:25:19Z"
 content="""
works with an updated git-annex!
"""]]


code:
hi,

i have a usecase that i think many people have. a cental server which should be non-bare to be able to browse the files also via webdav.
multiple clients behind nat so only pushes via xmpp are possible.

i set everything up without xmpp, it works but if files are updated, none of the clients gets a git push of course because ssh works only unidirectional.
i couldnt figure out how to set up xmpp push and have a non-bare central repo at the same time because i have to choose between ssh and xmpp git remote on the clients to the server.

thanks!


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 1"
 date="2013-07-25T17:05:00Z"
 content="""
Simply set up xmpp on your clients in addition to the central repository you have now. When one client pushes over ssh to the central repo, it will use XMPP to inform other clients of the change.
"""]]


code: Using webapp I cannot get rid of box.com repository, it is locked in 'cleaning out' state for more than two weeks. How can I remove the repository from the list?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-06-14T16:21:54Z"
 content="""
We need to first understand why it's not finishing the process. Does the box.com repository still contain files? You can find out by running the command: `git annex find --in box.com` (substitute the name of your remote for box.com as needed).
"""]]


code:
[[!comment format=mdwn
 username="tomas"
 ip="188.167.111.235"
 subject="comment 2"
 date="2013-06-16T13:32:31Z"
 content="""
The result of that command returned without writing any files.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 3"
 date="2013-06-17T17:34:04Z"
 content="""
In that case, I'd expect the assistant will notice it can remove the remote on the next full scan, and prompt you to finish the removal. Full scans are run in a few situations, but the only guaranteed way to run one is to stop and restart the assistant.

You could also enable debug mode when starting it, I'm sure the log would be useful if it somehow doesn't behave how it should.
"""]]


code:
[[!comment format=mdwn
 username="tomas"
 ip="188.167.111.235"
 subject="comment 4"
 date="2013-06-17T20:17:30Z"
 content="""
Hi, I cannot enable debug logging via webapp, because after restart the checkbox is unchecked again. I saw nothing at all in logs after restart, the box repo is still hanging there. I would prefer using command line to remove the last traces of this repository.
Another similar question - how do I unassociate two repositories? The web app warns me that about deleting repo, not sure if it would also wipe it out. Is there a CLI command to add and remove the associations (git remote rm?)? Overall great work, but I am lost with the webapp. It is good for learning the concepts, but I would love to know which commands it actually executes on the backend to make the gui -> cli transition easier.

Version: 4.20130516.1
Build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 5"
 date="2013-06-17T23:53:24Z"
 content="""
Use the --debug switch to enable debugging on startup.
"""]]


code:
git-annex can already be used to store data in several cloud services:
Amazon S3, rsync.net, Tahoe-LAFS, The Internet Archive.

I would like to support as many other cloud services as possible/reasonable.

* [[swift|todo/wishlist:_swift_backend]]
* Dropbox (I had been reluctant to go there due to it using a non-free client,
  which I have no interest in installing, but there is actually an API,
  and already a
  [haskell module to use it](http://hackage.haskell.org/package/dropbox-sdk).
  Would need to register for an API key.
  <http://www.dropbox.com/developers/start/core>.
  Annoyingly, Dropbox reviews each app before granting it production status.
  Whoops my interest level dropped by 99%.)

Post others in the comments. --[[Joey]]


code:
Hi,

Glad to see this project appears to be doing well.  I just wanted to mention that there's a similar project called cloudcmd that has many of the same use cases in spirit: https://github.com/briangu/cloudcmd

Cheers,
Brian


code:
I'd like to commit the current workdir state in direct mode in a script. This is what git annex watch does but I can not reliably start and stop git annex watch in a cron script.

Is this already possible with the current api or could we have something like git annex watch --once?

Thank you, Thomas Koch


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.21"
 subject="comment 1"
 date="2013-07-31T16:25:34Z"
 content="""
It should be sufficient to run: `git annex add; git annex sync`

That adds any new files and then commits those along with any changed or deleted files.
"""]]


code:
(I put this in a separate topic because it is a different question)

I still do not fully understand when (in direct mode!) a file is in the work copy and when in the repository.
Because what git-annex always refers to a "repository" is actually a repository (i.e., the server part for subversion) *and* a work copy (the client part in subversion), right? (except for bare repositories which have only the former)

When I change a file in the work copy, I still need to "git annex sync", right? But do I also need to "git annex sync --content" to move the changes from the work copy to the repository?

Because I have a headless server where the annex should also be accessed via samba. When files are added/changed, the annex clients do not see the changes until I manually do a couple of "git annex sync" and "git annex sync --content" (it's not deterministic yet for me how often).

Is this the intended behavior? If yes, is there a better way to automatically sync the changes on the headless server other than a cron job?