code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.236"
 subject="comment 1"
 date="2012-09-05T19:56:06Z"
 content="""
I'd recommend linking to the annexed file. Which is what `cp -a` does BTW. Then it'll just be a peice of content that is used by more than one filename, which git-annex handles well. However, if you then edited
one file, git-annex would not update the other link to point to the new content.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnh0pv5PyOJH8dHB1Ly2rHJM4nvVd17CZ0"
 nickname="Kevin"
 subject="comment 2"
 date="2012-09-05T19:57:36Z"
 content="""
Thank you very much! (for your answer, and for git annex)
"""]]

repo_name: kakafarm/git

license: unknown


code:
I've been doing a sort of experiment but I'm not sure if it's working or, really, how to even tell.

I have two macbooks that are both configured as clients as well as a USB HDD, an rsync endpoint on a home NAS, and a glacier endpoint.

For the purposes of this example, lets call the macbooks "chrissy" and "brodie". Chrissy's was initially configured with a remote for brodie with the url as

    ssh://Brodie.88195848.members.btmm.icloud.com./Users/akraut/Desktop/annex

This allows me to leverage the "Back To My Mac" free IPv6 roaming I get from Apple.  Now, occasionally, that dns resolution fails. Since I'm frequently on the same network, I can also use the mDNS address of brodie.local. which is much more reliable.

So my brilliant/terrible idea was to put this in my git config:

    [remote "brodie"]
    	url = ssh://Brodie.88195848.members.btmm.icloud.com./Users/akraut/Desktop/annex
    	fetch = +refs/heads/*:refs/remotes/brodie/*
    	annex-uuid = BF4BCA6D-9252-4B5B-BE12-36DD755FAF4B
    	annex-cost-command = /Users/akraut/Desktop/annex/tools/annex-cost6.sh Brodie.88195848.members.btmm.icloud.com.
    [remote "brodie-local"]
    	url = ssh://brodie.local./Users/akraut/Desktop/annex
    	fetch = +refs/heads/*:refs/remotes/brodie/*
    	annex-uuid = BF4BCA6D-9252-4B5B-BE12-36DD755FAF4B
    	annex-cost-command = /Users/akraut/Desktop/annex/tools/annex-cost.sh brodie.local.

Is there any reason why I shouldn't do this? Is annex smart enough to know that it can reach the same remote through both urls? Will the cost calculations be considered and the "local" url chosen if it's cost is less than the other?

(I posted the annex-cost.sh stuff at [[forum/Calculating Annex Cost by Ping Times]].)


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.246.6"
 subject="comment 1"
 date="2012-12-30T04:19:48Z"
 content="""
Yes, this is absolutely supported. If it's trying to `get` a file and it fails to access the first remote, it'll even automatically fall back to using the second. I do similar things with my own remote, for example I have a foo that uses ssh with a foo-local that uses NFS.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="multiple ways to access a special remote?"
 date="2013-07-10T05:52:28Z"
 content="""
The use case is like this: an external USB drive could be accessed both locally (as a special directory remote) or it could be mounted remotely (then it would need to be an rsync special remote).
Is there a way to handle this?

Failing that, can one switch the same remote from being a directory to an rsync without having to --move all of the content?

Finally, what's the best way to check directory remote's directory= parameter?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.235"
 subject="comment 3"
 date="2013-07-10T16:38:04Z"
 content="""
@Michael, like I said above, this use case is completely supported. That's why git-annex uses UUIDs to uniquely identify repositories, no matter where or how many urls are used for them.

Just set up the remotes you need, and if they end up pointing to the same repository by different routes, git-annex will automatically notice.

---

The directory= parameter used when initializing a directory remote is only used to set up the remote in the .git/config file. It is not stored anywhere else, since the directory could be mounted at different locations on different computers, eg when a drive is moved between computers.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="comment 4"
 date="2013-07-10T21:26:06Z"
 content="""
Very cool, thanks Joey.
"""]]


code:
I've added an arm build to the autobuilds in [[install/Linux_standalone]].

I'm curious to see how this works out. I tried to make it as self-contained
as possible. It should work even on systems that do not use glibc, as long
as the kernel is new enough for the glibc included in it, and supports the
arm EABI.

If it seems sufficiently useful, I might try to add the webapp to the
build, which would be somewhat complicated, but doable (since I'm building
using qemu, it can run a build on amd64 first to get the TH splices).

--[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmH9ARM62C6zcEpzh2muCs4wq-GkLRntgQ"
 nickname="Randy"
 subject="raspbian"
 date="2014-04-16T10:59:10Z"
 content="""
This works fairly well for me on Raspian. However I am getting the same error as Justin.

    ERROR: ld.so: object '/usr/lib/arm-linux-gnueabihf/libcofi_rpi.so' from /etc/ld.so.preload cannot be preloaded: ignored.

I'm ignoring the errors for now, but it's a lot of noise that actually makes it slightly difficult to see the important output.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.191"
 subject="comment 11"
 date="2014-04-17T19:39:41Z"
 content="""
I can't see any good way to prevent ld-linux.so from preloading things listed in `/etc/ld.so.preload`. I don't know why raspbian wants to preload that -- probably for optimisation purposes?

I could modify the ld-linux.so shipped in the git-annex tarball, but that way lies pointless complication..
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmH9ARM62C6zcEpzh2muCs4wq-GkLRntgQ"
 nickname="Randy"
 subject="comment 12"
 date="2014-04-18T13:38:13Z"
 content="""
There is no need to modify anything here. The errors were mostly just confusing at first and made me wonder if it was working properly. It is indeed working perfectly! Thanks!!!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="works on raspbian"
 date="2013-12-16T23:26:51Z"
 content="""
This build has been verified to work on raspbian.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="works on Synology NAS"
 date="2013-12-17T02:52:22Z"
 content="""
Using latest build (fixed some issues).
"""]]


code:
[[!comment format=mdwn
 username="http://julien.lefrique.name/"
 nickname="jlefrique"
 subject="QNAP"
 date="2013-12-17T07:57:37Z"
 content="""
Hi Joey,

Thank you for taking the time to setup an ARM build. I am trying to run the last standalone build on a QNAP TS-219PII. I get the following error. Do you have any ideas?

    $ uname -a
    Linux willow 2.6.33.2 #1 Fri Mar 1 04:41:48 CST 2013 armv5tel unknown
    $ ./runshell
    $ cd ../annex
    $ git annex version
    git-annex version: 5.20131216-g07252d6
    build flags: Assistant Pairing S3 Inotify DBus XMPP Feeds Quvi TDFA CryptoHash
    key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
    remote types: git gcrypt S3 bup directory rsync web glacier hook
    local repository version: 3
    default repository version: 3
    supported repository versions: 3 5
    upgrade supported from repository versions: 0 1 2 4
    $ git annex info
    repository mode: indirect
    trusted repositories: fatal: cannot get RLIMIT_NOFILE: Bad address
    fatal: cannot get RLIMIT_NOFILE: Bad address
    fatal: cannot get RLIMIT_NOFILE: Bad address
    fatal: cannot get RLIMIT_NOFILE: Bad address
    fatal: cannot get RLIMIT_NOFILE: Bad address
    fatal: cannot get RLIMIT_NOFILE: Bad address

I am still using the version 3.8.3 of the QNAP distribution. I will try to update to the last version (4.0.5) to see if it helps.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="RLIMIT_NOFILE"
 date="2013-12-17T16:05:12Z"
 content="""
This is clearly failing when it runs git. Looking at the git source code, it does have a `RLIMIT_NOFILE` ifdef, so I could disable this, but would have to maintain my own build of git.

What kernel version is this? Upgrading is likely to fix it.
"""]]


code:
[[!comment format=mdwn
 username="http://julien.lefrique.name/"
 nickname="jlefrique"
 subject="QNAP kernel"
 date="2013-12-17T16:27:21Z"
 content="""
It's kernel 2.6.33.2 (see uname in my previous message).
A new version with kernel 3.4.x is available, I will try to update when the data on my NAS will be (more) safely backup-ed.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="Re: RLIMIT_NOFILE"
 date="2013-12-19T02:10:45Z"
 content="""
I've passed this problem on to the git developers, and they have quickly developed a patch that will solve it, so I'll just wait for the next git release.
"""]]


code:
[[!comment format=mdwn
 username="http://julien.lefrique.name/"
 nickname="jlefrique"
 subject="Thanks"
 date="2013-12-19T07:20:31Z"
 content="""
That's perfect!
On my side, I upgraded to firmware 4.0.2 but the kernel is still the same.
The kernel 3.4.x is available in the beta of QTS 4.1.0, I will wait the official release to upgrade.
"""]]


code:
[[!comment format=mdwn
 username="https://renaud.casenave.fr/"
 ip="126.10.66.235"
 subject="comment 8"
 date="2014-02-17T13:15:24Z"
 content="""
Hi,

You might be happy to know git-annex works well on sailfishOS, even the webapp.
It does segfault when trying to setup a xmpp account, though.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmUJBh1lYmvfCCiGr3yrdx-QhuLCSRnU5c"
 nickname="Justin"
 subject="comment 9"
 date="2014-03-01T19:13:16Z"
 content="""
This seems to work great for me on Raspbian, except I get many errors of the form

    ERROR: ld.so: object '/usr/lib/arm-linux-gnueabihf/libcofi_rpi.so' from /etc/ld.so.preload cannot be preloaded: ignored.

I've tried, but I haven't been able to figure out why ld doesn't like this particular lib.  I can keep ignoring the error, but help would certainly be appreciated!
"""]]


code:
I'm soliciting ideas for new small features that let git-annex do things that currently have to be done manually or whatever.

Here are a few I've been considering:

---

* --numcopies would be a useful command line switch.
  > Update: Added. Also allows for things like `git annex drop --numcopies=2` when in a repo that normally needs 3 copies, if you need
  > to urgently free up space.
* A way to make `drop` and other commands temporarily trust a given remote, or possibly all remotes.

Combined, this would allow `git annex drop --numcopies=2 --trust=repoa --trust=repob` to remove files that have been replicated out to the other 2 repositories, which could be offline. (Slightly unsafe, but in this case the files are podcasts so not really.)

> Update: done --[[Joey]]

---

[[wishlist:_git-annex_replicate]] suggests some way for git-annex to have the smarts to copy content around on its own to ensure numcopies is satisfied. I'd be satisfied with a `git annex copy --to foo --if-needed-by-numcopies`

  > Contrary to the "basic" solution, I would love to have a git annex distribute which is smart enough to simply distribute all data according to certain rules. My ideal, personal use case during the next holidays where I will have two external disks, several SD cards with 32 GB each and a local disk with 20 GB (yes....) would be:

    cd ~/photos.annex # this repository does not have any objects!
    git annex inject --bare /path/to/SD/card  # this adds softlinks, but does **not** add anything to the index. it would calculate checksums (if enabled) and have to add a temporary location list, though
    git annex distribute # this checks the config. it would see that my two external disks have a low cost whereas the two remotes have a higher cost.
     # check numcopies. it's 3
     # copy to external disk one (cost x)
     # copy to external disk two (cost x)
     # copy to remote one (cost x * 2)
     # remove file from temporary tracking list
    git annex fsck # everything ok. yay!

Come to think of it, the inject --bare thing is probably not a microfeature. Should I add a new wishlist item for that? -- RichiH

> I've thought about such things before; does not seem really micro and I'm unsure how well it would work, but it would be worth a [[todo]]. --[[Joey]]

>> Update: Done as --auto. --[[Joey]]

---

Along similar lines, it might be nice to have a mode where git-annex tries to fill up a disk up to the `annex.diskreserve` with files, preferring files that have relatively few copies. Then as storage prices continue to fall, new large drives could just be plopped in and git-annex used to fill it up in a way that improves the overall redundancy without needing to manually pick and choose.

> Update: git annex get --auto basically does this; you can tune
> --numcopies on the fly to make it get more files than needed by the
> current numcopies setting. --[[Joey]]

---

If a remote could send on received files to another remote, I could use my own local bandwith efficiently while still having my git-annex repos replicate data. -- RichiH

---

Really micro:

    % grep annex-push .git/config
        annex-push = !git pull && git annex add . && git annex copy . --to origin --fast --quiet && git commit -a -m "$HOST $(date +%F--%H-%M-%S-%Z)" && git push
    %

-- RichiH
--[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 1"
 date="2011-06-01T17:36:50Z"
 content="""
I've been longing for an automated way of removing references to a remote assuming I know the exact uuid that I want to remove. i.e. I have lost a portable HDD due to a destructive process, I now want to delete all references to copies of data that was on that disk. Unless this feature exists, I would love to see it implemented.
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 2"
 date="2011-06-01T20:24:33Z"
 content="""
@jimmy [[walkthrough/what_to_do_when_you_lose_a_repository]].. I have not seen a convincing argument that removing the location tracking data entirely serves any purpose
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnpdM9F8VbtQ_H5PaPMpGSxPe_d5L1eJ6w"
 nickname="Rafaël"
 subject="git annex unlock --readonly"
 date="2011-06-02T11:34:42Z"
 content="""
This was already asked [here](http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=606577), but I have a use case where I need to unlock with the files being hardlinked instead of copied (my fs does not support CoW), even though 'git annex lock' is now much faster ;-) . The idea is that 1) I want the external world see my repo \"as if\" it wasn't annexed (because of its own limitation to deal with soft links), and 2) I know what I do, and am sure that files won't be written to but only read.

My case is: the repo contains a snapshot A1 of a certain remote directory. Later I want to rsync this dir into a new snapshot A2. Of course, I want to transfer only new or changed files, with the --copy-dest=A1 (or --compare-dest) rsync's options. Unfortunately, rsync won't recognize soft-links from git-annex, and will re-transfer everything.

Maybe I'm overusing git-annex ;-) but still, I find it is a legitimate use case, and even though there are workarounds (I don't even remember what I had to do), it would be much more straightforward to have 'git annex unlock --readonly' (or '--readonly-unsafe'?), ... or have rsync take soft-links into account, but I did not see the author ask for microfeatures ideas :) (it was discussed, and only some convoluted workarounds were proposed). Thanks.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnpdM9F8VbtQ_H5PaPMpGSxPe_d5L1eJ6w"
 nickname="Rafaël"
 subject="git annex unused"
 date="2011-06-02T11:55:58Z"
 content="""
Before dropping unsused items, sometimes I want to check the content of the files manually.
But currently, from e.g. a sha1 key, I don't know how to find the corresponding file, except with
'find .git/annex/objects -type f -name 'SHA1-s1678--70....', wich is too slow (I'm in the case where \"git log --stat -S'KEY'\"
won't work, either because it is too slow or it was never commited). By the way,
is it documented somewhere how to determine the 2 (nested) sub-directories in which a given
(by name) object is located?

So I would like 'git-annex unused' be able to give me the list of *paths* to the unused items.
Also, I would really appreciate a command like 'git annex unused --log NUMBER [NUMBER2...]' which would do for me the suggested command
\"git log --stat -S'KEY'\", where NUMBER is from the 'git annex unused' output.
Thanks.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnpdM9F8VbtQ_H5PaPMpGSxPe_d5L1eJ6w"
 nickname="Rafaël"
 subject="git annex unused"
 date="2011-06-02T19:51:49Z"
 content="""
ps: concerning the command 'find .git/annex/objects -type f -name 'SHA1-s1678--70....' from my previous comment, it is \"significantly\" faster to search for the containing directory which have the same name: 'find .git/annex/objects -maxdepth 2 -mindepth 2 -type d -name 'SHA1-s1678--70....'. I am just curious: what is the need to have each file object in its own directory, itself nested under two more sub-directories?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnpdM9F8VbtQ_H5PaPMpGSxPe_d5L1eJ6w"
 nickname="Rafaël"
 subject="git annex fetch"
 date="2011-07-03T14:39:41Z"
 content="""
I'm not sure it is worth adding a command for such a small feature, but I would certainly use it: having something like \"git annex fetch remote\" do \"git fetch remote && git annex copy --from=remote\", and \"git annex push remote\" do \"git push remote && git annex copy --to=remote\". And maybe the same for a pull operation?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnpdM9F8VbtQ_H5PaPMpGSxPe_d5L1eJ6w"
 nickname="Rafaël"
 subject="git annex fetch"
 date="2011-07-03T17:57:00Z"
 content="""
My last comment is a bit confused. The \"git fetch\" command allows to get all the information from a remote, and it is then possible to merge while being offline (without access to the remote). I would like a \"git annex fetch remote\" command to be able to get all annexed files from remote, so that if I later merge with remote, all annexed files are already here. And \"git annex fetch\" could (optionally) call \"git fetch\" before getting the files.

It seems also that in my last post, I should have written \"git annex get --from=remote\" instead of \"git annex copy --from=remote\", because \"annex copy --from\" copies all files, even if the local repo already have them (is this the case? if yes, when is it useful?)
"""]]


code:
Over the last few weeks i've been working on a seperate python nntp project. But decided that it would be fun to make it usable with git-annex.

The code is almost complete. One major feature left(make it usable without a mysql/sqlite database, and without fetching headers). And of course a lot of bugfixes.

I'm wondering if there is any interest in this special remote hook.

The thought is, buy unlimited usenet account, and you can have a git-annex repository without any upper size limit. You want 100GB, well that'll be 10$ a month, want 100000GB, well that'll be 10$ a month.

Of course there are plenty of caveats(retention on server, it is doubtfull the data will stay up for more than 4 years.). Because the code is in python3 i had to use a lot of external tools (uudeview, par2, ydecode/yydecode) and parse their output. Which is a horrid horrid solution. Since the code is nearing "beta" stage i might see if i can improve on that(possibly with ctypes), if there is any interest.

Currently working on freebsd, ubuntu, debian, mac os x(10.7, not tested in 10.8)

The last feature to be done (and this is UUUUUGLY) is to make it get nzb files from a source like this: http://nzbindex.nl/search/?q=GPGHMACSHA (yes, that is my git-annex repo) instead of from a mysql database.

So, this is a call to see if anyone is interested, and if anyone proficient in *NIX want to test it out.

Sincerely
Tobias


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-05-19T20:49:06Z"
 content="""
I think it's a neat idea. I can't see using it for my own data due to retention, but if I had a use case where I wanted to spread data around and didn't care if it vanished after X months, I can see using it.

I think you should publish the details, if getting it to a state others can use is not too much work,
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8"
 nickname="develop"
 subject="comment 2"
 date="2013-09-11T07:46:07Z"
 content="""
Only played a tiny bit with this since the initial post.

I've decided that sql backend should be dropped entirely. Mostly the SQL backend was there because the code does something else entirely, and this was just a test.

But since there has been no interest it will probably take ages before i actually get this finished :)

"""]]


code:
Hi,

I wondered if it's possible to have a non-bare repo on a cloud remote. I want to have some sort of Dropbox-like workflow were I simply put stuff into a directory on my drive and it gets synced via SSH to a central VPS so I can also always access it via HTTP, for example.

Regards,
Lukas


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.1.10"
 subject="comment 1"
 date="2013-07-22T22:04:53Z"
 content="""
Sure you can.. That's how <http://downloads.kitenet.net/> works, which is where people download git-annex builds from!

[[tips/setup_a_public_repository_on_a_web_site]]
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawniayrgSdVLUc3c6bf93VbO-_HT4hzxmyo"
 nickname="Tobias"
 subject="comment 2"
 date="2013-08-08T19:05:07Z"
 content="""
Seems that I don't get it... Here is what I did:

First, I create a new repo with the assistant on `user@laptop ~/temp/annex-test`, then:

    user@laptop ~/temp/annex-test % echo test1 > test1
    user@laptop ~/temp/annex-test % echo test2 > test2

    user@server ~/tmp % mkdir annex-test
    user@server ~/tmp % cd annex-test
    user@server ~/tmp/annex-test % git init
    Initialized empty Git repository in /home/tobru/tmp/annex-test/.git/
    user@server ~/tmp/annex-test % git annex init
    init  ok
    (Recording state in git...)
    user@server ~/tmp/annex-test % mv .git/hooks/post-update.sample .git/hooks/post-update
    user@server ~/tmp/annex-test % cat .git/hooks/post-update
    #!/bin/sh
    #
    # An example hook script to prepare a packed repository for use over
    # dumb transports.
    #
    # To enable this hook, rename this file to \"post-update\".

    exec git update-server-info
    user@server ~/tmp/annex-test % chmod +x .git/hooks/post-update
    user@server ~/tmp/annex-test % echo '#!/bin/sh\ngit annex merge' > .git/hooks/post-receive
    user@server ~/tmp/annex-test % chmod +x .git/hooks/post-receive

    user@laptop ~/temp/annex-test % git remote add server ssh://user@server/home/tobru/tmp/annex-test
    user@laptop ~/temp/annex-test % git annex assistant --stop
    user@laptop ~/temp/annex-test % git annex sync
    commit
    ok
    pull server
    warning: no common commits
    remote: Counting objects: 5, done.
    remote: Compressing objects: 100% (3/3), done.
    remote: Total 5 (delta 1), reused 0 (delta 0)
    Unpacking objects: 100% (5/5), done.
    From ssh://server.tobru.local/home/tobru/tmp/annex-test
     * [new branch]      git-annex  -> server/git-annex
    ok
    (merging server/git-annex into git-annex...)
    (Recording state in git...)
    push server
    Counting objects: 27, done.
    Delta compression using up to 4 threads.
    Compressing objects: 100% (20/20), done.
    Writing objects: 100% (25/25), 2.36 KiB, done.
    Total 25 (delta 2), reused 0 (delta 0)
    remote: merge git-annex (merging synced/git-annex into git-annex...)
    remote: ok
    To ssh://tobru@server.tobru.local/home/tobru/tmp/annex-test
     * [new branch]      git-annex -> synced/git-annex
     * [new branch]      master -> synced/master
    ok
    user@laptop ~/temp/annex-test % git annex assistant

Now I expect the files `test1` and `test2` to be at `user@server ~/tmp/annex-test`. But they are not displayed.
What am I doing wrong? The main thing I wan't to achieve: Push the files from `laptop` to `server` without adding `laptop` as remote on the server.
"""]]


code:
I have set up an annex on a remote machine and I am connecting via ssh.  But, since it is a managed machine, I installed the git-annex binary in my own ~/bin.  Well, when I try
$git annex sync

I get:
  $git annex sync
(merging origin/git-annex into git-annex...)
(Recording state in git...)
bash: git-annex-shell: command not found

  Remote origin does not have git-annex installed; setting annex-ignore
commit  ok
pull origin

merge: refs/remotes/origin/master - not something we can merge

merge: refs/remotes/origin/synced/master - not something we can merge
failed
git-annex: sync: 1 failed

The git remote -v looks correct.  So, how do I tell git annex on my local machine where to use $HOME/bin in PATH on the remote machine when syncing  with remotes?


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawk3HGoDpnOPob5jOjvIootmkve1-nCpRiI"
 nickname="Kalle"
 subject="Declare path on first line of bashrc?"
 date="2014-02-15T18:36:35Z"
 content="""
If i don't misremember some systems including Debian require you to set the path on the very first line of .bashrc for it to work. Can't remember why just now.

in other words paste the following into the very first line of `$HOME/.bashrc`

`PATH=$PATH:$HOME/bin:$HOME/bin/git-annex.linux`

Modify the line above if you haven't installed to `~/bin/git-annex.linux`

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn3rK4VDzxyhmrIc18z7F5OuXvEbUsgUac"
 nickname="Srinath"
 subject="Progress, but still issues"
 date="2014-02-17T04:09:23Z"
 content="""
\"I now realize that the git annex system still requires the standard \"add\" and \"commit\" process. But I'm still getting:

$git annex sync
commit  ok
pull origin
remote: Counting objects: 37, done.
remote: Compressing objects: 100% (35/35), done.
remote: Total 36 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (36/36), done.
From ssh://stampede.tacc.utexas.edu/work/02463/srinathv/cesm1_3_beta07/scripts
 * [new branch]      master     -> origin/master

merge: refs/remotes/origin/synced/master - not something we can merge
failed
push origin
Counting objects: 11, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (8/8), 736 bytes | 0 bytes/s, done.
Total 8 (delta 3), reused 1 (delta 0)
To ssh://srinathv@stampede.tacc.utexas.edu/work/02463/srinathv/cesm1_3_beta07/scripts
 * [new branch]      git-annex -> synced/git-annex
 * [new branch]      master -> synced/master
ok
git-annex: sync: 1 failed

So the fails appear, and the suggestion of \"export PATH\" placement did not help, though appreciated.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn3rK4VDzxyhmrIc18z7F5OuXvEbUsgUac"
 nickname="Srinath"
 subject="no more issue"
 date="2014-02-17T04:25:31Z"
 content="""
After doing 1 more sync, the error messages are now gone.  I wonder if the refs directory needed to be synched correctly.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.172"
 subject="comment 4"
 date="2014-02-20T19:41:05Z"
 content="""
@Srinath I wonder if the machine you are running git-annex sync on is a Windows machine? I have seen that \" refs/remotes/origin/synced/master - not something we can merge\" intermittently when testing on Windows, but not other times.

(It's not related to the original PATH configuration problem on your server.)
"""]]


code: I have 2 computers successfully exchanging files, but the contexts are incomplete.  Used text files, documents, mp3s as tests. Both computers report in the webapp that they are synched and file names are there.  But file sizes are wrong by 75% less and contents are unusable.  Is there something in the config?  Both machines on LinuxMint 13.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.235"
 subject="comment 1"
 date="2013-07-11T16:06:59Z"
 content="""
That certainly shouldn't be happening!

Are these computers syncing via local pairing, or are you using a transfer remote, and if so, which one?
"""]]


code:
[[!comment format=mdwn
 username="trilobite"
 ip="71.17.134.14"
 subject="comment 2"
 date="2013-07-11T23:23:41Z"
 content="""
I did this with local synch in home lan.  Filenames do travel both directions.  Just not contents.  They are identifed as symlinks rather than files.
"""]]


code:
I'm curious how other people are using git-annex:

 * A single annex containing everything they want to keep track of (albeit with some files only available in some remotes).
 * Several annexes for different purposes.

I'm mostly asking because I don't want to get too far into using git-annex without thinking clearly about which is the most suitable choice for me.  At present I have things split into two annexes, one for things I really wouldn't want to lose (and some of which are somewhat private), and another for comparatively throwaway items.  However, given the possibility of setting different annex.numcopies on different file globs or directories, it seems like it might be kind of silly to make my life more complicated by having more than one annex.

Primarily, I'm curious if there are some implications of the decision that I haven't considered.  It seems like an obvious point that one would want different annexes is if they're used in distinctly different security-level environments (e.g. I'm happy to copy my FLAC (music) files over to my work computer which lives at the office, but I wouldn't want to copy personal financial documents - or even the filenames - there.)

-Mike


code:
Just a question about other peoples' annex usage.  I've got one uber-annex (in ~/annex of course) where I keep ALL THE THINGS, movies, music, pictures, whatever.

It's very big, of course.  And I worry about doing anything unusual with it (like trying out direct mode) because, hey, that's all my stuff.

Like, just now, I tried flipping it into direct mode, and a handful of files stayed symlinks, which weirded me out.  (Maybe because I had the assistant running while I switched it, and that caused chaos?)  Then I flipped it back to indirect, and a bunch of files were left un-tracked, as real files rather than links!  I kind of freaked out, and did a big "git annex add" to add them all back. I haven't left a bug report because it's not something I want to try again experimentally to reproduce, I just want my repo back, you know?

Anyways, this kind of thing might not happen if I kept multiple small repos.  I was wondering if anybody else worked that way, and what advantages/disadvantages you've found compared to having one super-repo.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn7gQ1zZDdWhXy9H51W2krZYShNmKL3qfM"
 nickname="Karsten"
 subject="comment 1"
 date="2013-01-21T21:25:12Z"
 content="""
That's an interesting question. Personally, I have two annexes. One for my \"digital library\": movies, music, ebooks, downloads, everything I want to keep but was not created by me. I don't want to loose them because getting them back would be expensive and cumbersome. The other is my personal archive for Photos and Videos I've taken, which are valuable as they contain many memories which would be impossible to recreate if lost. With the first one, I'm experimenting quite a lot, with the second one I'm very conservative. Hope that helps.
"""]]


code:
Consider the following scenario:

- A Server which holds the full repository
- A Laptop with a small disk, cloned the repository

Now when i am online i'd just like to mount the repository from the server on over to the client, shadowing the local repository. But when offline and the server is not
mounted the local repository takes place.

The Question is now what would be a viable concept to get this right. Especially will the assistant become confused when it runs on the server side and locally on the laptop
while the mount is in effect. Would it be sensible not to mount the whole repository tree but only parts like `.git/annex` or `.git/annex/objects`?

Not tried this yet, but I am wondering whats the most viable approach would be.


code:
[[!comment format=mdwn
 username="https://openid.stackexchange.com/user/e65e6d0e-58ba-41de-84cc-1f2ba54cf574"
 nickname="Mica Semrick"
 subject="sounds like a bad idea... but..."
 date="2014-06-24T04:18:51Z"
 content="""
From the few details you give, this sounds like a bad idea. git-annex tells different repos apart by using a UUID. You can see this UUID in `.git/config`.

But if you are really insistent on doing this, you could always system link the folder to the spot you want it, then write a script that mounts the server in an arbitrary location and redoes the system link. I'd try this on a copy of the data or with some data that doesn't matter before going for it.
"""]]


code:
[[!comment format=mdwn
 username="cehteh"
 ip="217.8.62.137"
 subject="comment 2"
 date="2014-06-24T14:03:01Z"
 content="""
I know that this is a bad idea right now when done in a naive approach, I am wondering if git-annex could be made aware of this case and how it could be implemented.
A immature idea for example would be only to overmount the .git/annex/objects folder and let the git-annex check if the objects lay on a different device, if so, refuse to do operation on the object store.

I started this thread to figure out if there is some interest in such functionality and if so, polishing it out to see how it could be done.
"""]]


code:
[[!comment format=mdwn
 username="http://ypid.wordpress.com/"
 ip="213.153.84.215"
 subject="Already on todo list"
 date="2014-06-24T16:58:05Z"
 content="""
As is appears to me, the feature you are suggesting is already on \"the todo list\" as [[union_mounting|todo/union_mounting]]. I would also really appreciate this feature …
"""]]


code:
hi

i have a repository that is 30 gb large. i would like to sync some content onto my android phone. take mp3's for example. i would like to see whole content on android but my phone does not have that much flash space. i would like to manualy select what folders, files will be copied onto phone.

this use case may not even refer strictly to android devices but even pc's. for exaple i have small ssd drive on laptop. whole contentis kept on some hudge  raid array. i will see whole directory structure but whenever i want a file i will just shedule it for download.

is it possible ? or maybe i just can not find the answer on project page ?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 1"
 date="2013-12-29T21:57:05Z"
 content="""
You can either put a repository in manual mode, which requires you run `git annex get` etc to get just the files you want, or you can find or write a [[preferred_content]] expression that matches only the files you want in a repository, and then the assistant will make it contain only those.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkNE-H4vEcbcGndxq5daT8qUb7yIf7r1OE"
 nickname="Łukasz"
 subject="comment 2"
 date="2013-12-30T22:13:15Z"
 content="""
thx for reply
"""]]


code:
Hi,

I just was setting um my git-annex repository and started to sync my whole stuff in it.

Background: I have choosen git-annex to sync my whole stuff (pictures, mp3s, documents, etc) between my pc, notebook and a home-server

My Problems:
1) When I'm starting the git-annex deamon, the "Performing startup scan" message occurs for hours
2) git-annex synchronizes folders from the server which already on my pc, and that every time I restart the deamon on client

My Questions:
For 1) is git-annex when running one repository suitable to manage > 100gb and > 50000 files?
For 2) do I have to wait until every tasks are completed (everything is committed) to get rid of multiple downloads of the same folders/files
3) what is the best schema to sync between >2 devices should I use a Mesh or Star Schema (where my server is in the middle)

Thank You in advance!
Regards J


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-06-11T15:06:58Z"
 content="""
It sounds like you are using git-annex to sync a *lot* of files. The startup scan is a basic check that there are no new, deleted, or modified files since the last time it ran. This requires a little work, like statting the files, but won't take long for reasonable numbers of files.

The size of files doesn't affect the length of the scan. I have a repository with on the order of 50k files, and the startup scan takes only a few minutes. One thing that could matter is that my repository is in indirect mode. Direct mode is less efficient. You could try to switch your repository to indirect mode: `git annex indirect` (you can always switch back: `git annex direct`)

It would be possible to disable that scan, but at the expense of not being able to sync changes made while git-annex is not running. There's also a trick you can use: Start the assistant running in a subdirectory and it will only scan that subdirectory (it will only notice new files in that subdirectory too..)

I don't quite understand what you mean with problem #2. If files were repeatedly being uploaded or downloaded, that have already been sent, that would be a bug. Please file a bug report with full debug logs if that is the case.

Which topology is best? I think the best way is to start with the one you like, and if it doesn't work well, add more links between repositories. A star topology will certainly work ok. A mesh can work ok but can be hard to maintain.
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/bBy7WkgQicYHIiiyj.Vm0TcMbxi2quzbPFef#6f9f7"
 nickname="Frederik Vanrenterghem"
 subject="comment 2"
 date="2013-06-12T00:47:21Z"
 content="""
I have witnessed that second problem as well, to the point where I've stopped autostarting (and hence using) git annex for the moment. I'll try to get some debugging data.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.146"
 subject="comment 4"
 date="2014-03-05T22:06:00Z"
 content="""
@maurizio, that's a good motivating example.

So, I have made `git config annex.startupscan false` disable the startup scan (except for a minor tree walk that needs to be done to get inotify working).

Of course, if you set that, the assistant won't notice any changes that are made when it's not running. It would work well to set it if the assistant is started at boot and left running until shutdown.

My goal for git-annex is for it to scale to how ever many files git scales to. So I try to always make git be the limiting factor. I feel that git scales fairly well into the 100k file range.
"""]]


code:
[[!comment format=mdwn
 username="zardoz"
 ip="134.147.14.84"
 subject="comment 5"
 date="2014-07-08T12:26:58Z"
 content="""
Something to keep in mind for some point in the future: btrfs supports an efficient method by which one can inquire exactly which files have changed since a specified point in time, where the point in time is measured by a unit called «generation».

A program can request the current «generation» index, and later ask btrfs which files have changed after that index, without needing to walk the file-system. This is currently implemented in the user-space program «btrfs filesystem find-new». While it’s called «find-new», this not only find newly created files, but also changed files.

Currently, it seems one needs to be super-user to use «find-new», because it lists changed files for complete subvolumes, but since generations are stored per-file, in the future there will likely be a user-space method for regular users.
"""]]


code:
This works with bind-mount, I might try with softlinks as well.

Going through git's data on push/pull can take ages on a spindle disk even
if the repo is rather small in size. This is especially true if you are
used to ssd speeds, but ssd storage is expensive. Storing the annex objects
on a cheap spindle disk and everything else on a ssd makes things a _lot_
faster.

> Update: git-annex supports `.git/annex/` being moved to a different disk
> than the rest of the repisitory, but does *not* support individual
> subdirectories, like `.git/annex/objects/` being on a different disk
> than the main `.git/annex/` directory. --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2011-04-02T17:48:29Z"
 content="""
Either option should work fine, but git gc --aggressive will probably avoid most of git's seeking.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 2"
 date="2011-04-02T21:34:24Z"
 content="""
I'll give it a try as soon as I get rid of this:

    % git annex fsck
fatal: index file smaller than expected
fatal: index file smaller than expected
    % git status
fatal: index file smaller than expected
    %

And no, I am not sure where that is coming from all of a sudden... (it might have to do with a hard lockup of the whole system due to a faulty hdd I tested, but I didn't do anything to it for ages before that lock-up. So meh. Also, this is prolly off topic in here)

Richard
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 3"
 date="2011-04-03T01:48:57Z"
 content="""
For future reference, git can recover from a corrupted index file with `rm .git/index; git reset --mixed`.

Of course, you lose any staged changes that were in the old index file, and may need to re-stage some files.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 4"
 date="2011-04-03T09:03:22Z"
 content="""
Thanks a lot. I tried various howtos around the net, but none of them worked; yours did. (I tried it in one of the copies of the broken repo which I keep around for obvious reasons).
"""]]


code:
[[!comment format=mdwn
 username="stoile"
 ip="2a01:198:242:0:219:66ff:fef2:c021"
 subject="Error with .git/annex as a symlink"
 date="2013-11-16T12:20:58Z"
 content="""
I had my git annex directory on a ssd and .git/annex on a hdd (using a symlink). I got errors when adding files because a move failed. I forgot to copy the error message. When I had .git/annex/objects on a hdd, everything worked as expected, there were no problems.

So what exactly will go wrong if I only have this one subdirectory on another filesystem?
"""]]


code:
[[!comment format=mdwn
 username="stoile"
 ip="2a01:198:242:0:219:66ff:fef2:c021"
 subject="Bug report answering my question"
 date="2013-11-17T10:28:40Z"
 content="""
My question was answered in <http://git-annex.branchable.com/bugs/not_possible_to_have_annex_on_a_separate_filesystem/>.
"""]]


code:
    [2014-05-18 22:38:21 SAST] main: starting assistant version 5.20140517-g0aed6d9
    Already up-to-date.

    (scanning...) [2014-05-18 22:38:22 SAST] Watcher: Performing startup scan
    gpg: WARNING: unsafe permissions on homedir `/var/folders/m6/zkd11n111m38ff37zbtgq0lr0000gp/T/git-annex-gpg.tmp.0'
    gpg: keyring `/Applications/git-annex.app/Contents/MacOS/trustedkeys.gpg' created
    gpg: Signature made Sun May 18 04:16:11 2014 SAST using DSA key ID XXXXXXXX
    gpg: Can't check signature: public key not found
    (started...)

And still facing:

    git-annex-remote-skydrive is not installed in PATH (/Applications/git-annex.app/Contents/MacOS/bundle:/Applications/git-annex.app/Contents/MacOS/bundle:/usr/bin:/bin:/usr/sbin:/sbin)


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkAUMhKOSkh9JaBA6xst3XxQIIsDEq5Zd4"
 nickname="Ovidiu"
 subject="comment 1"
 date="2014-06-21T13:07:51Z"
 content="""
What exactly are safe permissions?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.55"
 subject="comment 2"
 date="2014-07-03T19:49:03Z"
 content="""
That error message seems to occur when git-annex is trying to upgrade itself to a newer version, but for some reason the temporary directory it uses for gpg allows some other user to write to it. Perhaps because of a wacky umask, I don't know. AFAICS it's only a warning and the upgrade still works, but I have made git-annex force a sane umask when creating this temp directory.
"""]]


code:
Hi,

I have the following setup:
- normal git repository with website code.
- git annex repository to hold large set of binary data (pdfs, flashmovies, etc) that belongs to the site.

I use git annex so I (and other developers) don't need to copy 1.4Gb+ of binary data for every working copy. (Data that is mostly left untouched.) Using git annex copy --to=origin I can simply only add new additions to this media/binary repository, without first pulling all the data. So far so good.

When commits are pushed to a certain branch on the normal git repository, a post-receive hook exports (GIT_WORK_TREE=/data/site/ git checkout $branch -f) the updated repository to an apache documentroot. Thereby updating the staging server of the website.

My question is, how can I do the same thing for my git annex repository? Since post-receive fires on receiving the annex hashes, and not the actual files. Those are rsynced, and I cannot find a way to trigger an action after all files are copied by git annex via rsync.

Any tips?


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2012-03-14T16:23:25Z"
 content="""
I've made git-annex-shell run the git `hooks/annex-content` after content is received or dropped.

Note that the clients need to be running at least git-annex version 3.20120227 , which runs git-annex-shell commit, which runs the hook.

"""]]


code:
Hello all,

I'm trying to use "preferred content" with git-annex but I suspect that my version doesn't provide it.

I'm using Debian "wheezy" that packs git-annex 3.20120629.

I appreciate any help you can give.

Thanks.

Zaka.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 1"
 date="2013-11-07T18:38:22Z"
 content="""
The first version of git-annex to support preferred content settings was 3.20121009.

There is a quite up-to-date backport available for Debian stable.
"""]]


code:
I have my music library in `music/` and some really old files I recently added in `reallyold/`.  There are some MP3s in the really old files and some are the same as my library, so of course git annex is only keeping one copy.  Now, I have an rsync remote, `ma`, which prefers content from `music/` but doesn't want anything from `reallyold/`.  So while right now it is trying to drop stuff, I suspect at some point that it will try to re-add them in virtue of being in `music/`, as I've got a loop.

I want to eliminate this by using the present keyword to disable dropping for stuff in `reallyold/` and `music/`.  Here is my attempt, which doesn't work--I am hoping someone can spot what's wrong.

    (present and include=music/*) or (present and include=reallyold/*) or (exclude=reallyold/* and exclude=video/* and exclude= ...)

Note that music is included by virtue of not being excluded so it should satisfy the third disjunct.  Thanks.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.210"
 subject="Ever figure this out?"
 date="2013-02-24T22:00:40Z"
 content="""
I find it's useful, when stuck on this kind of thing, to run `git annex find` at the command line passing it parts of the expression and making sure it works, and gradually building up to the complete expression.

You have to do some conversion due to differences between command line and preferred content expression format. For example `git annex find -\( --in here --and --include \"music/*\" -\)`
"""]]


code:
[[!comment format=mdwn
 username="spwhitton"
 ip="82.36.235.9"
 subject="comment 2"
 date="2013-03-12T09:34:08Z"
 content="""
Thanks for the tip--I will definitely make another attempt at some point.
"""]]


code:
[[!comment format=mdwn
 username="spwhitton"
 ip="82.36.235.9"
 subject="Impossible?"
 date="2013-06-24T17:48:02Z"
 content="""
I have now investigated this further and believe that what I am trying to do is impossible.  Suppose we have a file `song.mp3` in both `music/` and `old/` and the special remote's preferred content expression is set-up to prefer `music/` and not to prefer `old/`.  So initially, git-annex won't try to upload the content when it scans `old/song.mp3`, but it will when it scans `music/song.mp3`.  Fine.  Now consider a later scan.  On scanning the filename `music/song.mp3` git-annex will suppose the file was dropped from the remote, and then see it would be preferred again so won't drop it.  But then when it gets to scanning the filename `old/song.mp3` it will see that it wouldn't be preferred and will drop the content.  So we get the loop.

In order to stop the unwanted drop here, the preferred content expression must prefer \"files in old/* that is also in music/*\" but since preferred content expressions concern filenames without reference to their contents, this is impossible to express.  The expression `include=music/* and include=old/*` will never match anything.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.193"
 subject="comment 4"
 date="2013-06-25T17:22:53Z"
 content="""
spwhitton, your analysis is correct -- except this problem was finessed in git-annex version 4.20130621. Now, when using direct mode, it knows that `old/song.mp3` is also `music/song.mp3`, and so it only drops it if the preferred content expression allows dropping both files.

Still doesn't work in indirect mode, and I think not likely to in the forseeable future. See [[bugs/Handling_of_files_inside_and_outside_archive_directory_at_the_same_time]]
"""]]


code:
[[!comment format=mdwn
 username="spwhitton"
 ip="82.36.235.9"
 subject="comment 5"
 date="2013-06-25T19:47:19Z"
 content="""
I assume you mean that if I use direct mode locally, it will get things right.  I didn't think a special remote could ever be in direct mode.

Thank you for the info.  I don't want to lose the safeguards of indirect mode just for this, so I'll stick with my inelegant manual preferred content strings for now.
"""]]


code:
Hi,

Use case: I would like to have a "Public" top-level directory in my annex, which gets files in there published over HTTP on a particular server.

How I see doing this:

1. Put my annex to an http server with exported-over-http `/Public/` directory.
2. Configure a `post-update` hook with the following:

        $ git annex fix

3. Push files on `/Public/` to that annex.

Does it make sense? If yes, are there any gotchas I should beware of?

Thanks.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 subject="should work.."
 date="2012-08-05T17:18:47Z"
 content="""
I see no need for `git annex fix` here.

Web server default configurations may not allow following symlinks outside the web server document root. On apache, it can be enabled with `Options FollowSymLinks`
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmdLmVxigDvxiGhnQ34xhyAaxl0CRRe9gs"
 nickname="Bryan"
 subject="Alternate version - smarter integration with other remotes/web special remote?"
 date="2013-03-03T14:19:12Z"
 content="""
I love the idea of pushing files to \"visible\" using annex, but I'm wondering how difficult to would be to make this even fancier.

For instance, if there are files that were originally imported from the web special remote (and have http:// prefixes, since that's not guaranteed), and they're not actually in the pushed annex, generate a redirect for all of them and write a special remote-local .htaccess file.

Or, similarly, allow each remote to have some metadata which specifies what, if any, the URL prefix for a directory in annex is. Then, whenever you update a web remote, the auto-generated redirects list could contain not only files not present but with web-remote locations, but also other remotes which have the file under a web-visible path prefix. Bonus for allowing URL prefixes for S3-style remotes that are not encrypted.

My use case for this sort of fancy is that I have a large server with gobs of space but on a personal (slower) internet connection, and a much smaller hosted instance with much faster bandwidth. With this smarts, I could push and pull large/expected high traffic files to the higher bandwidth/smaller machine ahead of, say, announcing a new feature or sharing a link to a video via e-mail.. or even reactively if the lower-bandwidth server is getting overwhelmed.
"""]]


code:
I would like to use a public, read only annex to publish photos. I have a server, running gitolite, with git-annex setup, which I use successfully to sync content between my own devices.

But, with a public annex, I would like the view of the repository available from the server to be that only the server has the content, and not to have, or give out any of the location tracking information about any other annexes?

A more concrete example would be, how do I get a photo in to git annex locally, and then push this to the server for public access, without publishing information about my local repository?

From the public annexes I have looked at, this does not appear to be done. So I am unsure if this is even possible, however it seems a desirable thing to do?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.55"
 subject="comment 1"
 date="2014-07-03T19:35:21Z"
 content="""
There's not a way to do it yet.

The fundamental problem is that there is a variety of information stored on the git-annex branch, including location tracking information that you don't want (locations on non-public repositories), location tracking information you do want (locations on public repositories), urls, metadata, repository configurations and descriptions (some for public and some for non-public repositories). It seems fairly hard to draw a line. And once a line is drawn, there would be two diverged git-annex branches on the public and private repos, and the private repo would need to synthesize an updated version of the public branch every time it synced.

Instead, the last time this came up, I added the remote.<name>.annex-readonly setting. This allows for at least having a private repository (or a whole network of repositories that all communicate together but remain private) that pulls changes from a public repo, but avoids making changes to it. Certainly not a complete solution, since changes have to be contributed to the public repo in some out-of-band way, like perhaps using git-format-patch and git-am.
"""]]


code:
[[!comment format=mdwn
 username="cbaines"
 ip="82.19.50.118"
 subject="comment 2"
 date="2014-07-03T20:13:20Z"
 content="""
Thanks for your response.

The method you describe sounds like it might just do. I'll have a try, and see if it works out :)
"""]]


code:
Is there a way to pull from an encrypted remote?

Use case:

1. Have annex in an encrypted public rsync remote
2. Have USB stick with PGP keys (but not the annex repository)
3. Get to a new computer
4. Set up a new annex using the PGP keys I have.

1-3 work fine :) However, 4'th is the issue:

How would I do `git pull <remote>` for an encrypted remote? Is it possible?


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
 nickname="Justin"
 subject="comment 1"
 date="2012-07-10T18:14:31Z"
 content="""
You just need to `git clone` the existing repository and make sure the `git remote`s are setup.

the 'rsync remote' is not actually the annex, it's just a collection of encrypted files with obfuscated names.  You need a copy of the actual repository to restore the files.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmLB39PC89rfGaA8SwrsnB6tbumezj-aC0"
 nickname="Tobias"
 subject="Thats a fair solution"
 date="2012-07-22T13:51:25Z"
 content="""
Until you don't have(access to) an existing repository to clone from.

I really hope you revisit this when you come to the encryption part of the assistant.

Btw, I also run FreeBSD if you need a tester on that at some point.

Sincerely
Tobias Ussing

"""]]


code:
I’m using git annex to manage my movie collection on various devices – my laptop, a NSLU tucked away somewhere with lots of space, some external hard drives. For this use case, I do not need the full power of git as a version control system, so having to run "git commit" and coming up with commit messages is annoying. Also, this makes sense for a version control system, but not for my media collection:

	$ git annex add Hot\ Fuzz\ -\ English.mkv
	add Hot Fuzz - English.mkv (checksum...) ok
	(Recording state in git...)
	$ git commit -m 'another movie added'
	[master 851dc8a] another movie added
	 1 files changed, 1 insertions(+), 0 deletions(-)
	 create mode 120000 00 Noch nicht gesehen/Hot Fuzz - English.mkv
	$ git push jeff
	Counting objects: 38, done.
	Delta compression using up to 2 threads.
	Compressing objects: 100% (20/20), done.
	Writing objects: 100% (26/26), 2.00 KiB, done.
	Total 26 (delta 11), reused 0 (delta 0)
	remote: error: refusing to update checked out branch: refs/heads/master
	remote: error: By default, updating the current branch in a non-bare repository
	remote: error: is denied, because it will make the index and work tree inconsistent
	remote: error: with what you pushed, and will require 'git reset --hard' to match
	remote: error: the work tree to HEAD.
	remote: error:
	remote: error: You can set 'receive.denyCurrentBranch' configuration variable to
	remote: error: 'ignore' or 'warn' in the remote repository to allow pushing into
	remote: error: its current branch; however, this is not recommended unless you
	remote: error: arranged to update its work tree to match what you pushed in some
	remote: error: other way.
	remote: error:
	remote: error: To squelch this message and still keep the default behaviour, set
	remote: error: 'receive.denyCurrentBranch' configuration variable to 'refuse'.
	To jeff:/mnt/media/Movies
	 ! [rejected]        git-annex -> git-annex (non-fast-forward)
	 ! [remote rejected] master -> master (branch is currently checked out)
	error: failed to push some refs to 'jeff:/mnt/media/Movies'
	To prevent you from losing history, non-fast-forward updates were rejected
	Merge the remote changes (e.g. 'git pull') before pushing again.  See the
	'Note about fast-forwards' section of 'git push --help' for details.

It seems that to successfully make the new files known to the other side, I have to log into jeff and pull _from_ my current machine.

What I would like to have is that

* git annex add does not require a commit afterwards.
* Changes to the files are automatically picked up with the next git-annex call (similar to how etckeeper works).
* Commands "git annex push" and "git annex pull" that will sync the metadata (i.e. the list of files) in both directions without further manual intervention, at least not until the two repositories have diverged in a way that is not possible to merge sensible.

Summay: git-annex is great. git is not always. Please make it possible to use git annex without having to use git.


code:
[[!comment format=mdwn
 username="http://www.joachim-breitner.de/"
 nickname="nomeata"
 subject="Finally some code"
 date="2011-12-29T19:58:31Z"
 content="""
The repository at http://git.nomeata.de/?p=git-annex.git;a=summary contains changes to Commands/Sync.hs (and to the manpage) that implements this behavior. The functionality should be fine; the progress output is not very nice yet, but I’m not sure if I really understood the various Command types. It also should be more easily discoverable how to activate the behavior (by running \"git branch synced/master\") by providing a helpful message, at least unless git annex init creates the branch by default.
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 11"
 date="2011-12-30T21:49:06Z"
 content="""
OMG, my first sizable haskell patch!

So trying this out..

In each repo I want to sync, I first `git branch synced/master`

Then in each repo, I found I had to pull from each of its remotes, to get the tracking branches that `defaultSyncRemotes` looks for to know those remotes are syncable. This was the surprising thing for me, I had expected sync to somehow work out which remotes were syncable without my explicit pull. And it was not very obvious that sync was not doing its thing before I did that, since it still does a lot of \"stuff\".

Once set up properly, `git annex sync` fetches from each remote, merges, and then pushes to each remote that has a synced branch. Changes propigate around even when some links are one-directional. Cool!

So it works fine, but I think more needs to be done to make setting up syncing easier. Ideally, all a user would need to do is run \"git annex sync\" and it syncs from all remotes, without needing to manually set up the synced/master branch.

While this would lose the ability to control which remotes are synced, I think that being able to `git annex sync origin` and only sync from/to origin is sufficient, for the centralized use case.

---

Code review:

Why did you make `branch` strict?

There is a bit of a bug in your use of Command.Merge.start. The git-annex branch merge code only runs once per git-annex run, and often this comes before sync fetches from the remotes, leading to a push conflict. I've fixed this in my \"sync\" branch, along with a few other minor things.

`mergeRemote` merges from `refs/remotes/foo/synced/master`. But that will only be up-to-date if `git annex sync` has recently been run there. Is there any reason it couldn't merge from `refs/remotes/foo/master`?
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 12"
 date="2011-12-30T23:45:57Z"
 content="""
I have made a new `autosync` branch, where all that the user needs to do is run `git annex sync` and it automatically sets up the synced/master branch. I find this very easy to use, what do you think?

Note that `autosync` is also pretty smart about not running commands like \"git merge\" and \"git push\" when they would not do anything. So you may find `git annex sync` not showing all the steps you'd expect. The only step a sync always performs now is pulling from the remotes.
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 13"
 date="2011-12-31T18:34:31Z"
 content="""
I have merged my autosync branch, the improved sync command will be in this year's last git-annex release!
"""]]


code:
[[!comment format=mdwn
 username="http://www.joachim-breitner.de/"
 nickname="nomeata"
 subject="comment 14"
 date="2012-01-02T14:02:04Z"
 content="""
Sorry for not replying earlier, but my non-mailinglist-communications-workflows are suboptimal :-)

> Then in each repo, I found I had to pull from each of its remotes, to get the tracking branches that defaultSyncRemotes looks for to know those remotes are syncable.  This was the surprising thing for me, I had expected sync to somehow work out which remotes were syncable without my explicit pull. And it was not very obvious that sync was not doing its thing before I did that, since it still does a lot of \"stuff\".

Right. But \"git fetch\" ought to be enough.

Personally, I’d just pull and push everywhere, but you pointed out that it ought to be manageable. The existence of the synced/master branch is the flag that indicates this, so you need to propagate this once. Note that if the branch were already created by \"git annex init\", then this would not be a problem.

It is not required to use \"git fetch\" once, you can also call \"git annex sync <remote>\" once with the remote explicitly mentioned; this would involve a fetch.

> While this would lose the ability to control which remotes are synced, I think that being able to git annex sync origin and only sync from/to origin is sufficient, for the centralized use case.

I’d leave this decision to you. But I see that you took the decision already, as your code now creates the synced/master branch when it does not exist (e290f4a8).

> Why did you make branch strict?

Because it did not work otherwise :-). It uses pipeRead, which is lazy, and for some reason git and/or your utility functions did not like that the output of the command was not consumed before the next git command was called. I did not investigate further. For better code, I’d suggest to add a function like pipeRead that completely reads the git output before returning, thus avoiding any issues with lazyIO.

> mergeRemote merges from refs/remotes/foo/synced/master. But that will only be up-to-date if git annex sync has recently been run there. Is there any reason it couldn't merge from refs/remotes/foo/master?

Hmm, good question. It is probably save to merge from both, and push only to synced/master. But which one first? synced/master can be ahead if the repo was synced to from somewhere else, master can be ahead if there are local changes. Maybe git merge should be called on all remote heads simultaniously, thus generating only one commit for the merge. I don’t know how well that works in practice.

Thanks for including my code,
Joachim

"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 15"
 date="2012-01-02T16:01:49Z"
 content="""
With a lazy branch, I get \"git-annex: no branch is checked out\". Weird.. my best guess is that it's because this is running at the seek stage, which is unusual, and the value is not used until a later stage and so perhaps the git command gets reaped by some cleanup code before its output is read.

(pipeRead is lazy because often it's used to read large quantities of data from git that are processed progressively.)

I did make it merge both branches, separately. It would be possible to do one single merge, but it's probably harder for the user to recover if there are conflicts in an octopus merge. The order of the merges does not seem to me to matter much, barring conflicts it will work either way. Dealing with conflicts during sync is probably a weakness of all this; after the first conflict the rest of the sync will continue failing.
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2011-12-09T22:56:11Z"
 content="""
First, you need a bare git repository that you can push to, and pull from. This simplifies most git workflow.

Secondly, I use [mr](http://kitenet.net/~joey/code/mr/), with this in `.mrconfig`:

<pre>
[DEFAULT]
lib =
        annexupdate() {
                git commit -a -m update || true
                git pull \"$@\"
                git annex merge
                git push || true
        }

[lib/sound]
update = annexupdate
[lib/big]
update = annexupdate
</pre>

Which makes \"mr update\" in repositories where I rarely care about git details take care of syncing my changes.

I also make \"mr update\" do a \"git annex get\" of some files in some repositories that I want to always populate. git-annex and mr go well together. :)

Perhaps my annexupdate above should be available as \"git annex sync\"?
"""]]


code:
[[!comment format=mdwn
 username="http://www.joachim-breitner.de/"
 nickname="nomeata"
 subject="comment 2"
 date="2011-12-10T16:28:29Z"
 content="""
Thanks for the tips so far. I guess a bare-only repo helps, but as well is something that I don’t _need_ (for my use case), any only have to do because git works like this.

Also, if I have a mobile device that I want to push to, then I’d have to have two repositories on the device, as I might not be able to reach my main bare repository when traveling, but I cannot push to the „real“ repo on the mobile device from my computer. I guess I am spoiled by darcs, which will happily push to a checked out
remote repository, updating the checkout if possible without conflict.

If I introduce a central bare repository to push to and from; I’d still have to have the other non-bare repos as remotes, so that git-annex will know about them and their files, right?

I’d appreciate a \"git annex sync\" that does what you described (commit all, pull, merge, push). Especially if it comes in a \"git annex sync --all\" variant that syncs all reachable repositories.
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 3"
 date="2011-12-10T19:43:04Z"
 content="""
Git can actually push into a non-bare repository, so long as the branch you change there is not a checked out one. Pushing into `remotes/$foo/master` and `remotes/$foo/git-annex` would work, however determining the value that the repository expects for `$foo` is something git cannot do on its own. And of course you'd still have to `git merge remotes/$foo/master` to get the changes.

Yes, you still keep the non-bare repos as remotes when adding a bare repository, so git-annex knows how to get to them.

I've made `git annex sync` run the simple script above. Perhaps it can later be improved to sync all repositories.
"""]]


code:
[[!comment format=mdwn
 username="http://www.joachim-breitner.de/"
 nickname="nomeata"
 subject="comment 4"
 date="2011-12-13T18:16:08Z"
 content="""
I thought about this some more, and I think I have a pretty decent solution that avoids a central bare repository. Instead of pushing to master (which git does not like) or trying to guess the remote branch name on the other side, there is a well-known branch name, say git-annex-master. Then a sync command would do something like this (untested):

    git commit -a -m 'git annex sync' # ideally with a description derived from the diff
    git merge git-annex-master
    git pull someremote git-annex-master # for all reachable remotes. Or better to use fetch and then merge everything in one command?
    git branch -f git-annex-master # (or checkout git-annex-master, merge master, checkout master, but since we merged before this should have the same effect
    git annex merge
    git push someremote git-annex-master # for all reachable remotes

The nice things are: One can push to any remote repository, and thus avoid the issue of pushing to a portable device; the merging happens on the master branch, so if it fails to merge automatically, regular git foo can resolve it, and all changes eventually reach every repository.

What do you think?

"""]]


code:
[[!comment format=mdwn
 username="http://www.joachim-breitner.de/"
 nickname="nomeata"
 subject="comment 5"
 date="2011-12-13T18:47:18Z"
 content="""
After some experimentation, this seems to work better:

    	git commit -a -m 'git annex sync'
	git merge git-annex-master
	for remote in $(git remote)
	do
		git fetch $remote
		git merge $remote git-annex-master
	done
	git branch -f git-annex-master
	git annex merge
	for remote in $(git remote)
	do
		git push $remote git-annex git-annex-master
	done

Maybe this approach can be enhance to skip stuff gracefully if there is no git-annex-master branch and then be added to what \"git annex sync\" does, this way those who want to use the feature can do so by running \"git branch git-annex-master\" once. Or, if you like this and want to make it default, just make git-annex-init create the git-annex-master branch :-)
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 6"
 date="2011-12-13T20:53:23Z"
 content="""
It would be clearer to call \"git-annex-master\" \"synced/master\" (or really \"synced/$current_branch\"). That does highlight that this method of syncing is not particularly specific to git-annex.

I think this would be annoying to those who do use a central bare repository, because of the unnecessary pushing and pulling to other repos, which could be expensive to do, especially if you have a lot of interconnected repos. So having a way to enable/disable it seems best.

Maybe you should work up a patch to Command/Sync.hs, since I know you know haskell :)
"""]]


code:
[[!comment format=mdwn
 username="http://www.joachim-breitner.de/"
 nickname="nomeata"
 subject="comment 7"
 date="2011-12-18T12:08:51Z"
 content="""
I agree on the naming suggestions, and that it does not suit everybody. Maybe I’ll think some more about it. The point is: I’m trying to make live easy for those who do not want to manually create some complicated setup, so if it needs configuration, it is already off that track. But turning the current behavior into something people have to configure is also not well received by the users.

Given that \"git annex sync\" is a new command, maybe it is fine to have this as a default behavior, and offer an easy way out. The easy way out could be one of two flags that can be set for a repo (or a remote):

* \"central\", which makes git annex sync only push and pull to and that repo (unless a different remote is given on the command line)
* \"unsynced\", which makes git annex sync skip the repo.

Maybe central is enough.
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 8"
 date="2011-12-19T18:29:01Z"
 content="""
I don't mind changing the behavior of git-annex sync, certainly..

Looking thru git's documentation, I found some existing configuration that could be reused following your idea.
There is a remote.name.skipDefaultUpdate and a remote.name.skipFetchAll. Though both have to do with fetches, not pushes.
Another approach might be to use git's remote group stuff.
"""]]


code:
[[!comment format=mdwn
 username="http://www.joachim-breitner.de/"
 nickname="nomeata"
 subject="comment 9"
 date="2011-12-19T22:56:26Z"
 content="""
Another option that would please the naive user without hindering the more advanced user: \"git annex init\", by default, creates a synced/master branch. \"git annex sync\" will pull from every <remote>/sync/master branch it finds, and also push to any <remote>/sync/master branch it finds, but will not create any. So by default (at least for new users), this provides simple one-step syncing.

Advanced users can disable this per-repo by just deleting the synced/master branch. Presumably the logic will be: Every repo that should not be pushed to, because it has access to some central repo, should not have a synced/master branch. Every other repo, including the (or one of the few) central repos, will have the branch.

This is not the most expressive solution, as it does not allow configuring syncing between arbitrary pairs of repos, but it feels like a good compromise between that and simplicity and transparency.

I think it's about time that I provide less talk and more code. I’ll see when I find the time :-)
"""]]


code:
Hi guys, as a result of some improper handling I am left with a couple of files with no copies scattered all around my repos. Now, while my data loss is unfortunate, it is not tragic. What I would like to do now is to purge all of these files without having to look for them one by one. Is there any easy way to do this?

Cheers


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.14.105"
 subject="comment 1"
 date="2013-09-19T20:59:32Z"
 content="""
This will output all files in your repository that git-annex thinks have no copies left:

`git annex find --not --copies=1`

Piping it to git rm is left as an excercise for the reader..
"""]]


code:
I have a git-annex repository located at ~/annex which has been set up using git-annex assistant.

This repository is configured as "client".

My other repository is a huge USB drive configured as "full archive".

Now everything seems to work fine except there is one thing I don't understand:

    alip@client:~/annex> git-annex whereis ./archive/kus.png
    whereis archive/kus.png (1 copy)
        e79a4cf6-4c48-4833-93de-98ba6eb625d6 -- deniz
    ok

Fine, there is only one copy according to git-annex but the file is still present
in **this** client repository:

    alip@client:> du -hs ~/annex/archive
    20G	/home/alip/annex/archive/

How do I free this space? Am I supposed to call git-annex drop manually?

git-annex version: 3.20130124


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
 nickname="Justin"
 subject="comment 1"
 date="2013-01-30T18:44:52Z"
 content="""
Give this a try:

http://git-annex.branchable.com/walkthrough/unused_data/
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkZktNHFhxC1kYA9KKdKpYJO4clq9WDsjE"
 nickname="Jason"
 subject="comment 2"
 date="2013-01-30T19:14:48Z"
 content="""
I'm seeing the same thing and I think it's a bug.  Everything I'm reading says that when you move a file from anywhere into an archive directory, it's supposed to upload it to an archive and drop it locally.

If I copy the file from outside my annex into the archive directory it will usually do exactly that and I'll end up with a symlink, but not always.  Sometimes I'll end up with a file.

If I move the file from within my annex into the archive, everything gets confused.  Here is an example:

    [jchu@april annex(master)]$ mkdir testdir
    [jchu@april annex(master)]$ cd testdir/
    [jchu@april testdir(master)]$ ls
    [jchu@april testdir(master)]$ cat > test.txt << EOF
    > I have a file.  It is very nice.
    > EOF
    [jchu@april testdir(master)]$ ls -l
    total 4
    -rw-r--r-- 1 jchu users 33 Jan 30 10:53 test.txt
    [jchu@april testdir(master)]$ mkdir archive
    [jchu@april testdir(master)]$ mv test.txt archive/
    [jchu@april testdir(master)]$ ls -l
    total 0
    drwxr-xr-x 2 jchu users 21 Jan 30 10:54 archive
    [jchu@april testdir(master)]$ ls -l archive/
    total 4
    -rw-r--r-- 1 jchu users 33 Jan 30 10:53 test.txt
    [jchu@april testdir(master)]$ # Shouldn't this be a symlink?
    [jchu@april testdir(master)]$ git show
    commit 2fb2cba3fae98e0e0b8b7891b1af0cc655754896
    Author: Jason Chu <jchu@xentac.net>
    Date:   Wed Jan 30 10:54:11 2013 -0800

    diff --git a/testdir/archive/test.txt b/testdir/archive/test.txt
    new file mode 120000
    index 0000000..5bd7982
    --- /dev/null
    +++ b/testdir/archive/test.txt
    @@ -0,0 +1 @@
    +../../.git/annex/objects/x5/Q1/SHA256E-s33--1859a57b635b0702dd3813dd7637f11842730b0464c35a4ad361dd0d37fc3dab.txt/SHA256E-s33--1859a57b635b0702dd3813dd7637f11842730b0464c35a4ad3
    \ No newline at end of file
    [jchu@april testdir(master)]$ git show HEAD^
    commit 410f58f782348599d42496db2c9b21cfb2e59124
    Author: Jason Chu <jchu@xentac.net>
    Date:   Wed Jan 30 10:54:10 2013 -0800

    diff --git a/testdir/test.txt b/testdir/test.txt
    deleted file mode 120000
    index 82343e3..0000000
    --- a/testdir/test.txt
    +++ /dev/null
    @@ -1 +0,0 @@
    -../.git/annex/objects/x5/Q1/SHA256E-s33--1859a57b635b0702dd3813dd7637f11842730b0464c35a4ad361dd0d37fc3dab.txt/SHA256E-s33--1859a57b635b0702dd3813dd7637f11842730b0464c35a4ad361d
    \ No newline at end of file

From what I can see here, the file was archived properly and the delete in git was issued correctly.  The daemon.log shows me that the file was uploaded correctly when it was initially created and then added to the archive directory, but I don't see a direct delete in the log file (even though the git log shows me it happened).

Now on to the really weird part.  If I restart the daemon (or there are other ways to trigger it, but this seems to be the easiest for me), I see that log lines that say testdir/archive/test.txt was dropped (good) but then testdir/test.txt is re-downloaded and appears back in that directory.  I wanted the file deleted from testdir, why is it back?!?  Since it gets redownloaded and readded, my new logs look like this:

    [jchu@april testdir(master)]$ git show
    commit 850464dd2f8ae5cd407a214543afab668d4b34eb
    Author: Jason Chu <jchu@xentac.net>
    Date:   Wed Jan 30 11:00:03 2013 -0800

    diff --git a/testdir/archive/test.txt b/testdir/archive/test.txt
    deleted file mode 120000
    index 5bd7982..0000000
    --- a/testdir/archive/test.txt
    +++ /dev/null
    @@ -1 +0,0 @@
    -../../.git/annex/objects/x5/Q1/SHA256E-s33--1859a57b635b0702dd3813dd7637f11842730b0464c35a4ad361dd0d37fc3dab.txt/SHA256E-s33--1859a57b635b0702dd3813dd7637f11842730b0464c35a4ad3
    \ No newline at end of file
    [jchu@april testdir(master)]$ git show HEAD^
    commit 8cf9019e66e5a2cb0478e965fda352c742972591
    Author: Jason Chu <jchu@xentac.net>
    Date:   Wed Jan 30 11:00:00 2013 -0800

    diff --git a/testdir/test.txt b/testdir/test.txt
    new file mode 120000
    index 0000000..82343e3
    --- /dev/null
    +++ b/testdir/test.txt
    @@ -0,0 +1 @@
    +../.git/annex/objects/x5/Q1/SHA256E-s33--1859a57b635b0702dd3813dd7637f11842730b0464c35a4ad361dd0d37fc3dab.txt/SHA256E-s33--1859a57b635b0702dd3813dd7637f11842730b0464c35a4ad361d
    \ No newline at end of file

And the filesystem looks like this:

    [jchu@april testdir(master)]$ ls -lR
    .:
    total 4
    drwxr-xr-x 2 jchu users 21 Jan 30 11:00 archive
    -rw-r--r-- 1 jchu users 33 Jan 30 11:00 test.txt

    ./archive:
    total 4
    -rw-r--r-- 1 jchu users 33 Jan 30 11:00 test.txt

I have 3 remotes set up: a usb drive (client, it was created like that directly by annex when I created a Removable drive repo, it's a bare repo), s3 (archive), and nas (a remote server repo, that connects over ssh and uses rsync).  I thought maybe that the usb drive still had a record of testdir/test.txt existing, but the logs in refs/heads/synced/master match.

It's as if the git annex thinks that the original content is still within a non-archive directory, so the data can't be dropped, but everywhere I look the references to that content are gone.  The only way I've found to fix this is to convert to indirect mode and delete the symlinks from there (which seems to work better).  It seems to be an interaction between direct mode, the archive, and maybe my filesystem (xfs).
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.3.125"
 subject="comment 3"
 date="2013-02-05T17:46:56Z"
 content="""
@alip I've just fixed a bug which made the assistant get location tracking info wrong for files it added in direct mode. That probably explains what you're seeing. You can run `git annex fsck` to fix the location log info, and if you restart the assistant it'll probably then realize the files are locally present and drop them. Of course, the USB drive has to be connected for that to work; git-annex needs to verify the files have been moved to it before it can drop them.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.3.125"
 subject="comment 4"
 date="2013-02-05T19:05:49Z"
 content="""
@Jason That is a bug.  Please don't report what are obviously bugs in the forum!

I've posted a bug report about it here: [[bugs/direct_mode_renames]]
"""]]


code:
Hi,

I apologize, but I can't retrace my steps, so my problem is not replicable. The current situation is: 1 main repo and 1 directory/encrypted special remote. From the main repo, I see 0 available copy of the files I need. But when I do a fsck I do see the file. The problem is that the location log does not seem to get fixed after fsck, as I cannot use git-annex get to obtain the file, even after the fsck command.

Do you have any suggestions for next diagnostic/repair steps?

Thanks a lot for your time and answer!

Vincent

macbook% git annex fsck foo.zip --from needle
fsck foo.zip (gpg)
You need a passphrase to unlock the secret key for
user: “me <email@email.edu>”
2048-bit RSA key, ID ???????, created 2013-01-11 (main key ID ????????)

(checking needle...)
GPGHMACSHA1--d7036fd17214c06c83d53479dcec9554a7dc2109
    13815014 100%   14.16MB/s    0:00:00 (xfer#1, to-check=0/1)

sent 13816832 bytes  received 42 bytes  9211249.33 bytes/sec
total size is 13815014  speedup is 1.00
(fixing location log) (checksum...)
  ** No known copies exist of foo.zip
failed
(Recording state in git...)
git-annex: fsck: 1 failed

macbook% git annex get foo.zip
get foo.zip (not available)
  No other repository is known to contain the file.
failed
git-annex: get: 1 failed
macbook%


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 1"
 date="2013-12-01T19:59:16Z"
 content="""
I think you should try to run `git annex copy --from needle`, which will be able to get the files if needle has them, even if the location log is busted.

(Sometimes people complain that `git annex copy` goes and checks a remote, which can be unncessarily slow, rather than just trusting the location log  ... but this is why it does it!)

From what I can tell, the fsck downloads some file from the remote, but the download is not entirely successful. One possibility is that it downloads a file, but the file fails to be decrypted, either because you don't have the right encyption key, or because the encrypted file has actually gotten corrupted on the remote. It might be that the same problem will affect `git annex copy --from needle`, but if so it should be more clear what's happening.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmLD-g4d-EGXD5KJh1kVKnRX0a9dTnEigg"
 nickname="Vincent"
 subject="comment 2"
 date="2013-12-02T01:28:18Z"
 content="""
Thanks a lot for the quick response! Unfortunately, copy is just silent and does not copy the files over.

I don't want to waste more of your time on this. I just wish I had kept better track of the steps I took to get myself into this mess...
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 3"
 date="2013-12-02T19:53:11Z"
 content="""
Hmm, it seems I was wrong about copy --from not checking the location log. It does.. Probably to avoid large copies trying to copy a lot of files that are not present in a remote.

I have now made `git annex copy --force --from remote` do what I said it would do, and that should work in your situation, I think.
"""]]


code:
I've been playing with a mixed setup, and I frequentely end up with conflicts which can be ascribed to mixing direct windows repos with indirect linux one(s), and making renames on the indirect ones.
Possibly someone can address what I miss of the git-annex/git interaction.

#### versions involved ####

linux: git-annex version: 4.20131024
win: git-annex version: 4.20131024-gca7b71e

### Steps to reproduce behaviour ###

###### 1. On linux, i setup bare origin "casa" and client repo "local":

    [michele@home ~]$ git init --bare casa
    Initialized empty Git repository in /home/sambahome/michele/casa/
    [michele@home ~]$ cd casa
    [michele@home casa]$ git annex init casa
    init casa ok
    (Recording state in git...)
    [michele@home ~]$ cd ..; git clone casa local
    Cloning into 'local'...
    done.
    warning: remote HEAD refers to nonexistent ref, unable to checkout.
    [michele@home ~]$ cd local; git annex init local
    init local ok
    (Recording state in git...)
    [michele@home local]$ echo lintest > lintest
    [michele@home local]$ git annex add lintest
    add lintest (checksum...) ok
    (Recording state in git...)
    [michele@home local]$ git annex sync
    (merging origin/git-annex into git-annex...)
    (Recording state in git...)
    commit
    ok
    pull origin
    ok
    push origin
    Counting objects: 18, done.
    Delta compression using up to 4 threads.
    Compressing objects: 100% (12/12), done.
    Writing objects: 100% (16/16), 1.48 KiB | 0 bytes/s, done.
    Total 16 (delta 1), reused 0 (delta 0)
    To /home/sambahome/michele/casa
     * [new branch]      git-annex -> synced/git-annex
     * [new branch]      master -> synced/master
    ok
    ```

###### 2. On windows I clone origin, and I sync empty

    ```cmd
    M:\>git clone ssh://michele@home/home/michele/casa win
    Cloning into 'win'...
    remote: Counting objects: 20, done.
    remote: Compressing objects: 100% (15/15), done.
    remote: Total 20 (delta 3), reused 0 (delta 0)
    Receiving objects: 100% (20/20), done.
    Resolving deltas: 100% (3/3), done.
    M:\>cd win
    M:\win>git annex status
      Detected a crippled filesystem.
      Enabling direct mode.
      Detected a filesystem without fifo support.
      Disabling ssh connection caching.
    repository mode: direct
    trusted repositories: (merging origin/git-annex origin/synced/git-annex into git-annex...)
    (Recording state in git...)
    0
    semitrusted repositories: 4
            00000000-0000-0000-0000-000000000001 -- web
            598ecfac-087d-49a3-b48d-beafd0d71805 -- origin (casa)
            b2699c17-d0bc-40a0-b447-a64ad109b2a2 -- here (ALICUDI:M:\win)
            bd4166eb-296b-4f0f-a3be-6c25e4c7cbb0 -- local
    untrusted repositories: 0
    transfers in progress: none
    available local disk space: unknown
    local annex keys: 0
    local annex size: 0 bytes
    known annex keys: 1
    known annex size: 8 bytes
    bloom filter size: 16 mebibytes (0% full)
    backend usage:
            SHA256E: 1
    ```

###### 3. I copy content from local to win

    ```bash
    [michele@home local]$ git annex copy --to origin lintest
    copy lintest (to origin...) ok
    (Recording state in git...)
    [michele@home local]$ git annex sync
    ...runs ok...
    ```

###### 4. and

    ```cmd
    M:\win>git annex sync
    ...works
    M:\win>git annex get .
    ...works
    M:\win>cat lintest
    lintest
    ```

so far so good.
###### 5. Now the renaming part (performed on linux indirect repo)

    ```bash
    [michele@home local]$ git mv lintest renamed
    [michele@home local]$ git annex list
    here
    |origin
    ||web
    |||
    XX_ renamed
    [michele@home local]$ git annex sync
    ...works
    ```

###### 6.  now, by issuing sync on windows I start getting a "push issue":

    ```
    M:\win>git annex sync
    commit
    ok
    pull origin
    remote: Counting objects: 3, done.
    remote: Total 2 (delta 0), reused 0 (delta 0)
    Unpacking objects: 100% (2/2), done.
    From ssh://home/home/michele/casa
       c3b7a63..a0854bf  master     -> origin/master
       c3b7a63..a0854bf  synced/master -> origin/synced/master
    ok
    push origin
    Counting objects: 9, done.
    Delta compression using up to 2 threads.
    Compressing objects: 100% (4/4), done.
    Writing objects: 100% (5/5), 484 bytes, done.
    Total 5 (delta 1), reused 0 (delta 0)
    To ssh://michele@home/home/michele/casa
       6c18669..8cc74a0  git-annex -> synced/git-annex
     ! [rejected]        master -> synced/master (non-fast-forward)
    error: failed to push some refs to 'ssh://michele@home/home/michele/casa'
    hint: Updates were rejected because a pushed branch tip is behind its remote
    hint: counterpart. Check out this branch and merge the remote changes
    hint: (e.g. 'git pull') before pushing again.
    hint: See the 'Note about fast-forwards' in 'git push --help' for details.
    failed
    git-annex: sync: 1 failed

    M:\win>
    ```

at this stage I tried to issue a git annex merge, git annex sync, leading to same result.

somewhere in the forum I read i could try issuing a git pull origin master (this could be the problem).
and the result is as such:

    ```
    M:\win>git pull master
    fatal: 'master' does not appear to be a git repository
    fatal: The remote end hung up unexpectedly

    M:\win>git pull origin master
    From ssh://home/home/michele/casa
     * branch            master     -> FETCH_HEAD
    Updating c3b7a63..a0854bf
    error: Your local changes to the following files would be overwritten by merge:
            lintest
    Please, commit your changes or stash them before you can merge.
    Aborting

    M:\win>git status
    # On branch master
    # Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
    #
    # Changes not staged for commit:
    #   (use "git add <file>..." to update what will be committed)
    #   (use "git checkout -- <file>..." to discard changes in working directory)
    #
    #       modified:   lintest
    #
    no changes added to commit (use "git add" and/or "git commit -a")

    M:\win>cat lintest
    lintest
    ```

well, ok it appears modified for some weirdness (crlf?), we can live with it.

    ```
    M:\win>git checkout -->> this replaces files contents with simlink (due to git pull above?)
    ```

at this stage content is lost, and annex has no knowledge about it.

    ```
    M:\win>git annex fsck
    fsck lintest ok

    M:\win>cat lintest
    .git/annex/objects/9Z/82/SHA256E-s8--2b721dbe9afe6031cce3004e909dd62e0b4b2f3944438b6a000dffc7ad657715/SHA256E-s8--2b721dbe9afe6031cce3004e90
    M:\win>git annex list
    here
    |origin
    ||web
    |||
    XX_ lintest
    ```

still I cannot sync, but now i can pull origin master:

    ```cmd
    M:\win>git pull origin master                          From ssh://home/home/michele/casa                       * branch            master     -> FETCH_HEAD          Updating c3b7a63..a0854bf                              Fast-forward                                            lintest => renamed | 0                                 1 file changed, 0 insertions(+), 0 deletions(-)        rename lintest => renamed (100%)
    ```

this doesnt restore content (annex thinks its already there:

    ```
    M:\win>cat renamed
    .git/annex/objects/9Z/82/SHA256E-s8--2b721dbe9afe6031cce3004e909dd62e0b4b2f3944438b6a000dffc7ad657715/SHA256E-s8--2b721dbe9afe6031cce3004e909dd62e0b4b2f3944438b6a000dffc7ad657715
    ```

I think my mistake is the use of the ```git checkout``` in direct mode. But why is the file detected as modified in the first place ?

note: as long as i didn't drop on origin, i still can recover contents by 'forcing' a content refresh:

    ```
    M:\win>git annex get renamed --from origin
    get renamed (from origin...)
    SHA256E-s8--2b721dbe9afe6031cce3004e909dd62e0b4b2f3944438b6a000dffc7ad657715
               8 100%    7.81kB/s    0:00:00 (xfer#1, to-check=0/1)

    sent 30 bytes  received 153 bytes  8.13 bytes/sec
    total size is 8  speedup is 0.04
    ok
    (Recording state in git...)

    M:\win>cat renamed
    .git/annex/objects/9Z/82/SHA256E-s8--2b721dbe9afe6031cce3004e909dd62e0b4b2f3944438b6a000dffc7ad657715/SHA256E-s8--2b721dbe9afe6031cce3004e909d
    M:\win>git annex fsck
    fsck renamed (fixing direct mode) (checksum...) ok

    M:\win>cat renamed
    lintest
```


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 1"
 date="2013-10-26T19:35:46Z"
 content="""
You should **never, ever, ever** run `git pull`, or `git checkout` inside a direct repository. You need to read [[direct_mode]]. If you do not fully understand it, you should avoid running any git commands that are not `git annex foo` in a direct mode repository. If you forget and do run such git commands, you can generally recover by running `git annex fsck`, although it's possible that the git command you run overwrites your only copy of a file, and so you'd lose it.

<pre>
To ssh://michele@home/home/michele/casa
   6c18669..8cc74a0  git-annex -> synced/git-annex
 ! [rejected]        master -> synced/master (non-fast-forward)
error: failed to push some refs to 'ssh://michele@home/home/michele/casa'
</pre>

I'll bet that if you look at the `git config` of this repository it failed to push to, you'll find that it has `receive.denyNonFastforwards` set to true. If you unset that, the push should work.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkJafmCf-sg9_OM0pynFYM3AO4WCgJiaMI"
 nickname="Michele"
 subject="still cannot push when remote has renames"
 date="2013-10-27T23:06:03Z"
 content="""
now, I went again through docs, and i realized how stupid was issuing a git pull on a direct repo. thanks for your patience.

but, i double checked the configuration, I assume \"receive.denyNonFastForwards\" is false by default, but anyway I set it up explicitely so that now my git config (on the linux indirect repo - with respect to my previous example, I got rid of the \"extra\" bare repo in the middle) shows:

    $ git config --list
    user.email=m@g.com
    user.name=michele
    core.repositoryformatversion=0
    core.filemode=true
    core.bare=false
    core.logallrefupdates=true
    annex.uuid=d084e0fd-95a7-4c98-a206-fbf2c85b779d
    annex.version=3
    receive.denynonfastforwards=false

still I am receiving the push refusal:

    M:\win>git annex sync
    commit
    ok
    pull origin
    ok
    push origin
    To ssh://michele@home/home/michele/casa
       ! [rejected]        master -> synced/master (non-fast-forward)
    error: failed to push some refs to 'ssh://michele@home/home/michele/casa'
    hint: Updates were rejected because a pushed branch tip is behind its remote
    hint: counterpart. Check out this branch and merge the remote changes
    hint: (e.g. 'git pull') before pushing again.
    hint: See the 'Note about fast-forwards' in 'git push --help' for details.
    failed
    git-annex: sync: 1 failed

Same happens with a bare repository in the middle. BTW: the windows \"client\" repository is behind NAT, so that the linux indirect doesn't actively sync against it: could that be source of the problem ?
"""]]