code:
Hi,

Git-annex is really awesome. It has made my life really easier when having to
move files around.

Yet, I have been struggling with a use case that I cannot get working with git
annex.

In short, my request is: could it be possible to have --want-get and --want-drop
accept a repository as argument to match the preferred content of that
repository instead of here?

Now, let me explain why I need this:a

All my files are stored into a NAS accessible via a local network.

I have an annex in my desktop computer. Using preferred content (via "git annex
wanted") and "git annex get|drop --auto", I am able to almost automatically
handle what files are put into my computer. What I do is to "git annex wanted"
to indicate what I want to be here and launch a home made script that basically
does "git annex get --auto" and "git annex drop --auto".

Let's say I have a android phone to which I connect via ssh over adb. It
contains a git repository but few files are in it. It has no wifi and so no
access to the network, meaning no access to the NAS.

The links between annexes then looks like:

    NAS <-> Computer <-> Phone

When I want to put a file into my phone, I generally launch "git annex get file"
from my computer (then I get the file from the NAS) and "git annex copy --to
phone file".

I want to be able to automatise this a bit by playing with preferred content
(like I do with my computer). This means that I want to launch "git annex
wanted" to edit the preferred content of the phone annex and then "git annex get
--auto" and launch "git annex copy --auto --to phone". This way, when I am not
in front of my computer, I can still from my phone run "git annex wanted here
'preferred content'" and hope for my synchronisation scripts (run in my
computer) to put the good files into my phone.

Obviously those commands won't work since the git annex get --auto command will
only get what my computer wants, not what my phone wants.

The intuitive (IMHO) way to do would be to launch:

    git annex get --want-get phone
	git annex copy --auto --to phone
	git annex drop --auto

With "--want-get repository" meaning, "Matches files that the preferred content
settings for the repository make it want to get.".

For the time being, I succeed in doing this with

	OLD_WANTED=$(git annex wanted here)
	git annex wanted here $(git annex wanted phone)
	git annex copy --auto --to phone
	git annex wanted ${OLD_WANTED}
	git annex drop --auto

This is complicated and adds two extra commits in the git-annex branch (one for
each setting of git annex wanted) each time I call the script.

What do you think?

Thanks for reading.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.154"
 subject="comment 1"
 date="2014-03-12T17:15:23Z"
 content="""
Well, I suppose this is doable, but the way this scenario is generally handled is to make a transfer repository (which your desktop is serving as here) have a preferred content expression that makes it want files that the client repositories (phones) want, until the files have reached the clients:

In your case you could have:

    (not inallgroup=phones and ($phone_preferred_content)) or ($desktop_preferred_content)

Where `$desktop_preferred_content` is whatever files the desktop actually wants on its own, and `$phone_preferred_content` is a copy of the preferred content setting for the phone.

To make this work, you also need to put your phone in the phones group.
"""]]

repo_name: kakafarm/git

license: unknown


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.154"
 subject="comment 2"
 date="2014-03-12T17:16:59Z"
 content="""
BTW expressing this in the preferred content as above also has the benefit that you can use the new `git annex sync --content` on the desktop and it will automatically get files, copy the right ones to the phone, and drop any then-unwanted files.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm3vKzS4eOWYpKMoYXqMIjNsIg_nYF-loU"
 nickname="Konubinix"
 subject="That does it but not totally"
 date="2014-03-13T11:36:02Z"
 content="""
I agree with the fact that it answers my initial question.

Nevertheless, I guess it is really annoying to have to change the preferred content in two locations each time I change want to modify what is on my phone.
Indeed, I quite often change what should be really present on my phone, depending on several factors (my mood, the time I will have in transports and the phase of the moon).

The reason why I use \"git annex wanted\" is that it is straightforward: I just launch : \"git annex wanted phone include=some/file\" and that's all.

With the solution you propose, I would have to each time additionally launch \"git annex wanted here '(not inallgroup=phones and (include=some/file)) or ($desktop_preferred_content)'\"

Where I probably have to previously launch \"git annex wanted here\" to remember what is the preferred content of my computer (and put it in $desktop_preferred_content).

Another option would be to run \"git annex vicfg\" and edit both fields manually, but IMHO this appears also to be too complicated relatively to the use case.

About your second comment, I really enjoy the idea if \"git annex sync --content\", but it is really long in big repositories. I guess it is because I cannot restrict the command to a directory like I do with get, drop, move and copy.

Besides, the use case wants to get the files whenever they are and copy them only to the phone. With git annex sync --content, the files are also put in the other repositories.

For the time being, I prefer falling back to explicit commands that are much faster when I know a directory to sync \"git annex get --auto directory && git annex copy --auto --to phone directory\".

For example, I just tried \"git annex sync --content phone\" and I killed it after 5 minutes and nothing was copied yet. With the set of two commands above, the synchronization of directory (get --to phone directory + drop --from here directory) took about 3 minutes.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm3vKzS4eOWYpKMoYXqMIjNsIg_nYF-loU"
 nickname="Konubinix"
 subject="Precision of the use case"
 date="2014-03-13T11:47:59Z"
 content="""
I realized that the directory restriction described earlier is not clear. Let me explain it.

Say I have a big repository of files with the following structure.

A/...
B/...
C/...
D/D1/...
D/D2/...
D/D3/...

Imagine that each of A, B, C and D contains a lot of files.

Now imagine that I have often put in the preferred content of my phone files in one of D1, D2 or D3. (for instance, include=D/D1/*)

I implicitly know that I can restrict the command to D, and I can rely on preferred content to know what file from D I have to put into my phone.

Then, I can run
      $ git annex get --auto D
      $ git annex copy --auto --to phone D
      $ git annex drop --auto D

This in my use case takes 3 minutes.

When I run
     $ git annex sync --content phone

git annex goes recursively through A, B and C and that takes a long time (much more than 3 minutes). This time is really wasted since I know I only want to sync files from D.

I cannot test the behavior of \"git annex sync --content\", but if I restrict the sync to phone, will it take the files from the NAS to put them on the phone?
If I don't precise phone in the command line, it will try to sync with other repositories not in sync that I don't want to be in sync for the time being.
"""]]


code:
From git-annex(1), init should precede any operation of git-annex as a safety feature.

However git cloning a repo and running `git annex get` created a new uuid for that repo, which got
propagated via sync.

Is this intended behaviour?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.21"
 subject="comment 1"
 date="2013-07-30T20:24:06Z"
 content="""
From the man page:

> Until a repository (**or one of its remotes**) has been initialized
> git-annex will refuse to operate on it, to avoid accidentially
> using it in a repository that was not intended tohave an annex.

So, yes, it assumes that if a repository has a git-annex branch already, git-annex is being used, and no explicit init is necessary. You can still run `git annex init` after the fact if you like.
"""]]


code: I just started experimenting with git annex, and I found that I would like to have a way to figure out metadata (well, size. Maybe modification date) of a non-local file. I first checked if there is "git annex ls" (which could list known files in an ls-like way) and found "git annex whereis" as somewhat a replacement, but it does not give metadata information.


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2011-11-14T22:46:35Z"
 content="""
When  I want that, I ls -l the file and look at the symlink to the key. Ie, in SHA1-s10481423--efc7eec0d711212842cd6bb8f957e1628146d6ed the size is 10481423 bytes.
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 2"
 date="2011-11-14T22:48:03Z"
 content="""
It might make sense to put this functionality in git annex find. Perhaps a format string with a %s for example.
"""]]


code:
I'm testing out git-annex between a few computers one being a mac (osx 10.9) and a laptop (linux mint 16).  With a vps running git annex as a transfer annex.

Anyway sync is *almost* working...

When I add files on my mac's annex they upload, and go to my laptop as they should.

When I add files on my laptop's annex they upload, and the mac "downloads" them but only creates broken symlinks to the files.

I looked in the log and nothing out of the ordinary is happening... is this a bug with the osx version or what?

**more info**

* the symlink on the osx annex is symlinking to a file in .git/annex/objects
* the osx annex changes the aliases to real files every time I restart the git annex daemon

Thanks!


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 1"
 date="2013-12-12T17:11:59Z"
 content="""
It sounds like the contents of the files are being successfully transferred.

It may be that you're using an old and buggy version of git-annex, in which the direct mode code lost track of things and so failed to update the tree correctly when receiving the content of a file. If so, you could fix it by upgrading git-annex and running `git annex fsck`.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkipQLNyt8RHREHpg2k5wdYeRSCCvSNSBg"
 nickname="Tim"
 subject="I had this same problem too"
 date="2014-01-04T01:00:10Z"
 content="""
I was using the 2013-12-13 build and now I've upgrade to the 2014-01-03 build.  git-annex has symlinked some important workspace.xml files to nowhere.  One looks like this:

workspace.xml -> ../../../.git/annex/objects/M2/8W/SHA256E-s68106--90e9cc4f617c9034db1bf462d058b82c59ade0be58de1d3a3e2f8c02606631c2.xml/SHA256E-s68106--90e9cc4f617c9034db1bf462d058b82c59ade0be58de1d3a3e2f8c02606631c2.xml

The referenced file doesn't exist.  Actually, that entire M2 directory doesn't exist.

This has cost me a lot of time trying to fix this.  I tried git-annex fsck and I get this message:

 ** No known copies exist of XXXXXXX/workspace.xml

\"XXXXXXX\" is a path I needed to obscure

I can see it in the git-annex repo but it is a reference to a symlink that doesn't exist.  Is there any way to get this file back?  git-annex whereis tells me it is nowhere.  This appears to be real data loss.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkipQLNyt8RHREHpg2k5wdYeRSCCvSNSBg"
 nickname="Tim"
 subject="Sorry for the bad formatting on that last post"
 date="2014-01-04T17:16:51Z"
 content="""
I have some additional information that may be helpful.  I had git-annex syncing my Mac with a Debian system.  It appears that at some point in time the Debian system's NFS connection disconnected and then may have reconnected with sshfs.  I don't know if this happened when the syncing issue occurred or not and I can't validate that since the machine was rebooted but it is possible that that caused git-annex on the Mac to choke while syncing and create the phantom symlink.  I am experiencing NFS issues at the moment so it may have contributed to this hiccup.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.35"
 subject="comment 4"
 date="2014-01-06T15:48:36Z"
 content="""
@Tim, you do not seem to be having the problem that the OP described.

It is completely normal for git-annex to represent a file whose content is not present in the repository as a broken symlink.
Typically, git-annex preserves at least one copy of the file in one of your repositories. But if you `git annex add $file; git annex drop --force $file`, you will be left in exactly the situation you describe.

You can use `git annex log $file` to see a log of which repositories contained a copy of the file in the past, and see what times the file was removed from each repository. This might give clues to what operation caused the last copy of the file to be lose.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkipQLNyt8RHREHpg2k5wdYeRSCCvSNSBg"
 nickname="Tim"
 subject="I'll start another thread"
 date="2014-01-06T18:17:44Z"
 content="""
Let me start another thread then and not pollute this one.  Thanks!
"""]]


code:
I've set up git annex on my laptop and on a remote server using the gitolite v3 git-annex branch.
When I 'git annex sync' from my laptop (mac OSX) to the server, and 'git annex copy . --to server' all works fine.

Later, I tried to clone the repository on another machine (linux 2.6.32). again all is well. I 'git annex init' in the new clone and then try to 'git annex sync' -- now I run into problems.

Specifically, the first request for the configlist (when I do 'git annex --debug sync' all is fine until the configlist request) somehow breaks and my ssh session shows:

muxserver_listen bind(): Input/output error

This seems to confuse the client, who now believes that git annex is not installed on the server (it is).

If I issue the same command as git annex via ssh, all seems to work ok with the caveats below:

I have a feeling it's related to the -o 'ControlPersist=yes' argument that git-annex appears to be giving ssh.  If I include this option when I run via ssh, ssh dies:

>ssh git@server -o "ControlMaster=auto" -o "ControlPersist=yes" "git-annex-shell 'configlist' '/~/PRJ'"
command-line: line 0: Bad configuration option: ControlPersist

(note that it appears to be ok to leave "ControlMaster=auto" in).

any thoughts?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="64.134.31.139"
 subject="comment 1"
 date="2013-10-15T18:16:23Z"
 content="""
Since this seems to be a problem with ssh connection caching, you can disable that:

git config annex.sshcaching false

Seems like the problem would be in the version of ssh you have installed on your linux client. Or possibly some interaction between its version and the server's version.

Since linux 2.6.32 is quite old, I'll bet you have an old ssh too.. what version?
"""]]


code:
is it worth sharing back git annex test failures occourring on some windows setup ?
i'm having 3 failures, what kind of report can i produce shall it be of any use ?
is it enough a test log coming from git annex test > test.log 2>&1 ?
thanks again


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.246"
 subject="comment 1"
 date="2013-11-15T20:53:17Z"
 content="""
Yes, it's fine to file a bug about test failures. Be sure you try with a current autobuild.

The redirection you show would work on linux, I don't know about windows. But capturing a transcript of the test output is certainly what I need.
"""]]


code:
Hi,

trying to make git annex work locally with an apache webdav server.

I have the webdav server working without issue on computers. When we try to init the repository there we get the following error:

WEBDAV_USERNAME=user WEBDAV_PASSWORD=xxxxxx git annex initremote webdavtest type=webdav url=http://webdavserver/webdavsgare/annextest4 encryption=none
initremote webdavtest (testing WebDAV server...)

git-annex: WebDAV failed to delete file: "Locked": user error
failed
git-annex: initremote: 1 failed

Does anyone have any thoughts? I can post config of webdav if it helps, though the error I receive in the error_log of apache is as follows:

This resource is locked and an "If:" header was not supplied to allow access to the resource.  [423, #0]

but I can manage the files through command line, web interface and mounted drive with no issue.

thank you in advance.

Damien


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.146"
 subject="comment 1"
 date="2014-02-26T19:40:18Z"
 content="""
The webdav library that git-annex uses does some webdav file locking. When deleting a file, it first locks it. It sounds like your webdav server does not like to delete a file that has been locked. Whether this is in spec I don't know.

I do know that got rid of all the webdav locking stuff in a revamp of git-annex's webdav support 2 days ago! git-annex does not need locking the way it uses webdav.

However, the daily builds are still built with an old version of the DAV library, which still does the locking ... and will probably remain like that for some time. To get the new version without locking, you need to build git-annex from its git master on a system with http://hackage.haskell.org/package/DAV version 0.6.1. Building from source with cabal is one way to do it. I'll be releasing a build for Debian unstable in the next day or two.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm7hS2LGu4sLUxLMdBA16PAMcVO7CDJmjw"
 nickname="Damien"
 subject="comment 2"
 date="2014-03-10T12:16:43Z"
 content="""
I'm curious if anyone has set this up using a local apache webdav server?

I'm stuck and could use some insight.

thanks
Damien
"""]]


code:
I thought I'd followed the walk through when initially setting up my repos.

However I find that I have to do the following to sync my annex's.

    git pull remote master
    git checkout git-annex
    git pull remote git-annex
    git checkout master
    git annex get .

Has something gone wrong? I see no mention of syncing git-annex repos in the walk-through...


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2011-12-06T16:43:29Z"
 content="""
You're taking a very long and strange way to a place that you can reach as follows:

git pull remote
git annex get .

Which is just as shown in [[walkthrough/getting_file_content]].

In particular, \"git pull remote\" first fetches all branches from the remote, including the git-annex branch.
When you say \"git pull remote master\", you're preventing it from fetching the git-annex branch.
If for some reason you want the slightly longer way around, it is:

git pull remote master
git fetch remote git-annex
git annex get .

Or, eqivilantly but with less network connections:

git fetch remote
git merge remote/master
git annex get .

BTW, notice that this is all bog-standard git branch pulling stuff, not specific to git-annex in the least.
Consult your extensive and friendly git documentation for details. :)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnXybLxkPMYpP3yw4b_I6IdC3cKTD-xEdU"
 nickname="Matt"
 subject="comment 2"
 date="2011-12-06T23:23:29Z"
 content="""
Doh! Total brain melt on my part. Thanks for the additional info.  Not taking my time and reading things properly - kept assuming that the full remote pull failed due to the warning:

    You asked to pull from the remote 'rss', but did not specify
    a branch. Because this is not the default configured remote
    for your current branch, you must specify a branch on the command line.

Rookie mistake indeed.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnXybLxkPMYpP3yw4b_I6IdC3cKTD-xEdU"
 nickname="Matt"
 subject="comment 3"
 date="2011-12-21T16:06:25Z"
 content="""
hmmmm - I'm still not sure I get this.

If I'm using a whole bunch of distributed annexs with no central repo, then I can not do a `git pull remote` without either specifying the branch to use or changing default tracked remote via `git branch --set-upstream`.  The former like you note doesn't pull the git-annex branch down the latter only works one-at-a-time.

The docs read to me as though I ought to be able to do a `git pull remote ; git annex get .` using anyone of my distributed annexs.

Am I doing something wrong?  Or is the above correct?
"""]]


code:
[[!comment format=mdwn
 username="http://adamspiers.myopenid.com/"
 nickname="Adam"
 subject="I think Matt is right."
 date="2011-12-23T14:04:44Z"
 content="""
I got bitten by this too.  It seems that the user is expected to fetch
remote git-annex branches themselves, but this is not documented
anywhere.

The man page says of \"git annex merge\":

    Automatically merges any changes from remotes into the git-annex
    branch.

I am not a git newbie, but even so I had incorrectly assumed that git
annex merge would take care of pulling the git-annex branch from the
remote prior to merging, thereby ensuring all versions of the
git-annex branch would be merged, and that the location tracking data
would be synced across all peer repositories.

My master branches do not track any specific upstream branch, because
I am operating in a decentralized fashion.  Therefore the error
message caused by `git pull $remote` succeeded in encouraging me to
instead use `git pull $remote master`, and this excludes the git-annex
branch from the fetch.  Even worse, a git newbie might realise this
and be tempted to do `git pull $remote git-annex`.

Therefore I think it needs to be explicitly documented that

    git fetch $remote
    git merge $remote/master

is required when the local branch doesn't track an upstream branch.
Or maybe a `--fetch` option could be added to `git annex merge` to
perform the fetch from all remotes before running the merge(s).
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 5"
 date="2011-12-23T16:50:26Z"
 content="""
My goal for `git-annex merge` is that users should not need to know about it, so it should not be doing expensive pulls.

I hope that `git annex sync` will grow some useful features to support fully distributed git usage, as being discussed in [[pure_git-annex_only_workflow]]. I still use centralized git to avoid these problems myself.
"""]]


code:
[[!comment format=mdwn
 username="http://adamspiers.myopenid.com/"
 nickname="Adam"
 subject="comment 6"
 date="2011-12-23T17:14:03Z"
 content="""
Extending `git annex sync` would be nice, although auto-commit does not suit every use case, so it would be better not to couple one to the other.
"""]]


code:
[[!comment format=mdwn
 username="http://adamspiers.myopenid.com/"
 nickname="Adam"
 subject="comment 7"
 date="2011-12-23T17:24:58Z"
 content="""
P.S. I see you already [fixed the docs](http://source.git-annex.branchable.com/?p=source.git;a=commitdiff;h=a0227e81f9c82afc12ac1bd1cecd63cc0894d751) - thanks! :)
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="git tweak-fetch"
 date="2011-12-26T18:50:35Z"
 content="""
The git tweak-fetch hook that I have been developing, and hope will be accepted into git soon, provides some abilities that could be used to make \"git pull remote\" always merge remote/master. Normall, git can only be configured to do that merge automatically for one remote (ie, origin). But the tweak-fetch hook can flag arbitrary branches as needing merge.

So, it could always flag tracking branches of the currently checked out branch for merge. This would be enabled by some setting, probably, since it's not necessarily the case that everyone wants to auto-merge when they pull like this. (Which is why git doesn't do it by default after all.)

(The tweak-fetch hook will also entirely eliminate the need to run git annex merge manually, since it can always take care of merging the git-annex branch.)
"""]]


code: Well, the subject pretty much says it all :)


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="fixed that"
 date="2011-10-13T15:36:59Z"
 content="""
:)
"""]]


code: It was fast to git annex a bunch of files. But git unannex seems a lot slower. Is there a faster way to get files out of git annex? Or to replace the symlinks with real files and then I could just remove the .git directory? I shouldn't have put so many in as a test but they are there now.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 subject="comment 1"
 date="2012-08-02T16:58:41Z"
 content="""
It currently commits once per file, which is slow. `Command/Unannex.hs` explains why this is necessary. If I think of a way to avoid that, I will.
"""]]


code:
I propose to add some functionality to git-annex, to automatically "throttle" the data retrieval from amazon glacier to stay within the daily free retrieval allowance. If someone would need to get his/her files faster, there should be an option to disable this throttling (or even better, specify the retrieval rate).

As far as I understand glacier, this could be implemented using range retrievals. In short range retrievals enable you, to only retrieve a part of an archive in glacier. This can be used to only retrieve / request so much data, that you stay within the free retrieval allowance. ( please see [Q: Why would I retrieve only a range of an archive?](http://aws.amazon.com/glacier/faqs/#Why_would_I_retrieve_only_a_range_of_an_archive) ).

This would be somewhat similar to the [smart retrieval feature in cloudberry](http://blog.cloudberrylab.com/2012/12/introducing-smart-restore-for-amazon.html) .


code:
Hello,

I'm still looking for a way to version control the metadata (title, artist, album name, ...) of my MP3s, I wonder if git annex could help for this problem ?
The method I use now (without git annex) is to export the MP3 metadata to an textual format with one line per tag.
It's this textual file I handles with git.
The problem is to handle the mapping between the orignal file and the export file with file renaming or moving.
I consider to use the checksum of the audio content (without metadata, this checksum never changes) to handle this problem.
Maybe git annex has a different approach (better) to solve this problem ?
How git annex would be use to solve the orignal problem ?

Regards,
Emmanuel Berry


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.154"
 subject="comment 1"
 date="2014-03-18T19:29:53Z"
 content="""
Check out [[metadata]]
"""]]


code:
I'm wondering how easy the addition of hashing to the directories of the objects would be.

Currently a tree directory structure becomes a flat two level tree under the .git/annex/objects directory ([[internals]]).  This, through the 555 mode on the directory prevents the accidental destruction of content, which is _good_.  However file and directory numbers soon add up in there and as such any file-systems with sub directory limitations will quickly realize the limit (certainly quicker than maybe expected).

Suggestion is therefore to change from

 `.git/annex/objects/SHA1:123456789abcdef0123456789abcdef012345678/SHA1:123456789abcdef0123456789abcdef012345678`

to

 `.git/annex/objects/SHA1:1/2/3456789abcdef0123456789abcdef012345678/SHA1:123456789abcdef0123456789abcdef012345678`

or anything in between to a paranoid

 `.git/annex/objects/SHA1:123/456/789/abc/def/012/345/678/9ab/cde/f01/234/5678/SHA1:123456789abcdef0123456789abcdef012345678`

Also the use of a colon specifically breaks FAT32 ([[bugs/fat_support]]), must it be a colon or could an extra directory be used? i.e. `.git/annex/objects/SHA1/*/...`

`git annex init` could also create all but the last level directory on initialization. I'm thinking `SHA1/1/1, SHA1/1/2, ..., SHA256/f/f, ..., URL/f/f, ..., WORM/f/f`

> This is done now with a 2-level hash. It also hashes .git-annex/ log
> files which were the worse problem really. Scales to hundreds of millions
> of files with each dir having 1024 or fewer contents. Example:
>
> `me -> .git/annex/objects/71/9t/WORM-s3-m1300247299--me/WORM-s3-m1300247299--me`
>
> --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2011-03-14T16:12:49Z"
 content="""
My experience is that modern filesystems are not going to have many issues with tens to hundreds of thousands of items in the directory. However, if a transition does happen for FAT support I will consider adding hashing. Although getting a good balanced hash in general without, say, checksumming the filename and taking part of the checksum, is difficult.

I prefer to keep all the metadata in the filename, as this eases recovery if the files end up in lost+found. So while \"SHA/\" is a nice workaround for the FAT colon problem, I'll be doing something else. (What I'm not sure yet.)

There is no point in creating unused hash directories on initialization. If anything, with a bad filesystem that just guarantees worst performance from the beginning..
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 2"
 date="2011-03-15T13:52:16Z"
 content="""
Can't you just use an underscore instead of a colon?

Would it be feasible to split directories dynamically? I.e. start with SHA1_123456789abcdef0123456789abcdef012345678/SHA1_123456789abcdef0123456789abcdef012345678 and, at a certain cut-off point, switch to shorter directory names? This could even be done per subdirectory and based purely on a locally-configured number. Different annexes on different file systems or with different file subsets might even have different thresholds. This would ensure scale while not forcing you to segment from the start. Also, while segmenting with longer directory names means a flatter tree, segments longer than four characters might not make too much sense. Segmenting too often could lead to some directories becoming too populated, bringing us back to the dynamic segmentation.

All of the above would make merging annexes by hand a _lot_ harder, but I don't know if this is a valid use case. And if all else fails, one could merge everything with the unsegemented directory names and start again from there.

-- RichiH
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 3"
 date="2011-03-16T03:13:39Z"
 content="""
It is unfortunatly not possible to do system-dependant hashing, so long as git-annex stores symlinks to the content in git.

It might be possible to start without hashing, and add hashing for new files after a cutoff point. It would add complexity.

I'm currently looking at a 2 character hash directory segment, based on an md5sum of the key, which splits it into 1024 buckets. git uses just 256 buckets for its object directory, but then its objects tend to get packed away. I sorta hope that one level is enough, but guess I could go to 2 levels (objects/ab/cd/key), which would provide 1048576 buckets, probably plenty, as if you are storing more than a million files, you are probably using a modern enough system to have a filesystem that doesn't need hashing.
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 4"
 date="2011-03-16T04:06:19Z"
 content="""
The .git-annex/ directory is what really needs hashing.

Consider that when git looks for changes in there, it has to scan every file in the directory. With hashing, it should be able to more quickly identify just the subdirectories that contained changed files, by the directory mtimes.

And the real kicker is that when committing there, git has to create a tree object containing every single file, even if only 1 file changed. That will be a lot of extra work; with hashed subdirs it will instead create just 2 or 3 small tree objects leading down to the changed file. (Probably these trees both pack down to similar size pack files, not sure.)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 5"
 date="2011-03-16T15:47:17Z"
 content="""
If you can't segment the names retroactively, it's better to start with segmenting, imo.

As subdirectories are cheap, going with ab/cd/rest or even ab/cd/ef/rest by default wouldn't hurt.

Your point about git not needing to create as many tree objects is a kicker indeed. If I were you, I would default to segmentation.
"""]]


code:
## Question
Can git-annex run passively with an existing repo?

## Scenario
We have an existing web application with large binary assets spread throughout modules in the repo. The project is in constant development with weekly deploys to production and there are many developers working on the project.

## Goal
We need to maintain the directory structure for these assets without actually committing large binaries to the main remote (hosted on GitHub). I need a solution that has a low barrier of entry when on-boarding new developers. I was thinking maybe a script the is executed on every commit that would filter binaries files based on extensions and commit them to the git-annex remote instead. We are flexible on the type of remote storage type (SSH, S3, etc)

## Notes
I have gone through the last couple months of forum posts and done a bit of Googling but have come up empty. If anyone can point me in the right direction that would be great.
Thanks!


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn7gQ1zZDdWhXy9H51W2krZYShNmKL3qfM"
 nickname="Karsten"
 subject="comment 1"
 date="2013-06-01T09:21:54Z"
 content="""
Mixing files controlled by git directly and files controlled by git annex is possible, simply 'git annex add' the large files and 'git add' the rest. You will not be able to use direct mode and you should not run the git annex assisstant, as it will add all files to the annex.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9FMhhhM2sJ68Zjx_RmWd8cTdpI-mrkbE"
 nickname="Hans"
 subject="what about with git-svn?"
 date="2013-06-04T08:00:15Z"
 content="""
I'm trying to wrap my head around a similar situation.
I've tested this by git cloning my repo; the symlinks are copied, and end up broken because the annex directory under .git doesn't exist in the new repo.

So to be specific: can I conclude that when I use git to copy my repo, as long as I don't explicitly use git-annex in the process, I end up with a 'bare' git repo and I don't have to worry about my annexed files coming along?

Also: anyone know of anything that would be different about using git-svn? (i.e. git svn dcommit to push my changes to an svn repo)?
"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 3"
 date="2013-06-04T12:29:04Z"
 content="""
As long as you're not using Direct Mode, then a git-annex repo is just an absolutely ordinary git repo, which happens to contain some symlinks that point to .git/annex/objects/*

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9J51AO9t75xN5k0sJgg8taUo4y0a4hpQ"
 nickname="Daniel"
 subject="comment 4"
 date="2013-06-07T23:02:29Z"
 content="""
I'm trying to find a way to prevent developers from adding certain file types to the main repo. Is there something like a pre-add hook that could be used?

I found this in another forum but I'm not sure how it was intended to be implemented.

```
git check-attr addtoannex \"$FILE\" | grep -q \": set$\"
if [ $? -eq 0 ]; then
    git annex add \"$FILE\"
else
    git add \"$FILE\"
fi
```

[http://git-annex.branchable.com/forum/Let_watch_selectively_annex_files/](http://git-annex.branchable.com/forum/Let_watch_selectively_annex_files/)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 5"
 date="2013-06-11T14:30:08Z"
 content="""
I think the most useful thing for people in this thread to know about is [[tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant]].

This doesn't solve it at the command line, where you still need to choose between git add and git annex add, but you can use git annex watch to automatically commit small files to git, and large files to the git annex.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9J51AO9t75xN5k0sJgg8taUo4y0a4hpQ"
 nickname="Daniel"
 subject="comment 6"
 date="2013-06-11T22:28:52Z"
 content="""
This is what I ended up doing.
[https://gist.github.com/ifnull/5761255](https://gist.github.com/ifnull/5761255)

Basically you just add the extensions of the files you want to exclude to .gitignore_large_binaries and run \"git a .\" instead of \"git add .\"

    #######################
    # Setup
    #######################
    mkdir annex-test
    cd annex-test
    git init
    git annex init master

    #######################
    # Fab setup task
    #######################
    git config --local core.excludesfile ./.gitignore_large_binaries
    git config --local alias.a '! sh ./git-add.sh'

    #######################
    # git a (git-add.sh)
    #######################

    # Generate annex include arg from .gitignore_large_binaries
    include_str=\"--include='.lazy'\";

    while read line
    do
        if [[ \"$line\" != *\"#\"* ]] && [[ \"$line\" != \"\" ]]; then
            include_str=\"$include_str --or --include=${line}\";
        fi
    done < \"./.gitignore_large_binaries\"

    # git annex add
    git config --local core.excludesfile ./.gitignore;
    git annex add $1 $include_str;

    # git add
    git config --local core.excludesfile ./.gitignore_large_binaries;
    git add $1
"""]]


code:
I am working on rehauling my backup scripts here, and I was originally doing an rsync of everything to an external drive, but now I think I can be smarted and skip my annexes in that rsync, and use git-annex superpowers to do the sync instead.

I have added this to my backup script:

    ( cd /srv/video && git annex copy --to backup . )

And it works, so yaaay. :) However, I feel it could be faster. This seems to check each file one at a time, but doesn't git-annex keep a state of the remote internally, which would allow it to copy over only the missing files, without checking if each file is present individually?

I mean, it's still pretty fast considering the dataset, but I wonder if there isn't some fast/easier way. Keep in mind I am hesitant of using the assistant for this because I am [[confused_about_external_drives]], the [[bugs/webapp takes 100% of the cpu]] and the [[bugs/assistant eats all CPU]]. I would prefer to script this anyways. --[[anarcat]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 1"
 date="2014-01-01T20:43:05Z"
 content="""
Pulling location tracking data out of git is unlikely to be faster than statting a single file on the drive, unless the drive is dead slow.

If you really want to force it to use the location tracking information, use `--not --in backup`
"""]]


code:
I have an encrypted S3 remote with which I recently ran:

$> git annex move . --from cloud

which moved all my files to the local repo. When I looked at my S3 bucket, I saw some files left. I then ran:

$> git annex unused --from cloud

to get all the unused data. I dropped each unused data from cloud so that now I think git annex believes it has nothing on the cloud remote.

However there is still one file left on the bucket. Is this normal? (if not I'll create a bug report)
More importantly, how can I decrypt this file so I can run git log -S'KEY' to figure out what it is (I want to make sure I haven't lost any data).

To be clear, when I currently run 'git annex move . --from cloud' it moves nothing and when I run 'git annex unused --from cloud' it reports no unused data.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkiAsTXFXZbLE8iyy6yDtvz4MPFbzsk3c0"
 nickname="Tony"
 subject="comment 1"
 date="2013-07-27T22:14:11Z"
 content="""
one more thing, I forgot to mention I use gpg keys for encryption on the remote and not a shared key.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.246.110"
 subject="comment 2"
 date="2013-07-27T22:30:50Z"
 content="""
I suppose this could happen if you have a tag or a branch or another repository that still refers to the file that's stored in S3. It would then not be found by `unused`.

git-annex 4.20130709 has a nice new --all switch you can use: `git annex move --all --from cloud`

If that doesn't move the file, nothing in the entire history of the git repository refers to it. The question then would be how did that file get there.

It's by [[design|design/encryption]] not possible to get from the name of an encrypted file back to the key, unless you already know all the possible keys that the file could be. It's possible to decrypt the content if you have the gpg key and the git repository, but this space is a bit too small to explain every step in doing so, and I have some nice code that explains how in detail. ;)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkiAsTXFXZbLE8iyy6yDtvz4MPFbzsk3c0"
 nickname="Tony"
 subject="comment 3"
 date="2013-07-27T22:47:14Z"
 content="""
thanks, running

'git annex move --all --from cloud'

grabbed the one remaining file from the bucket.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.246.110"
 subject="comment 4"
 date="2013-07-27T22:52:00Z"
 content="""
Great! And it should have also printed out the key it downloaded, if you're curious about looking where that file is used in your git repository.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkiAsTXFXZbLE8iyy6yDtvz4MPFbzsk3c0"
 nickname="Tony"
 subject="comment 5"
 date="2013-07-27T23:02:19Z"
 content="""
yeah it did print out the name of the file so I was able to verify that it was nothing I was worried about losing. thanks again :D
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.246.110"
 subject="comment 6"
 date="2013-07-27T23:06:38Z"
 content="""
I'm curious, why didn't `git annex unused --from cloud` show the file? Was it a tag/branch/etc still referring to it?
"""]]


code:
I fear that while using git annex I will at some point accidentally `git add` some small files and not notice it until the only way to fix the problem is to rewrite history. What would be the best way to prevent myself from ever `git add`-ing a file into my annex repository instead of `git annex add`-ing it?

And secondly, how can I best search in my git annex repository whether I already did this mistake in the past or not? Currently I'm using this which returns everything that's not a symlink or a git submodule:

    git ls-files -s | awk ' $1 != 120000 && $1 != 160000 { print $4 }'


code:
[[!comment format=mdwn
 username="Xyem"
 ip="178.79.137.64"
 subject="comment 1"
 date="2014-06-25T10:35:07Z"
 content="""
I'm not sure if you can prevent 'git add' but you can at least prevent it getting commited with the pre-commit hook. Mine is this:

    #!/bin/sh

    FILES=$(find -name \".git\" -prune -o -type f -not -name \".gitignore\" -print);
    FILE_COUNT=$(echo -n \"$FILES\" | wc -l)

    if [ $FILE_COUNT -gt 0 ]; then
            echo \"$FILE_COUNT non-symlink files found.\"
            echo \"$FILES\"
            exit 1
    fi

    # automatically configured by git-annex
    git annex pre-commit .
"""]]


code:
[[!comment format=mdwn
 username="josch"
 ip="2001:638:709:5:2ad2:44ff:fe4b:56aa"
 subject="comment 2"
 date="2014-06-27T04:50:03Z"
 content="""
Xyem, your method does not work well because it finds all regular files in the whole annex directory and does not only check for those that are to be committed.
"""]]


code:
[[!comment format=mdwn
 username="Xyem"
 ip="178.79.137.64"
 subject="comment 3"
 date="2014-06-27T12:02:08Z"
 content="""
That is intentional. In my use case, there should never be any regular files in this particular annex at all. If there are, it means I have missed something! :)
"""]]


code:
It would be nice to have a way to drop files without leaving broken symlinks around, at least while in direct mode.

Here is my user case. I have a collection of music CDs ripped in FLAC formats. At the same time I convert all these files to MP3 files to that I can use them in various other devices and to save space.

The problem is that if I `git annex drop` the FLAC files once they are converted and copied, they leave broken symlinks around; this result in various annoying error messages in almost all the music playback I tried. At the same time, if I `rm` or `git rm` these symlinks, the content of these files will be removed also from the remotes as they are signalled as no longer wanted.

Couldn't git-annex keep a separate index of files that have been removed but are meant to be kept?


code:
[[!comment format=mdwn
 username="http://svario.it/gioele"
 ip="2001:1418:100:262::2"
 subject="Maybe using branches?"
 date="2013-05-16T10:22:48Z"
 content="""
A suggestion from #git-annex:

> <mhameed> i think others are already doing this, using branches, but not sure if we have a writeup/tutorial of it
>
> <mhameed> gioele:  instead of working on master branch, switch to laptop branch, git rm the files that are not wanted and commit
>
> <gioele> mhameed: I'm lazy, I'd like git-annex to do that for me :)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-05-19T20:43:43Z"
 content="""
The original poster seems to have a misunderstanding of what git-annex does with the content of files. Deleting a file does not remove the content of the file. You can always use git to check out a previous version of a file, and the content from the annex will still be there (or you can `git annex get` it to get it from whereever git-annex stored the content).

The only exception to this rule is is you, manually, chose to run `git annex unused` and then `git annex dropunused`. That can delete the contents of files, when no branch or tag refers to them. As long as some branch refers to the content of the files, even if it's not the currently checked out branch, the file content is retained.

So a branch *is* the \"index of files that have been removed but are wanted to be kept\".

For example, you could do:

[[!format sh \"\"\"
git checkout -b keepflacs

cdrom-rip-command
git annex add *.flac
git commit -m \"ripped a cd\"
flac-convert-command
git annex add *.mp3
git commit -m \"converted to mp3\"

git checkout master
git merge keepflacs
git rm *.flac
git commit -m \"removed flac files from this branch. They are still available in the keepflacs branch\"
\"\"\"]]

As long as you always switch to the keepflacs branch to add flac files, and never merge the master branch into keepflacs,
but only merge keepflacs into master, keepflacs will have every flac file you have ripped. And so git-annex will retain those files
even when you `git annex unused; git-annex dropunused`.

Or, just make a promise to yourself that you'll never run `git-annex unused`, similarly to how you'd probably never run `rm -rf .git/objects/$rand`, and you don't need the branches; like git, git-annex will retain all data that has ever been checked into it.

(The branches are still a good idea. For one thing, they let you run `git annex fsck` and other repository maintenance commands with the keepflacs branch checked out.)

I am going to move this thread to the [[forum]], because it's not really a TODO item, but is something others may want to read.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 3"
 date="2013-05-20T17:46:08Z"
 content="""
Erm, I missed that you want to use direct mode. All this fun with git won't really work in direct mode, and indeed direct mode is not able to guarantee that old versions of modified files are retained.

Direct mode is nice for some applications involving syncing and less than ideal devices, but not for this.
"""]]


code:
I have created two git-annexes (one on my laptop and one on my work pc) The are using nearly the same files with some files existing only on the pc but not on the laptop as they are too large for the laptop. How do I get the two into sync. I found the manual in the tips section but that will create normal git-annexes with all files linked only. However, I would like to have my exiting file structure unchanged (the way the assistant does it by default)

any ideas?

Gregor


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="71.80.94.56"
 subject="comment 1"
 date="2014-02-07T19:11:00Z"
 content="""
The webapp will help you do this, go to Add Another Repository -> Share With Your Other Devices.
"""]]


code:
I am using the importfeed [1] functionality.

How am I supposed to update the feed/feeds?

I understand that running

    cd annex; git annex importfeed http://url/to/podcast

a second time will 'do the right thing'.  But that is cumbersome as I have to know the url again.  Is there sth like git annex updatefeeds?

[1] http://joeyh.name/blog/entry/git-annex_as_a_podcatcher/


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.63"
 subject="comment 1"
 date="2013-08-26T18:11:07Z"
 content="""
I have purposefully kept a list of feeds out of git-annex. I handle this by having a `feeds` file, which I check into git. Then a cron job runs: `xargs git-annex importfeed < feeds`
"""]]


code:
[[!comment format=mdwn
 username="ringprince"
 ip="134.76.140.110"
 subject="comment 2"
 date="2013-08-26T19:04:21Z"
 content="""
Thanks for that.  I'll do the same ;-)
"""]]


code:
[[!comment format=mdwn
 username="http://a-or-b.myopenid.com/"
 ip="203.45.2.230"
 subject="comment 3"
 date="2013-08-28T01:32:38Z"
 content="""
One thing I do slightly differently is put comments in my ```feeds``` file so I know what feed is what...

    # NPR - Wait Wait... Don't Tell Me!
    http://www.npr.org/rss/podcast.php?id=35
    # NPR - Car Talk
    http://www.npr.org/rss/podcast.php?id=510208

Then I use

    grep -v '^#' feeds| xargs git-annex importfeed --relaxed

I reckon that makes maintenance easier.
"""]]


code:
[[!comment format=mdwn
 username="arand"
 ip="130.243.226.21"
 subject="comment 4"
 date="2013-09-01T18:06:51Z"
 content="""
Yet another solution, keeping it all in one script

    #!/bin/sh

    while IFS= read line
    do
            test -n \"${line%%#*}\" && echo git annex importfeed --relaxed \"$line\"
    done <<EOF
    # FooCast - Alice & Bob
    http://feeds.foo.com/foocast
    # FrobCast
    http://meep.moop.com/feed
    EOF
"""]]


code:
Hi,

I have two repo one in direct (on windows) and one in indirect mode. From time to time the files in the direct repo are replaced by empty files however running git annex fsck always solves it.
The problem is that today I did run git annex sync before running git annex fsck and git annex has then created two -variants for each of my files one empty and one with the content.
I guess the easier for me is to just scrap that repo and make a new one however how do I prevent the changes to propagate now? I guess that if I now run git annex sync on my other repo all those small files are going to have linked created for them there as well.

I hope this is clear,
Thanks in advance.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.244"
 subject="comment 1"
 date="2014-04-02T20:02:19Z"
 content="""
I don't see any need to scrap the repository. Since you have an indirect mode repository, you can use `git log` in there to find commits you don't like, and run `git revert` to revert them. So if a bad commit comes down from windows, you can just undo it. That's why we use git, yes?

I'm much more curious about the circumstances that cause empty files to end up in the direct mode repository.
"""]]


code:
[[!comment format=mdwn
 username="bvaa"
 ip="86.82.68.238"
 subject="comment 2"
 date="2014-04-03T19:20:31Z"
 content="""
Ok it makes indeed sense.
Regarding the empty files I am going to try to find out where that happens and if I can reproduce.
Thanks.
"""]]


code: Are versions 0.14 and 0.20110522 incompatible? I can't seem to copy files from a system running 0.14 to one running 20110522.


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2011-06-08T00:40:54Z"
 content="""
They are not. See [[upgrades]]
"""]]


code:
I would like to be able to use git-annex to not only sync binary files for a project but also to be able to use it more like git, that is adding manually files to index and only commiting that.
I think it is still possible in indirect-mode to manage files like I want, but as I am stuck in windows, I would like to be able to do that in direct-mode...
Due to the safety measures taken to avoid doing bad things in direct-mode, it doesn't seem possible to commit (and push) files without syncing the whole repository.

Is it currently possible to do what I would like to do, or will it be possible in the future ?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 1"
 date="2013-12-12T17:01:50Z"
 content="""
You can `git annex add` some files and leave others not added, and `git annex sync` will commit only the added ones.

What is missing is a way to have changed some files that were added, and avoid `git annex sync` committing them.

There is no technical reason this cannot be supported in direct mode, there is just no user interface built to do it.

One way is to temporarily bypass the direct mode guard, by using the appropriate git command line option to set core.bare=false temporarily while committing files. Note that it's then up to you to avoid staging the entire content of the file into the index (so no `git commit -a` etc).
"""]]


code:
[[!comment format=mdwn
 username="Renaud"
 ip="182.171.224.143"
 subject="comment 2"
 date="2013-12-13T06:16:12Z"
 content="""
Thanks for the reply!

I tried to commit manually but I got confused by the (recently introduced?) annex/direct/master branch and I am not sure which branch I need to commit/merge/pull/push my changes to.
I also got blocked when pulling from origin fail due to file changes needed to be overwritten.
I see git-annex cleverly merges those in .git/annex/merge folder so I could try doing the same but I am not too confortable doing a large part of git-annex's job...

Is there a simpler way to sync with a remote repository?
"""]]


code:
I have a direct repository with the assistant running (v4.20131002). The repo is in the backup group and should be grabbing every known annex file. Yet `git annex status` says:

    local annex keys: 1386
    local annex size: 94.53 gigabytes
    known annex keys: 1387
    known annex size: 94.53 gigabytes

As you can seem there is one more known annex key than there are local ones. Neither `git annex get` nor `git annex sync` changes the numbers. According to `tree` the repo dir contains exactly 1387 files, which means that the missing file must exist on disk.

1) How do I find out which is the file in question? How do I get it synchronized?

Yesterday there were 1376 known annex keys. Today there are 1387. For some reason the number of keys went up by 11, and I would like to know how and why. The log suggests that a couple of files have been added from inside the repo directory, but I am positive that these files already existed in the annex directory yesterday, and that I ran `git annex sync` manually without the number of local and/or known keys going up.

2) Why weren't the files synced before? Is there a way to find out whether they were or were not part of the annex yesterday - maybe list all files sorted by the date they were added?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 1"
 date="2013-10-28T16:56:20Z"
 content="""
I think you're stressing way too much about a number.

It's quite possible for the known and local key counts to diverge while no data is missing. For example:

<pre>
joey@darkstar:~/tmp/test>touch foo
joey@darkstar:~/tmp/test>touch bar
joey@darkstar:~/tmp/test>git annex add foo bar

local annex keys: 1
known annex keys: 2
</pre>

"""]]


code:
[[!comment format=mdwn
 username="RaspberryPie"
 ip="77.247.181.165"
 subject="comment 2"
 date="2013-10-28T18:31:21Z"
 content="""
I don't think I'm stressing too much. Rather, I'm inquiring about why git-annex is giving me wrong (or at least unclear) output that makes me worry about losing data. I plan to trust git-annex with a lot of important files, and in order to do that I want to make sure it's trustworthy. And that's where the inaccurate key numbers get in the way.

So why do the local and known numbers diverge anyway? Would you consider adjusting the status output to be more accurate, or at least less confusing? I just want to be able to check that every single file in the repo is being backed up.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 3"
 date="2013-10-28T19:06:00Z"
 content="""
I have adjusted the labels to be more clear.
"""]]


code:
[[!comment format=mdwn
 username="RaspberryPie"
 ip="77.247.181.165"
 subject="comment 4"
 date="2013-10-28T19:24:12Z"
 content="""
Thanks, I'll check it out. And if you ever got around to introduce a means that can answer the above-mentioned integrity questions, that'd be really awesome.
"""]]


code: Just saw that there is now a linux standalone tarball, I've a SL5 (RHEL5) based machine which I can churn out git-annex binaries if there is interest.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.149"
 subject="comment 1"
 date="2012-09-29T16:03:36Z"
 content="""
I'll be curious to see if the standalone tarball works on such a system.. it might need a newer libc
than RHEL5 has.

Conversely, that'd make it a good system to build the tarball on, so it's built to work with a nice old libc. :)

I have a machine where I could put chroots to do both 64 bit and 32 bit builds, but I don't know if I'll automate it there. Too lazy..
"""]]


code:
Hey !
I am trying to find ways to use git annex as a tool to share big binary files for our development projects (such as PSD files).
Our team has artists that will not use command line tools so I have been testing git-annex assistant with a lot of hope.
However all the tests I am doing are just failing. °.° '

in this thread I would like to focus on what would seem to me as an easy first approach : local pairing.

So my team and I are on mac. I am just trying to sync 2 computers on the same network. But the process is stuck on the "local pairing in process" step, just after I enter the secret phrase.

I have enabled sshd (preferences -> remote login) on both computers, checked the firewall to authorize git-annex, so I am probably missing something here.
the logs on my computer does not say anything usefull ([2013-12-04 22:04:34 CET] main: Pairing in progress) but on the other computer I have this :

git-annex: ssh-keygen failed
[2013-12-04 22:04:33 CET] PairListener: utku@MacBook-Air-de-utku.local:~/Documents/git-test is sending a pair request.
dyld: lazy symbol binding failed: Symbol not found: ___strlcpy_chk
 Referenced from: /Applications/git-annex.app/Contents/MacOS/bundle/ssh-keygen
 Expected in: /usr/lib/libSystem.B.dylib

dyld: Symbol not found: ___strlcpy_chk
 Referenced from: /Applications/git-annex.app/Contents/MacOS/bundle/ssh-keygen
 Expected in: /usr/lib/libSystem.B.dylib

git-annex: ssh-keygen failed

Don't know if that helps. I don't know which info I need to provide.

cheers


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnfC02DTOZygFDDvvg0maRciYYMhEIFgkw"
 nickname="Utku"
 subject="comment 10"
 date="2013-12-15T09:51:28Z"
 content="""
unfortunately, it is not working on the computer where ssh was not working
I click an got annex and nothing seems to open
I try to go on the system logs but I see nothing really obvious. The only log that may be related would be :
15/12/13 10:46:23,230 Dock[166]: find_shared_window: WID 21425
15/12/13 10:46:23,230 Dock[166]: CGSGetWindowTags: Invalid window 0x53b1

if you tell me where to look at I can give you something more useful.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 1"
 date="2013-12-12T17:47:30Z"
 content="""
What version of OSX is the failing computer running? What version of git-annex? How did you install git-annex on it?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnfC02DTOZygFDDvvg0maRciYYMhEIFgkw"
 nickname="Utku"
 subject="comment 2"
 date="2013-12-12T20:08:50Z"
 content="""
hello
both computers are on mac os maverick
latest version of git annex, and we used the git annex assistant dmg found on the install section...
I am on a personnal home router so I don't think there is any port that might be blocked by a firewall for local connections...

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 3"
 date="2013-12-12T20:37:55Z"
 content="""
It's not a firewall or network problem, what you pasted earlier shows that the ssh-keygen command bundled in the git annex dmg is trying to use a symbol that is not present in your libSystem library.

I am puzzled why ssh-keygen apparently works on one of your macs and not on the other, if they indeed both have the same build of git-annex installed.

Please paste output of `otool -L /Applications/git-annex.app/Contents/MacOS/bundle/ssh-keygen` on the mac with the problem.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnfC02DTOZygFDDvvg0maRciYYMhEIFgkw"
 nickname="Utku"
 subject="comment 4"
 date="2013-12-12T20:58:05Z"
 content="""
so as I don't really know which mac has a problem, here is what I get on both computers (looks they are quite the same):

/Applications/git-annex.app/Contents/MacOS/bundle/ssh-keygen:
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 855.0.0)
	/System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos (compatibility version 5.0.0, current version 6.0.0)
	/System/Library/Frameworks/OpenDirectory.framework/Versions/A/OpenDirectory (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 55456.0.0)
	@executable_path/O (compatibility version 1.0.0, current version 1.0.0)
	@executable_path/P (compatibility version 3.0.0, current version 3.0.0)
	@executable_path/B (compatibility version 1.0.0, current version 1.2.5)
	@executable_path/L (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

/Applications/git-annex.app/Contents/MacOS/bundle/ssh-keygen:
       /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 855.0.0)
       /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos (compatibility version 5.0.0, current version 6.0.0)
       /System/Library/Frameworks/OpenDirectory.framework/Versions/A/OpenDirectory (compatibility version 1.0.0, current version 1.0.0)
       /System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 55456.0.0)
       @executable_path/O (compatibility version 1.0.0, current version 1.0.0)
       @executable_path/P (compatibility version 3.0.0, current version 3.0.0)
       @executable_path/B (compatibility version 1.0.0, current version 1.2.5)
       @executable_path/L (compatibility version 1.0.0, current version 1.0.0)
       /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 5"
 date="2013-12-12T21:40:16Z"
 content="""
Well, I was wrong, we don't know only one of the 2 has the problem. The other one will probably not have run ssh-keygen yet.

Can you run `/Applications/git-annex.app/Contents/MacOS/bundle/ssh-keygen --help` on both of them and see if it manages to print its help, or fails with an error?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnfC02DTOZygFDDvvg0maRciYYMhEIFgkw"
 nickname="Utku"
 subject="comment 6"
 date="2013-12-12T21:50:15Z"
 content="""
ok so it is ok on my computer, but here is what I got on the other one :
dyld: lazy symbol binding failed: Symbol not found: ___strlcpy_chk
 Referenced from: /Applications/git-annex.app/Contents/MacOS/bundle/ssh-keygen
 Expected in: /usr/lib/libSystem.B.dylib

dyld: Symbol not found: ___strlcpy_chk
 Referenced from: /Applications/git-annex.app/Contents/MacOS/bundle/ssh-keygen
 Expected in: /usr/lib/libSystem.B.dylib

Trace/BPT trap: 5
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 7"
 date="2013-12-12T21:53:56Z"
 content="""
So it does only fail on one machine and not the other. I wonder what is different?

Can you also try running `/Volumes/git-annex/git-annex.app/Contents/MacOS/bundle/cp` on both? (I think it uses the same symbol.)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnfC02DTOZygFDDvvg0maRciYYMhEIFgkw"
 nickname="Utku"
 subject="comment 8"
 date="2013-12-12T22:03:54Z"
 content="""
you mean /Applications/git-annex.app/Contents/MacOS/bundle/cp ?

on both computers it prints :
usage: cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file target_file
       cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file ... target_directory

/Volumes/git-annex/git-annex.app/Contents/MacOS/bundle/cp doesn't work

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 9"
 date="2013-12-13T18:52:40Z"
 content="""
As a workaround, I have removed ssh-keygen and ssh from the OSX DMG. AIUI, OSX ships ssh by default anyway.
"""]]


code:
Here's a simple example on a repository with three branches, where we'll be adding images-annex as a subtree into master.

    $ git branch
        git-annex
        images-annex
      * master
    $ git subtree add --squash --prefix=images/ images-annex
      Added dir 'images'
    $ ls
      FILE_A        FILE_B        images/

...checkout images-annex, make changes, commit...

    $ git checkout master
    $ git subtree pull --squash --prefix=images/ . images-annex
      From .
      * branch images-annex -> FETCH_HEAD
      Merge made by the 'recursive' strategy.
      ...(files created/modified/etc)

I have tried a few different methods for merging the subtree in and so far have not been able to keep git-annex links up to date. Running `git-annex fix .` does what it's supposed to but then git sees everything as modified. Is this entirely the expected behavior because of the --prefix? I have not used subtrees much before but the model appears to be very helpful for what I'm trying to do.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="24.159.78.125"
 subject="comment 1"
 date="2014-07-30T14:55:16Z"
 content="""
Known bug: [[bugs/Git_annexed_files_symlink_are_wrong_when_submodule_is_not_in_the_same_path]]

I don't think there's much likelyhood of a fix though. Using direct mode probably works around the problem. Or you can use something like myrepos instead of git subtrees.
"""]]


code:
I recently started experimenting with git annex, adding files that I've had
floating across several computers to repositories. During the testing I had
a few occasions where I wrecked a repository somehow, and decided to wipe it
and start anew (at this point there was no important files in them so I thought
this is the easiest way). Well, as it turns out this interacts badly with location
tracking, since now `git annex whereis` shows files residing in all those destroyed
repositories, all having same names as some existing repositories. This makes it hard
to follow whether a repo actually has a file, or was the file only seen in some dead
repo with the same name.

I planned on cleaning this up by looking up the UUIDs of the now stable, existing
repos and untrusting all the dead copies (they should effectively disappear from
git annex´s output then, right?), but I didn't find an easy way to look up the UUID
of the current repository (maybe this could be included in `git annex status`?)
I also noticed that untrust cannot remove the trust based on the UUID -- if I try
it I simply get "there is no git remote named "11908472-...", so I guess untrust
works with git remote names, which I find a bit confusing, since trust.log logs the
trust levels based on the UUID. I could just write into trust.log manually, but I'm
unsure how the changes would get propagated.

What should I do? As a related wishlist item I would ask for some additional mechanisms
for purging known-dead repositories from the location tracking database. And the ability
to look up the UUID of the current repo, and to use the UUID to specify repositories when
applicable (untrust and describe maybe).


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2011-09-30T06:55:34Z"
 content="""
Specifying the UUID was supposed to work, I think I broke it a while ago. Fixed now in git.

I'm not sure why you need to look up the UUID of the current repository. You can always refer to the current repository as \".\". Anyway, the UUID of the current repository is in `.git/config`, or use `git config annex.uuid`.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawncBlzaDI248OZGjKQMXrLVQIx4XrZrzFo"
 nickname="Perttu"
 subject="comment 2"
 date="2011-09-30T11:55:35Z"
 content="""
Thanks for the quick reply :)

I wanted to look up the UUID of the current repo so that I can find out which repo is alive from the collection of repos with the same name.
I could have looked for it in .git/config though, since it's pretty obvious. I just looked into the git-annex branch and didn't find it there.
Thanks for the tip about using \".\". By the way, could there be some kind of warning about using non-unique names for repos? That would make this
scenario less likely. Or maybe that is a bad idea given the decentralized nature of git.

By the way, do the trust settings propagate to other repos? If I mark some UUID as untrusted on one computer does it become globally untrusted?
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 3"
 date="2011-09-30T16:47:27Z"
 content="""
`git annex status` now includes a list of all known repositories.

Yes, trust setting propigate on git push/pull like any other git-annex information.
"""]]


code:
I'm trying to follow the steps of the "walkthrough" but I'm experiencing the following issue: when
I sync one repository and do "git annex get ." I don't get the files from the other repository.
Here is the transcript of the steps I followed - I've put them in a script (ga.sh) so I can replay
them and show them on the shell while executing.
Basically I have two repositories, "/tmp/a/annex" and "/tmp/b/annex", the second cloned from
the first. All the other steps are the same as in the walkthrough.
-----------------------------------
> bash -x ga.sh
+ cd /tmp
+ mkdir a
+ mkdir b
+ cd a
+ mkdir annex
+ cd annex
+ git init
Initialized empty Git repository in /tmp/a/annex/.git/
+ git annex init a
init a ok
(Recording state in git...)
+ cd /tmp/b
+ git clone /tmp/a/annex
Cloning into 'annex'...
done.
warning: remote HEAD refers to nonexistent ref, unable to checkout.

+ cd annex
+ git annex init b
init b ok
(Recording state in git...)
+ git remote add a /tmp/a/annex
+ cd /tmp/a/annex
+ git remote add b /tmp/b/annex
+ dd if=/dev/urandom of=first bs=1024 count=1
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 9.9167e-05 s, 10.3 MB/s
+ dd if=/dev/urandom of=second bs=1024 count=1
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.000241635 s, 4.2 MB/s
+ git annex add .
add first (checksum...) ok
add second (checksum...) ok
(Recording state in git...)
+ git commit -am added
[master (root-commit) 5078564] added
 2 files changed, 2 insertions(+)
 create mode 120000 first
 create mode 120000 second
+ git mv first e
+ mkdir x
+ git mv second x
+ git commit -m moved
fix x/second ok
(Recording state in git...)
[master 422492d] moved
 3 files changed, 1 insertion(+), 1 deletion(-)
 rename first => e (100%)
 delete mode 120000 second
 create mode 120000 x/second
+ cd /tmp/b/annex
+ git annex sync a
(merging origin/git-annex into git-annex...)
(Recording state in git...)
commit
ok
git-annex: no branch is checked out
+ git annex get .
-------------------

The last "git annex get ." does not retrieve the files in /tmp/a/annex ... why?
I guess the issue starts when cloning /tmp/a/annex where no commit was done.

Emanuele

PS: I'm using git v1.7.9.5 (ubuntu 12.04) and the latest git-annex static binary
downloaded a few minutes ago from the git-annex website.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 1"
 date="2013-11-02T19:32:50Z"
 content="""
I have fixed `git annex sync` to work even when run in a repository with no master branch.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn2MI-Ci-bS9W5NITngsBJiL8LqWYcJ458"
 nickname="Emanuele"
 subject="comment 2"
 date="2013-11-02T23:36:54Z"
 content="""
Great thanks! I'm glad my feedback helped improve git-annex!

<https://github.com/joeyh/git-annex/commit/bc18cde32589cf53080c399e81b5680e4b>af6cef

"""]]


code:
Is there a way to easily access, say, a FDE (Full Disk Encrypted) external drive using git-annex?

My use case is that I do not wish to encrypt each file individually - and find it more reliable to use a real filesystem encryption on this external hard drive. --[[anarcat]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 1"
 date="2013-12-29T17:53:39Z"
 content="""
Mount the drive and put a git repository or directory special remote on it.

(This question seems to easy.. am I missing something?)
"""]]


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="2001:1928:1:9::1"
 subject="comment 2"
 date="2013-12-30T17:44:33Z"
 content="""
I guess this is related to [[confused about external drives]] - I would have liked the assistant to propose using the full drive for storing stuff, as opposed to encrypting data with GPG, including prompting me for the crypto password.
"""]]


code:
So I got a shiny new rsync.net account, but it's not (yet) big enough for my bloated, kitchen-sink annex.  How do I make the best use of it?   Well, on the whole small files are more valuable to me than large.  Large files are usually replaceable, like ripped DVDs.  Small files are more likely to be family photos and such.  So I shut down the assistant, and I've been doing a series of these:

    git annex copy --to=rsyncnet --smallerthan 10M --not --in=rsyncnet # I called my remote "rsyncnet"

    git annex copy --to=rsyncnet --smallerthan 20M --not --in=rsyncnet

    git annex copy --to=rsyncnet --smallerthan 40M --not --in=rsyncnet

until the repo fills up.  With a few of these thrown in to check on my progress and what's ahead.

    git annex status . --in=rsyncnet  # how much have I already done?
    git annex status . --smallerthan 40M --not --in=rsyncnet # how much will this next command copy?

It'd be possible to write a script to do all this automagically but at least this time it's easier to do by hand.

Just sharing in case anybody's in a similar situation.

(You could let the assistant do this for you if you configured a preferred content expression designating small files for the rsyncnet remote, of course.)


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="comment 1"
 date="2013-07-26T16:16:04Z"
 content="""
How about

    git annex find . --format='${bytesize} ${file}\n' |sort -n|sed 's/[0-9]* //'|while read file; do git annex copy \"$file\" --to=your_remote; done
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 2"
 date="2013-07-26T16:44:35Z"
 content="""
Note that you don't need to shut down the assistant; you can run multiple git-annex commands simulantaneously safely.
You will probably want to set that repository to be in the manual group in the assistant though.

Another way to do it is to make rsync.net a small archive, and move or copy your important files into `archive` directories.

You can also set up multiple remotes using the same rsync.net account. So you could have a transfer remote that's used to ship files between clients, as well as your archive remote.
"""]]


code: Is there an easy way for a noob to get the man pages, after installing the prebuilt tarball?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.6.49"
 subject="comment 1"
 date="2012-11-27T20:27:37Z"
 content="""
Yes, go to http://git-annex.branchable.com/git-annex/ and http://git-annex.branchable.com/git-annex-shell/

"""]]


code:
I tried about 2 weeks ago using the assistant to create on my netbook and my home laptop 3 repositores, for music, pictures and documents.  Since I was also going to be away from home, I set up jabber pairing and box.com for a transfer repository, and then watched everything crash as I didn't know yet about jabber only really working for 1 repository at a time.

So my new idea is this: one repository at ~/annex, and changing documents, music and pictures to be symlinks to actual documents, music and pictures folders inside ~/annex, allowing me to have just 1 annex to manage.  Is this the best way to put everything together if I want to just jabber pairing (and just 1 jabber account)?


code:
Let's suppose that I've manually modified files in .git/annex/objects,
for example I ran an rsync or some other file synchronization software
to copy files. As a result, some objects have disappeared, others have
appeared. After that `git annex whereis .' displays stale information,
it doesn't take the manual modifications to accound. `git annex fsck'
seems to fix this, but it runs the rehashing of all new files, so it's
slow. Is there a fast alternative, which notices all the object file
changes, trusts them, and just updates .git/annex/index quickly?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.51"
 subject="comment 1"
 date="2013-09-12T17:11:41Z"
 content="""
I do not recommend manually messing with the contents of .git/annex/objects. It is a good way to lose data.

If you look at the documentation for fsck in git-annex's man page, you will find the answer to your question.
"""]]


code:
[[!comment format=mdwn
 username="rasmus"
 ip="109.201.154.177"
 subject="But how to clean this folder?"
 date="2014-03-20T21:55:26Z"
 content="""
Joey,

I have sometimes experienced that there is way too much content in the `.git/annex/objects` folder.  For instance, my terminal emulator recently crashed during a sync which seemed to create a lot of folders.

In the annex repo where this happened I now got way too many folders (annex sometimes complain about no. of inodes or something like that) and I don't know how to clean it correctly.

Here's a \"screenshot\":

    [doc.annex] $ ls -a
    .  ..  documents  .git  .gitignore
    [doc.annex] $ find . -type d | wc -l
    38568
    [doc.annex] $ git annex fsck  > /dev/null 2>&1
    [doc.annex] $ find . -type d | wc -l
    38568
    [doc.annex] $ git annex repair  > /dev/null 2>&1
    [doc.annex] $ find . -type d | wc -l
    38568
    [doc.annex] $ find documents -type d | wc -l
    1513
    [doc.annex] $ find .git/annex/objects -type d | wc -l
    36712

This is a `direct`-mode repo.  With `.git/objectcts` I can use `git gc`.  How can I tell annex to tidy up?  I have tried `fsck`, `repair` and `forget`.

Any hints on how to deal with this?  Other than manually clean up by deleting `.git/annex/objects` and rerunning `fsck`?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.41"
 subject="comment 3"
 date="2014-03-26T19:09:07Z"
 content="""
1. This followup question seems to have nothing to do with the original question. (The aswer to the original question was `git annex fsck --fast`.)
2. I find it hard to believe that a crashing terminal emulator will create a lot of folders.
3. You show 36k subdirectories in .git/annex/objects. If I make a *tiny* 100 mb ext2 filesystem, it has 26k inodes. A 200 mb filesystem has 50k. So we're talking about a number of directories that is only large when using hardware from 1996.

I'm not convinced yet that anything is wrong, or that there's anything you can do to improve matters. In any case, see [[todo/wishlist:_pack_metadata_in_direct_mode]].
"""]]


code:
Thanks Joey for the great work.

I'm using git annex for my tv-shows and movies Folder.

I have 3 USB HDD (ext.150Gb,ext.200Gb, ext.2Tb) and a USB Stick (ATV), which are traveling between 3 Devices

2 Notebooks (Lappi,Kiste) and a Nas.

First of all, I'm made a mistake and mixed the remote Locations from the tvshows folder with the movies folder and did a git annex sync, ( this happened about two weeks ago)
I think i can't undo this, only unannex will help.

I've now watched the annex status output and noticed that there are many Remotes, some of them have a timestamp as name.

see log file at http://pastebin.com/79bRVkK6

Running git annex on ubuntu 12.04

    christian@Lappi:~/Serien$ git annex version
    git-annex version: 4.20130417
    local repository version: 3
    default repository version: 3
    supported repository versions: 3 4
    upgrade supported from repository versions: 0 1 2
    build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP DNS


code:
I have a large git repository with binary files scattered over different branches. I want to switch to git-annex mainly for performance reasons, but I don't want to loose my history.

I tried to rewrite the (cloned) repository with git-filter-branch but failed miserably for several reasons:

* --tree-filter performs its operations in a temporary directory (.git-rewrite/t/) so the symlinks point to the wrong destination (../../.git/annex/).
* annex log files are stored in .git-annex/ instead of .git-rewrite/t/.git-annex/ so the filter operation misses them

Any suggestions how to proceed?

EDIT 3/2/2010
I finally got it working for my purposes. Hardest part was preserving the branches while injecting the new `git annex setup` base commit.

#### Clone repository
    git clone original migrate
    cd migrate
    git checkout mybranch
    git checkout master
    git remote rm origin

#### Inject `git annex setup` base commit and repair branches
    git symbolic-ref HEAD refs/heads/newroot
    git rm --cached *
    git clean -f -d
    git annex init master
    echo \*.rpm annex.backend=SHA1 >> .gitattributes
    git commit -m "store rpms in git annex" .gitattributes
    git cherry-pick $(git rev-list --reverse master | head -1)
    git rebase --onto newroot newroot master
    git rebase --onto master mybranch~1 mybranch
    git branch -d newroot

#### Migrate repository
    mkdir .temp
    cp .git-annex/* .temp/
    MYWORKDIR=$(pwd) git filter-branch \
     --tag-name-filter cat \
     --tree-filter '
        mkdir -p .git-annex;
        cp ${MYWORKDIR}/.temp/* .git-annex/;
        for rpm in $(git ls-files | grep "\.rpm$"); do
            echo;
            git annex add $rpm;
            annexdest=$(readlink $rpm);
            if [ -e .git-annex/$(basename $annexdest).log ]; then
                echo "FOUND $(basename $annexdest).log";
            else
                echo "COPY $(basename $annexdest).log";
                cp ${MYWORKDIR}/.git-annex/$(basename $annexdest).log .git-annex/;
                cp ${MYWORKDIR}/.git-annex/$(basename $annexdest).log ${MYWORKDIR}/.temp/;
            fi;
            ln -sf ${annexdest#../../} $rpm;
        done;
        git reset HEAD .git-rewrite;
        :
        ' -- $(git branch | cut -c 3-)
    rm -rf .temp
    git reset --hard

TODO:

* Find a way to repair branches automatically (detect branch points and run appropriate `git rebase` commands)

I'll be happy to try any suggestions to improve this migration script.

P.S. Is there a way to edit comments?


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2011-02-25T05:16:48Z"
 content="""
I don't know how to approach this yet, but I support the idea -- it would be great if there was a tool that could punch files out of git history and put them in the annex. (Of course with typical git history rewriting caveats.)

Sounds like it might be enough to add a switch to git-annex that overrides where it considers the top of the git repository to be?
"""]]


code:
[[!comment format=mdwn
 username="tyger"
 ip="80.66.20.180"
 subject="comment 2"
 date="2011-03-01T14:07:50Z"
 content="""
My current workflow looks like this (I'm still experimenting):

### Create backup clone for migration

    git clone original migrate
    cd migrate
    for branch in $(git branch -a | grep remotes/origin | grep -v HEAD); do git checkout --track $branch; done

### Inject git annex initialization at repository base

    git symbolic-ref HEAD refs/heads/newroot
    git rm --cached *.rpm
    git clean -f -d
    git annex init master
    git cherry-pick $(git rev-list --reverse master | head -1)
    git rebase --onto newroot newroot master
    git rebase master mybranch # how to automate this for all branches?
    git branch -d newroot

### Start migration with tree filter

    echo \*.rpm annex.backend=SHA1 > .git/info/attributes
    MYWORKDIR=$(pwd) git filter-branch --tree-filter ' \
        if [ ! -d .git-annex ]; then \
            mkdir .git-annex; \
            cp ${MYWORKDIR}/.git-annex/uuid.log .git-annex/; \
            cp ${MYWORKDIR}/.gitattributes .; \
        fi
        for rpm in $(git ls-files | grep \"\.rpm$\"); do \
            echo; \
            git annex add $rpm; \
            annexdest=$(readlink $rpm); \
            if [ -e .git-annex/$(basename $annexdest).log ]; then \
                echo \"FOUND $(basename $annexdest).log\"; \
            else \
                echo \"COPY $(basename $annexdest).log\"; \
                cp ${MYWORKDIR}/.git-annex/$(basename $annexdest).log .git-annex/; \
            fi; \
            ln -sf ${annexdest#../../} $rpm; \
        done; \
        git reset HEAD .git-rewrite; \
        : \
        ' -- $(git branch | cut -c 3-)
    rm -rf .temp
    git reset --hard

There are still some drawbacks:

* git history shows that git annex log files are modified with each checkin
* branches have to be rebased manually before starting migration

"""]]


code:
[[!comment format=mdwn
 username="tyger"
 ip="80.66.20.180"
 subject="comment 3"
 date="2011-03-02T08:15:37Z"
 content="""
> Sounds like it might be enough to add a switch to git-annex that overrides where it considers the top of the git repository to be?

It should sufficient to honor GIT_DIR/GIT_WORK_TREE/GIT_INDEX_FILE environment variables. git filter-branch sets GIT_WORK_TREE to ., but this can be mitigated by starting the filter script with 'GIT_WORK_TREE=$(pwd $GIT_WORK_TREE)'. E.g. GIT_DIR=/home/tyger/repo/.git, GIT_WORK_TREE=/home/tyger/repo/.git-rewrite/t, then git annex should be able to compute the correct relative path or maybe use absolute pathes in symlinks.

Another problem I observed is that git annex add automatically commits the symlink; this behaviour doesn't work well with filter-tree. git annex commits the wrong path (.git-rewrite/t/LINK instead of LINK). Also filter-tree doesn't expect that the filter script commmits anything; new files in the temporary work tree will be committed by filter-tree on each iteration of the filter script (missing files will be removed).
"""]]


code:
[[!comment format=sh
 username="https://www.google.com/accounts/o8/id?id=AItOawll4Kgp7nMuOKdB0FfbcYZ3KRq7HCS0Slc"
 nickname="Laura"
 subject="Rebase all branches"
 date="2014-01-16T17:47:45Z"
 content="""

For the portion: git rebase master mybranch # how to automate this for all branches?

Try this:

branch_to_ignore='git-annex|master|newroot'
for branch in $(git for-each-ref --sort=-committerdate refs/heads --format='%(refname:short)' | egrep -v $branch_to_ignore )
    do git rebase --onto master \"$branch~\" \"$branch\"
    echo \"Rebasing branch $branch onto master....\"
done

Feel free to add/correct as necessary

"""]]


code:
When migrating large file repositories to git-annex that are backuped in a way that uses an rsync-style mechanism (e.g. [dirvish](http://www.dirvish.org/)) and thus keeps incremental backups small by using hardlinks, space can be saved by manually reflecting the migration on the backup. So, instead of making a last pre-git-annex backup, migrating, and duplicating all backupped data with the next backup, I used the <del>attached</del> migrate.py file below, and it saved me roughly a day of backuping.

A note on terminology: "migrating" here means migrating from not using git-annex at all to using it, not to the ``git annex migrate`` command, for which a similar but different solution may be created.

**WARNING**: This is a quickly hacked-together script. It worked for me, but is untested apart from that. It's just a dozen lines of code, so have a look at it and make sure you understand what it does, and what migrate.sh looks like. Take special care as this tampers with your backups, and if something goes wrong, well...

First, have an up-to-date backup; then, git annex init / add etc as described in the [[walkthrough]]. In the directory in which you use git-annex, run:

    $ python migrate.py > migrate.sh

Then copy the resulting migrate.sh to the equivalent location inside your backups and run it there. It will move all files that are now symlinked on the master to their new positions according to the symlinks (inside .git/annex/objects), but not create the symlinks (you will do a backup later anyway).

After that, do a backup as usual. As rsync sees the moved files at their new locations, it will accept them and not duplicate the data.

**migrate.py**:

    #!/usr/bin/env python

    import os
    from pipes import quote

    print "#!/bin/sh"
    print "set -e"
    print ""

    for (dirpath, dirnames, filenames) in os.walk("."):
    	for f in filenames:
    		fn = os.path.join(dirpath, f)
    		if os.path.islink(fn):
    			link = os.path.normpath(os.path.join(dirpath, os.readlink(fn)))
    			assert link.startswith(".git/annex/objects/")
    			print "mkdir -p %s"%quote(os.path.dirname(link))
    			print "mv %s %s"%(quote(fn), quote(link))


code:
My directory .git/annex/misctmp is quite filled up with files like P39923, P33083, and also with 7 characters P310000 up to P331998. The beginning of these filenames may come from the annexed files, which all start with 'P3'.

There is a total of 12198 of them (4.0TB... yes this repo is quite big ;)).

Each of those has the content of individual annexed files (about 300-400MB targzipped files). Sometimes they are hard liked to each other, up to 8 copies.

Since I have copied+dropped the whole repo content to other repos, there should be nothing left locally. I have verified several of those files (the annexed files which are identical to the misctmp/* files), are they indeed are located elsewhere, and not "here".

So I was wondering if it is safe to remove them, and why are they not listed by git annex unused?

Thanks for your help!!


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.114"
 subject="comment 1"
 date="2014-04-27T00:16:49Z"
 content="""
You can delete them.

AFAIK this should only happen if the assistant is interrupted while it's adding files. I plan to make the assistant clean up old tmp files on startup.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmdbVIGiDH8KarAGAy8y2FHJD_F990JzXI"
 nickname="François"
 subject="comment 2"
 date="2014-04-27T05:21:48Z"
 content="""
Indeed I had to interrupt the add a few times.

Thanks!
"""]]


code:
A couple times working in an annex manually, I accidentally did "git add" instead of "git annex add" and ended up with files checked into my repo instead of symlinks.  So now there is some file content in git, rather than in the annex.  Is there any way to root that out, or is it there forever?  (My limited knowledge of git says: it's there forever, unless maybe you go into the branches where it lives, do an interactive rebase to the time before it was added, remove that commit, replay history, and then garbage collect, but I have no idea if that would suddenly break git annex, and it sounds painful anyway.)

If say I were a neat freak and wanted to just start over with a clean annex, I imagine the thing to do would be to just start the hell over -- and if I wanted to do that, the way to go would be to get into a full repository, do a "git annex uninit" and then throw away the .git directory, then do "git init" and "git annex init" and then "git annex add ." ?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.6.49"
 subject="comment 1"
 date="2012-12-02T05:41:56Z"
 content="""
You can rebase or use git-filter-branch to rewrite history. As long as you don't touch the git-annex branch, git-annex won't care, at all. Have at it!

Your uninit plan would also work.
"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 2"
 date="2012-12-03T15:46:48Z"
 content="""
Thanks!  I ended up going with the uninit plan.
"""]]


code:
hi

is it possible to have multiple repositories that share single backup directory ?

for example.
i have mp3, docs on my laptop as separate repositories. i would like to use single backup directory that is on my usb drive.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 1"
 date="2013-12-29T20:29:26Z"
 content="""
If you set up an identically configured special remote in the same location for each of your separate mp3 and docs repositories, then yes, you can store files for both of them in it. In the unlikely case where you have the same file in both mp3 and docs, only one copy will be stored in the remote. (Unless it's using encryption, which will tend to defeat that.)

It should work ok, but unless you're trying to get deduplication like that, I don't see any reason to do it rather than having a separate special remote for each of the separate git repositories.

(Also, I'm talking about special remotes here, not git remotes. If you add the same git remote to 2 separate repositories, then `git annex sync` will connect them all together.)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkNE-H4vEcbcGndxq5daT8qUb7yIf7r1OE"
 nickname="Łukasz"
 subject="comment 2"
 date="2013-12-29T21:34:19Z"
 content="""
thx for reply. i think i will stick with separate directories for every repositiry

mp3 on laptop -> mp3-backup directory on usb drive
docs on laptop -> docs-backup on usb drive
"""]]


code:
[[!comment format=mdwn
 username="http://cstork.org/"
 nickname="Chris Stork"
 subject="Using different git branches"
 date="2014-01-14T09:12:19Z"
 content="""
This might not be practical (especially for a non-technical user) but wouldn't it be possible to share a (bare) git-annex repo for backup if the other non-bare repos use different branches of the same git repo? (These non-bare repos could also share their .git directory.) Hmm, I don't see this discussed much...
"""]]


code:
I am trying to configure the local repo to be connected with remote computer. It can either find this computer on local network using local host name, or go through public IP host where I have ssh tunnel. First confusing thing is, that when setting up remote server, it successfuly establishes connection, but then asks me if the folder should be git or rsync - I believe this step should be skipped if the git repo is already there. Also choosing repository group should be detected from the repo itself.
But the real problem is that I cannot add the public IP repo at all. Is this a known limitation?


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmZ9EwisYI1DDev8HXY6zFjPrLKt60-3QI"
 nickname="Yo"
 subject="comment 1"
 date="2013-06-24T12:23:15Z"
 content="""
I believe I have a similar set up as you're trying to achieve. What I did was just add a repository using the public domain name only (DDNS). My home router seems to be smart enough to realize the IP the domain name resolves to is its own, so I get gigabit speeds when at home, and I can still access it when I'm not at home.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.193"
 subject="comment 2"
 date="2013-06-25T20:05:46Z"
 content="""
I don't see anything that prevents you from adding two repositories that use different names for the same server.
"""]]


code:
I saw a cool page talking about "tagging" photos by using symlinks: http://www.trueelena.org/computers/articles/photo_management_with_git-annex_and_bash.html

so say I have a photo in git annex, called DSC_3285.JPG, which of course is really a symlink to ../.git/annex/objects/Zk/kj/WORM-s5296770-m1338516288--DSC_3285.JPG/WORM-s5296770-m1338516288--DSC_3285.JPG.

I want to make an additional link to that photo in a directory called tags/.

should I link to the symlink (DSC_3285.JPG), or to the annexed file? (../.git/annex/objects/Zk/kj/WORM-s5296770-m1338516288--DSC_3285.JPG/WORM-s5296770-m1338516288--DSC_3285.JPG)

I might occasionally rename DSC_3285.JPG or edit the photo itself. will the git annex commit hooks update both links, or should I prepare a script to update links in tags/ after I change DSC_3285 or the annexed data?

thank you.