code:
I use git annex in direct mode and I have renamed a file. What should I do to make git annex understand that the file has just been renamed and not deleted/added?

The file I renamed is quite big and I would like to avoid having to copy it again to the remotes and I would also like to avoid having to redownload it again in all the repositories.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.254.222"
 subject="comment 1"
 date="2013-07-07T17:14:18Z"
 content="""
You don't have to do anything special, git-annex automatically detects when a file has been renamed.

Just `git annex` add the new file, and commit and sync like usual.
"""]]

repo_name: kakafarm/git

license: unknown


code:
I've got a remote I setup while connected to my home network that is no longer syncing when I'm on the same network. I'm not sure how to debug the problem. Any tips?

I will say that when I edit the repository in the assistant, the url it gives for the "git repository location at" field looks incorrect. It's got an extra "git-annex-" prepended to the hostname. I'm not sure how to alter that.

I'm using version 5.20140517~bpo70+1, from the wheezy backports.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.176"
 subject="comment 1"
 date="2014-05-29T18:15:12Z"
 content="""
That extra git-annex- prefix is used in ~/.ssh/config to set up the ssh keys for the repository. So it's normal.

I suggest that you try running git-annex sync at the command line and paste the output so we can see how it fails.
"""]]


code:
[[!comment format=mdwn
 username="go8ose"
 ip="59.167.191.61"
 subject="comment 2"
 date="2014-05-30T10:28:38Z"
 content="""
I was shown a password prompt for the remote. Then I realised I was controlling my ~/.ssh/authorized_keys file with puppet, which would have removed the public key that annex would have added to that file. I'm sure that once I fix that up, I'll get it working again.

Thanks for the CLI suggestion.
"""]]


code:
I'm using git-annex to manage my files. Some of my remotes are available using 2 or 3 methods, for example ssh and nfs, but the nfs access is only possible on my local network, of course. Then, git-annex chooses the fastest method to sync or get the files (nfs, if available, ssh instead, using scores to choose). For now, I define a remote per repository and per access method (example: server-by-ssh, server-by-nfs), but that's not a really git-annex way to do the job, as it is not compliant with the numcopies option, which ckecks that the files in the annex are kept with enough replications (with my current method, many remotes are in fact the same folders on a hard drive).

Then, my question applies to both git and git-annex, since the remotes are added in git, not git-annex: I wonder if it is possible to define alternative url for a given remote.

I have tried to use the git remote set --url --add command, but it doesn't really work as I would expect. Indeed, if the nfs url is not reachable, git hangs up and waits.

Any idea?


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
 nickname="Justin"
 subject="comment 1"
 date="2012-08-21T12:42:16Z"
 content="""
I don't think having multiple remotes for the same repository confuses git-annex since it uses the UUID not the url to identify them.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.8.29"
 subject="Justin is right"
 date="2012-08-21T15:57:46Z"
 content="""
git-annex doesn't care how many git remotes you have pointing to a given repo, that repo's uuid still only counts as one copy.
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/speredenn#aaf38"
 nickname="Jean-Baptiste Carré"
 subject="comment 3"
 date="2012-08-21T18:15:48Z"
 content="""
You're totally right: The UUIDs are the same. So it shouldn't matter if there are many repositories pointing to the same folder, as you state it. Thanks a lot!
"""]]


code: I have a repository with an S3 remote named `cloud`. I now no longer want to use that remote. I've dropped all the data from it (`git annex drop . --from cloud`). How do I tell my repository to forget about the remote?


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnldTTAP8PAifJUmqhRar6RAWNWlRcencw"
 nickname="Marco"
 subject="You can set the trust to "dead""
 date="2013-01-23T19:02:22Z"
 content="""
Simple answer: You can't fully delete a remote - with annex commands. You can mark it as dead.

You can give this a try as a workaround: [[Truly_purging_dead_repositories/#comment-51ab797094f4c07bf5327fd21cad5835]]
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4830:1600:187::2"
 subject="comment 2"
 date="2013-09-04T06:43:26Z"
 content="""
Recently git-annex has gotten the ability to do this: `git annex forget --drop-dead`

That prunes all history relating to all dead remotes. You need to be running a git-annex that supports this on all computers you use the repos on, or the pruned history will get merged back in.

I don't recommend doing this just because you want to \"clean history\". Think of it as something you can do at some point in the future if the .git/objects somehow gets too large or too slow. Put off deleting data until tomorrow if you don't absolutely need to do it today.
"""]]


code:
[[!comment format=mdwn
 username="konubinix"
 ip="82.243.233.186"
 subject="Dropping dead repositories"
 date="2013-09-04T07:40:22Z"
 content="""
Actually, it may be a good idea to remove repositories made for tests purposes.

I now have 2 dead repositories that are USB_test1 and USB_test2 that I created before knowing I could reuse the annex uuid.

They are now there and it is difficult to remove them.

For that special case, the --drop-dead feature is very welcome.
"""]]


code:
Git annex is really amazing software, and this cute little scenario is actually Linux's fault. But it's a nasty situation nonetheless, and worth passing on.

...

Create a client repository on your laptop and two backup repositories on external USB drives. Keep all repositories mounted and connected. Now drop 60GB of data spread over 30,000 files into your annex, and watch git annex assistant start adding and syncing. So far, so good.

Now wait a few hours, and watch some kernel crypto code—probably ecryptfs—fall over with a segfault.

Since your laptop and USB drives are all running ext4, the sudden kernel panic will leave you with hundreds of 0-length files. Because git annex assistant was busily adding and syncing files, those 0-length files are spread randomly throughout all your git repositories (typically in `.git/objects`) and throughout all the associated annexes. Unfortunately, because `git annex assistant` generates _tons_ of commits, this is pretty much unrecoverable using standard git tools unless you're willing to get deep into the repositories' internals.

So what should you do, if you want to add 10s of 1000s of files and there's some risk of kernel panic or accidentally bumping a USB cable? Here's my recommendation to limit the damage:

1. Use the command line if possible.
2. Add all your files with your remotes offline.
3. Run `git gc` on your central repository, just on general principals.
4. Mount one repository at a time.
5. Sync the pure git data first, and then make sure that all disk I/O is flushed (`sync; sleep 10` is a good approximation).
6. Use `git annex copy --to` to move the annex data.
7. Unmount the USB repository cleanly and move onto the next one.

If you _do_ bump a USB cable in the middle of step (6), then:

1. Run 'git annex fsck' to clean up any garbage files.
2. Try another 'git annex copy --to' where you left off.

Wiser minds than I are encouraged to suggest optimizations for the recovery steps.

The theory behind these steps is to only do one thing at a time, and to expose as few remotes as possible to a power failure or crash. A secondary goal is to make sure that pure git operations complete very quickly, limiting the risk that they will be interrupted, because they're the hardest operations to recover from after a crash.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 1"
 date="2013-07-23T14:45:05Z"
 content="""
It seems to me that it would be better to avoid using kernel features that crash. And, set up a git repository on another machine, which will serve as a real backup if something goes catostrophically wrong. The only potential failure mode then would be if a kernel/hardware issue zeroed some of your files, in direct mode the assistant might commit those new corrupt files and ship them over to the backup. But, if your offsite backup is accumulating old versions of files, you can recover from this by reverting the bad commit.
"""]]


code:
[[!comment format=mdwn
 username="http://emk.myopenid.com/"
 ip="24.2.150.84"
 subject="Well, this is the first time it ever panicked"
 date="2013-07-23T16:32:10Z"
 content="""
Well, it's not like my machine kernel panics on a regular basis or anything. :-) This is the first time I ever saw the kernel encryption code do this. I'm running a boring stock install of Ubuntu 12.04 LTS that was preloaded by ZaReason, and I'm using the ecryptfs home directory encryption supplied by Ubuntu. So in this case, \"stop using kernel features that crash\" means \"stop using Ubuntu on supported hardware.\"

The underlying problem is that ext4 allocates file contents lazily and out-of-order, and it may wait a surprisingly long time before actually flushing data to disk:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/317781
http://linux.bihlman.com/2010/learn-linux-help/how-to-solve-zero-length-file-problem-in-linuxs-ext4-file-system/

The problem is that `git annex assistant` and `git` don't sync the disks all that often, and that the assistant can generate huge amounts of complicated disk I/O across multiple volumes for hours on end. All you need to do is go through the walkthrough, create the specified repositories including one on a USB drive, and throw 50GB of data into the annex directory. `git annex assistant` will happily grind away overnight, and it anything prevents ext4 from flushing data, there's a good chance you'll wind up with multiple corrupted repositories with hundreds of `git fsck` errors.

There are some potential workarounds:

1. Reconfigure ext4 to run in data write-back mode. This isn't really possible for non-technical users, but it's an excellent idea for large external USB drives.
2. Somehow convince `git` and `git annex assistant` to call `sync` or `fsync` more aggressively on local volumes.
3. Use a remote server for at least one remote, as you suggested.
4. Operate on the repository manually, so that you can ensure that `git`'s data is in a known-good state before trying to copy the annex files. The
annex files are much easier to recover than git's state.

Anyway, I doubt this is really fixable. And it's not really `git annex`'s fault, in any case. But I'm really glad I had recent backups of all my data last night, which allowed me to checksum everything and start from scratch.

...

Leaving aside this incident, `git annex` is one of the nicest pieces of open source software I've seen in a long time, and it's clearly going to change how I use my computer. And thank you for posting the crowd-funding campaign so we can say \"Thanks!\"
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 3"
 date="2013-07-23T17:17:43Z"
 content="""
`git config core.fsyncobjectfiles true`

This will make git fsync all the data it writes. Whether it's a good default, I don't know.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 4"
 date="2013-07-23T17:38:50Z"
 content="""
I have made the assistant enable the fsync option when it creates a repository on a removable drive. That is, at least in my experience, the thing that most likely needs it. I don't know if it would have saved you in your situation or not.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8"
 nickname="develop"
 subject="comment 5"
 date="2013-07-23T17:39:42Z"
 content="""
Oh i've had that type of crash from ecryptfs many times over many different versions of Ubuntu, also before i even encountered git-annex.

Git-annex might provoke this issue, but it is by no means the only thing that can crash your system when running ecryptfs.

I highly suggest you try LUKS for a stable experience.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 6"
 date="2013-07-23T17:54:00Z"
 content="""
If there's an easy way to detect when a directory is inside an ecryptfs volume, I'd be happy to make the assistant automatically enable fsync on it. ;)
"""]]


code:
[[!comment format=mdwn
 username="http://emk.myopenid.com/"
 ip="24.2.150.84"
 subject="Thank you!"
 date="2013-07-23T18:36:22Z"
 content="""
What good idea! I've turned on `core.fsyncobjectfiles`. This should definitely help avoid the worst damage when ext4 filesystems get unmounted while dirty. Thank you very much for addressing this issue so quickly, even if though has more to do with ext4 than git annex.

develop: Thank you for the suggestion to use LUKS; that's good to know.
"""]]


code:
[[!comment format=mdwn
 username="http://emk.myopenid.com/"
 ip="24.2.150.84"
 subject="Detecting ecryptfs"
 date="2013-07-23T18:49:08Z"
 content="""
If the string `\" ecryptfs \"` (with surrounding spaces) appears in `/etc/mtab`, then there's at least one ecryptfs volume on the system. More specific parsing will tell you where it's mounted.

Ubuntu released a new kernels fixing the worst ecryptfs bugs in 12.04 LTS. The current version is OK, but perhaps not exactly great. Some older versions are public menaces with awful data corruption bugs.
"""]]


code:
My annex contains several large files that I have unlocked, edited, and committed again, i.e. the annex contains the version history of those files. However, I don't want the history -- keeping the latest version is good enough for me. Running `git annex unused` won't detect those old versions, though, because they aren't unused as old Git revisions still refer to them. So I wonder:

1. What is the best way to get rid of the old versions of files in the annex?

2. What is the best way to detect old versions of files in the annex?

I guess, I could run `git rebase -i` to squash commits to those files into one commit, thereby getting rid of the references to the old copies, but that approach feels awkward and error prone. Is anyone aware of a better way?


code:
[[!comment format=mdwn
 username="http://peter-simons.myopenid.com/"
 ip="77.186.179.173"
 subject="comment 1"
 date="2012-02-09T18:53:00Z"
 content="""
Sorry for commmenting on my own question ... I think I just figured out that `git annex unused` *does* in fact do what I want. When I tried it, it just didn't show the obsolete versions of the files I edited because I hadn't yet synchronized all repositories, so that was why the obsolete versions were still considered used.
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 2"
 date="2012-02-09T19:42:28Z"
 content="""
Yes, contents are still considered used while tags or refs refer to them. Including remote tracking branches like `remotes/origin/master`
"""]]


code:
I've done a `git annex add` and `git commit` on my annex which included some files I don't want to add to the annex. I've tried to reverse it all out, but whenever I `git annex add` something, the unwanted files show up in the git-annex branch.

    git init forgetmenot
    cd forgetmenot
    git commit -m "create" --allow-empty
    git annex init fmn

    echo 'foo' > foo
    echo 'bar' > bar

    git annex add
    git commit -m "add foo"

    git log --oneline --name-only
    # 28634c0 add foo
    # bar
    # foo
    # 4a87050 create

    git log --oneline --name-only git-annex
    74e6969 update
    41d/a26/SHA256E-s4--b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c.log
    a70/4a5/SHA256E-s4--7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730.log
    95285ed update
    uuid.log
    2135e07 branch created

If you now try to get git-annex to forget by reverting *master* and *git-annex* and only adding/commiting *foo*, the master branch ends up correct, but git-annex magically remembers *bar*!

    git reset --hard HEAD^
    git branch -f git-annex git-annex^

    echo 'foo' > foo
    git annex add
    git commit -m "add foo"

    git log --oneline --name-only
    # 1b4889e add foo
    # foo
    # 4a87050 create

    git log --oneline --name-only git-annex
    # 3d0b9bc update
    # 41d/a26/SHA256E-s4--b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c.log
    # a70/4a5/SHA256E-s4--7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730.log
    # 2e17a19 update
    # uuid.log
    # 646776b branch created

How is git-annex remembering this and how can I get it to completely forget?

I have tried `git gc --aggressive --prune=all`, `git annex fsck --all` and `git annex drop unused` but somehow, git-annex is remembering bar existed.

This is an exercise in micro-managing the git-annex branch a bit, but this situation does also cause git-annex to complain about the missing files on fsck (0 out of 2 copies) so it isn't just being a control freak! Honest! :)


code:
[[!comment format=mdwn
 username="CandyAngel"
 ip="81.111.193.130"
 subject="comment 1"
 date="2014-07-17T15:43:03Z"
 content="""
Joey has pointed me to the solution.

git-annex was remembering these files due to them being present in *.git/annex/index*.

A simple `rm .git/annex/index` after moving the git-annex branch to the earlier commit prevents the \"ghost\" log files from being recreated and Joey confirmed this is safe to do (git-annex automatically recreates it).
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 2"
 date="2014-07-17T16:05:33Z"
 content="""
Of course, if the repository has any git remotes, there will also be remote-tracking branches for the git-annex branch that get automatically merged back in, etc.
"""]]


code:
Hi!

When I update one repository to/from another, am I to push/pull only the master branch (and other branches I'd have created myself) or also the git-annex branch?

From what I understand, the git-annex branch is local to one repository and has no interest in being merged in another, but some examples make use of git pull --all (which merges distant git-annex branch into local).


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2012-04-10T16:05:40Z"
 content="""
The git-annex branch is how the annex information for the different repositories is communicated around, so yes, you need to push/pull it.
"""]]


code:
I have git-annex set up with three clients and a transfer repository and everything was working fine, but a couple of days ago I noticed that git-annex was starting to upload to the transfer repository files that where already present on the three clients, and it hasn't stopped since. It's uploading all my files and I don't really know why.

Is there a way to know why does git-annex think it needs to upload this files?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.41"
 subject="comment 1"
 date="2014-03-26T17:46:30Z"
 content="""
A few likely reasons:

* If a 4th client repository had popped up.
* If you have configured a high number of copies, it might only be able to be met by keeping files on the transfer repository.
* Similarly, if a repository that used to have the files has been marked as dead or deleted, more copies might be needed to make up for that.
* For completeness, if the transfer repository accidentially had its type changed to some other kind of repository, like a full backup.

You can enable debugging (start with --debug or go into the webapp's preferences) and it might say a little more, but the debugging info is not very good.

The best thing is probably to look at one single file, use `git annex whereis` on the file to see what repositories contain it, and then think about how that interacts with the [[preferred_content_expression_of_the_transfer_repository|preferred_content/standard_groups]].
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawk9nck8WX8-ADF3Fdh5vFo4Qrw1I_bJcR8"
 nickname="Jon Ander"
 subject="comment 2"
 date="2014-04-01T08:01:00Z"
 content="""
I thought I had already checked all does possibilities, the repositories where in the correct group and no new repositories had been added. *git annex info* didn't show anything weird and nor did *git annex whereis*. I finally found out *git annex vicfg* and I found two new repositories with no name in the repository groups. It looked something like this:

    # (for )
    group repository-hash = client
    # (for )
    group repository-hash = client

No idea of how they got there, but setting both of them to unwanted solved the issue.
"""]]


code:
Hi,

In normal maven release procedure, it re-clones the gitrepo from the tag to do the release from either git or local file, But as git-annex uses symlinks when the repo is clones by maven, it misses the files. Any recommendations how i can make this work ?

Also would love some data on who is using git-annex in production environment would be good.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.21"
 subject="comment 1"
 date="2013-07-30T18:20:53Z"
 content="""
If you can find a way to run `git annex get` in the cloned repo, it will copy over the files. Perhaps there is some way to make Maven do that via a hook script or similar?
"""]]


code: How can I make one of my servers keep a copy of the files on the server, so I can easily SFTP in and download the files. I don't want to see the .git folder, I want to see the files synced there.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 1"
 date="2014-07-02T17:18:00Z"
 content="""
Make a non-bare git repository on the server, and make it have a git post-receive hook that runs
\"git annex merge\"
"""]]


code:
I have a repository on my laptop with a single remote on a USB drive. Both were created without the assistant.

The remote holds all the content. My understanding is that this would be a "backup" repository, using the group definitions.

The local repository generally holds no content. When I want a file, I make sure the remote is available, and then I `git annex get` it.

This works fine in git annex, but I'm now experimenting with the assistant. As soon as I fire up the assistant, it starts downloading all of the data from the remote. I don't ever want it to automatically get any content from the remote. If it sees new content in the repository, I would like it to push that out to the remote. But never get anything from the remote, and never drop anything that is in the repository.

How can I setup this behaviour in the assistant? I have set the remote's repository group to "backup". The local repository is not in any group, since none of the groups fit my model for this repo.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn7gQ1zZDdWhXy9H51W2krZYShNmKL3qfM"
 nickname="Karsten"
 subject="comment 1"
 date="2013-01-20T08:11:37Z"
 content="""
Run 'git annex vicfg', locate the preferred content setting for your repository, uncomment it and set it to 'present'. Then it shouldn't move content automatically.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 2"
 date="2013-01-24T02:01:00Z"
 content="""
A recent commit allows you to just choose \"manual mode\" in the webapp to get the same effect.
"""]]


code: I recently created WebDAV remote to my annex which I assumed to be just for testing purposes at the time, so I gave it a strange and long-wided name. It turned out, though, that the remote is stable, works fine, and that I'm going to keep it. Now, I would like to give it a snappier and more descriptive name than the one I originally chose. Is that possible somehow?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.3.194"
 subject="comment 1"
 date="2013-01-18T18:20:43Z"
 content="""
You can rename a git remote with just \"git remote rename <old> <new>\". However, for a special remote, git-remote will fail to rename it unless you first add a dummy remote.<name>.fetch value in .git/config. You can remove it later.

So, for example:

git config remote.$uglyname.fetch dummy
git remote rename $uglyname $nicename
git config remote.$nicename.fetch \"\"

(The webapp handles doing this when you edit a remote's name, BTW.)

---

There's actually another place the name of a special remote is recorded, in `remote.log`. That name is only used when you use `git annex initremote` though. It actually is possible to change it, by using `initremote` to change the vale of the name field. Example:

git annex initremote $uglyname name=$newname

Note that for some types of special remotes, this will require you to re-specify some other configuration. For example, with a directory special remote, it wanted me to include a directory= parameter.
"""]]


code:
I have just done a sync in a direct mode repo:

    git-annex sync sshremote

It has quite a few conflicts (both sides adding the same file with same content). But somehow the merge won't conclude. Here's some log:

    $ git-annex sync sshremote
    commit  ok
    pull sshremote
    warning: no common commits
    remote: Counting objects: 32162, done.
    remote: Compressing objects: 100% (31192/31192), done.
    remote: Total 32162 (delta 3), reused 32162 (delta 3)
    Receiving objects: 100% (32162/32162), 3.39 MiB | 1.61 MiB/s, done.
    Resolving deltas: 100% (3/3), done.
    From sshremote:/repo
     * [new branch]      annex/direct/master -> sshremote/annex/direct/master
     * [new branch]      git-annex  -> sshremote/git-annex
     * [new branch]      master     -> sshremote/master
     * [new branch]      synced/master -> sshremote/synced/master

    Auto-merging file1
    CONFLICT (add/add): Merge conflict in file1
    Auto-merging file2
    CONFLICT (add/add): Merge conflict in file2
    Auto-merging file3
    CONFLICT (add/add): Merge conflict in file3
    Auto-merging file4
    CONFLICT (add/add): Merge conflict in file4
    Automatic merge failed; fix conflicts and then commit the result.
    file1: needs merge
    file2: needs merge
    file3: needs merge
    file4: needs merge
    ^C

I waited for a few minutes, and checked that both cpu and disk activity is near zero on both server. So I interrupted it, thinking it might be some locking issue.

But now I cannot sync with the remote anymore because the merge is still in process. How should I resolve the conflicts? I could do it manually if I cannot use annex's auto merge.

    $ git-annex sync sshremote
    commit  ok
    pull sshremote

    fatal: You have not concluded your merge (MERGE_HEAD exists).
    Please, commit your changes before you can merge.

    fatal: You have not concluded your merge (MERGE_HEAD exists).
    Please, commit your changes before you can merge.
    failed

    $ git merge --abort
    fatal: This operation must be run in a work tree

Update: this is running on OSX with:

    git-annex version: 5.20131221-g00d1673
    build flags: Assistant Webapp Pairing Testsuite S3 WebDAV FsEvents XMPP DNS Feeds Quvi TDFA CryptoHash
    key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
    remote types: git gcrypt S3 bup directory rsync web webdav glacier hook
    local repository version: 3
    default repository version: 3
    supported repository versions: 3 5
    upgrade supported from repository versions: 0 1 2 4


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 1"
 date="2014-01-01T20:31:21Z"
 content="""
I think you can fix at least the immediate problem by deleting the .git/MERGE_HEAD file.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkYmMFDdf3GJ9Oba6NCVkzGc4JyB9WavMs"
 nickname="Xinruo"
 subject="Thanks"
 date="2014-01-02T00:41:27Z"
 content="""
That did the trick!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 3"
 date="2014-01-02T02:05:05Z"
 content="""
AFAICS, git-merge was still running when you ctrl-c'd it. So this is coming before git-annex processes the merge to update the work tree.

I don't know why it would take so long to run, especially if it was not using CPU or disk. You might try to reproduce the problem, passing --debug to git-annex, and perhaps strace the git-merge process to see what it's doing.
"""]]


code: Somehow the symlinks have vanished from one directory of my repository. How can I restore them?


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnldTTAP8PAifJUmqhRar6RAWNWlRcencw"
 nickname="Marco"
 subject="git annex fix"
 date="2012-11-24T08:51:19Z"
 content="""
You can try
    git annex fix yourDirectory

Hope that helps.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.6.49"
 subject="comment 2"
 date="2012-11-25T18:35:56Z"
 content="""
The key thing to realize is that the symlinks used by git-annex are checked into git like any other file would be. So you can use the entire git toolkit to manage them.

For example, you could run `git status` to see if git shows them as recently deleted, and then use `git checkout $file` to restore the deleted files.

Or perhaps the deletion has been committed to git, and then you'd use `git log --stat` to find the commit that deleted your files, and `git revert` could be used to undo it.

(`git annex fix` is not related to this and won't help.)
"""]]


code:
[[!comment format=mdwn
 username="jbee"
 ip="178.25.68.205"
 subject="Thanks"
 date="2012-11-28T11:52:24Z"
 content="""
> deletion has been committed to git, and then you'd use git log --stat to find the commit that deleted your files, and git revert could be used to undo it.

That did the trick.

Thanks Joey, for the answer and the magnificence that is git-annex.
"""]]


code:
I worked out how to retroactively annex a large file that had been checked into a git repo some time ago.  I thought this might be useful for others, so I am posting it here.

> This is a great tip, so I've moved it to
> [[tips|tips/How_to_retroactively_annex_a_file_already_in_a_git_repo]].
> --[[Joey]]


code:
This seems like something that should be completely obvious, but I've been trying to get it working for a while without success.

I have a few machines (at least 3) at different locations. I want them all to be synced with a remote server that I have. I've installed git-annex on the server, and setting up the assistant on one of the machines, I can create the remote repository, and all seems to work (it spits out info about syncing... though I don't have any easy way of checking if it's working, as all I have is the one client).

It says in the webapp that setting up a remote git repository in transfer mode will make it easy to have other clients. But I'm wondering how to set that up. I tried just adding the same server with the same path on another machine, but it doesn't seem to be syncing, and I imagine that it is trying to create a fresh repo there instead of syncing with an existing one.

So, how do I set this up? I don't mind adding git remotes, ssh keys, etc, manually, but I haven't been able to figure out what I should be doing! My end goal is to have one annex that is synced between the many computers (ala Dropbox, as the assistant is supposed to be).

Note that the configuration of the test machines:
client 1 - debian, git-annex built from current cabal
server - debian, git-annex from apt
client2 - mac osx, git-annex built from current cabal

Thanks,
Daniel


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.27"
 subject="comment 1"
 date="2012-11-08T21:29:26Z"
 content="""
Just adding the same server with the same path is the right thing to do, it will use the existing repo.

Probably the problem you're having is that one client doesn't know when the other client has sent data to the server. I've recently been adding XMPP (Jabber) support to deal with that. That is not yet in a released version of git-annex, but it is available in git master.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlsfh8P7pN2OBr24ztT_7AyYQLCcVqqa-4"
 nickname="Daniel"
 subject="comment 2"
 date="2012-11-09T17:49:32Z"
 content="""
The master is segfaulting on one of my machines (the mac), but XMMP sounds good in general.

In the meantime - does git annex sync do what the assistant does? Or is there a command to tell the assistant to check for remote changes? (or a way to get it to auto-poll)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.252.11.120"
 subject="comment 3"
 date="2012-11-13T17:57:49Z"
 content="""
I recently dealt with a segfault caused by a bug in the haskell gnutls library. It could be that's what you were seeing. It's fixed in gnutls (>= 0.1.4)

Yes, you can run `git annex sync` by hand. It does the same syncing of the git repository that the assistant does. I don't think making the assistant poll is a good idea.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawniCRkhl_W87gOK5eElfsef3FoUsUFpAr4"
 nickname="Alexandre"
 subject="Simplifying this kind of setup"
 date="2012-12-10T14:33:08Z"
 content="""
Maybe it is possible to avoid the XMPP account setup and transferring via XMPP, maybe getting notifications through the SSH connection is possible.

I'm thinking about a \"git-annex-shell server\" unix socket to which clients would connect using the SSH connection and get update notifications from other clients.
"""]]


code:
I have one Annex meant to be kept in sync across two computers by the assistant. These computers can't connect directly, so I use a transfer repository (at rsync.net) and xmpp. Recently the two repositories have started to diverge: files added here do not appear there and vice versa, files modified here are not updated there and vice versa. It may be related to this bug: [[bugs/Jabber__47__xmpp_not_supported_on_Debian_Wheezy_backport/]] since one of the two installed versions of git-annex is 5.20140117~bpo70+1, but I do not know whether the divergence started the day I upgraded to this version.

I tried this:

    git-annex fsck --from rsync.net_annex

on both machines. No error is reported, but files are missing here and there and files differ here and there.

I stopped the assistant on both machines and tried to sync with:

    git-annex sync

and got:

    commit
    ok
    pull 'XXX'
    fatal: Unable to find remote helper for 'xmpp'
    failed
    push 'XXX'
        Pushing to XXX failed
        (non-fast-forward problems can be solved by setting receive.denyNonFastforwards to false in the remote's git config)
    failed
    git-annex: sync: 2 failed

where 'XXX' stands for my jabber account. I then tried:

    git-annex sync rsync.net_annex

but got:

    git-annex: cannot sync special remotes: rsync.net_annex

This situation is unfortunate because the user is not made aware of anything failing. The assistant is running, the webapp shows files being uploaded to the transfer repository, but the two repositories are actually diverging.

How can I synchronize these two repositories?

Thanks,


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnNqLKszWk9EoD4CDCqNXJRIklKFBCN1Ao"
 nickname="maurizio"
 subject="downgrading is not an easy option"
 date="2014-01-30T05:44:48Z"
 content="""
In order to check whether this is related to  [[https://git-annex.branchable.com/bugs/Jabber__47__xmpp_not_supported_on_Debian_Wheezy_backport/]] I tried to downgrade git-annex to the previous version that was available on wheezy-backports (git-annex_4.20131106~bpo70+1_i386.deb) but then I get:

    git-annex: Repository version 5 is not supported. Upgrade git-annex.
    failed

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnNqLKszWk9EoD4CDCqNXJRIklKFBCN1Ao"
 nickname="maurizio"
 subject="clean up"
 date="2014-01-31T19:55:30Z"
 content="""
ok, now that git-annex has been updated to 5.20140117~bpo70+2 things start flowing again, so it was actually caused by [[http://git-annex.branchable.com/bugs/Jabber__47__xmpp_not_supported_on_Debian_Wheezy_backport/]]. Now there is a certain number of files called 'filename.variant.06b8' for instance. That is a bit messy.

I do not know wheter/how it would be possible to inform the user that something is going wrong, but certainely it would be better to avoid failing completely silently.

It would be useful if the webapp could give more precise information about the files the assistant is manipulating. For instance: \"now uploading file 'bla', updating version last uploaded from 'remote' on 'date'\".
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="71.80.94.56"
 subject="comment 3"
 date="2014-02-07T19:07:11Z"
 content="""
Normally the webapp will tell you whenever it fails to sync with a remote. However, in this case, it had XMPP support compiled right out of it, so had no idea that remote even existed, and never tried to sync with it, so had no failure to report.

The \".variant\" files are because you must have made files with the same names in the 2 repositories while they were disconnected.
"""]]


code:
Hello,

I have added a few repositories for testing purpose and would like to remove them.
Is it possible to remove a repository through assistant?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-03-17T19:05:27Z"
 content="""
There's no UI for it in the assistant. [[Relevant bug report|bugs/The_webapp_doesn't_allow_deleting_repositories]]
"""]]


code:
Hello.

My case: I have somehow managed to get my repo, with quite much stuff inside it messed up.

There is single directory 'cities' containing quite many files (>10k) that i would rather see gone (and keep the tar.bz version of it ..). I tried doing

    git drop --force

that works fine, but the files are still there after sync.

git annex unused says just 'ok' so i guess it thinks they are still used somewhere. I tried to look where, but i ended up just doing git annex drop -f my_remote_here test_file_name for each of my remotes. This doesnt help.
How can i get rid of these files? Doing git annex fsck shows that

So, try to search what is the key:

     % ls -lah cities/diskcache/config.cfg
     lrwxrwxrwx 1 XX XXX 190 Nov 29 05:52 cities/diskcache/config.cfg -> ../../../../.git/annex/objects/qm/M6/SHA256-s32--4f5ce34d1b0b8d854a315530b2fdcbfa9c3067636a2aa5433a04402db4151dce/SHA256-s32--4f5ce34d1b0b8d854a315530b2fdcbfa9c3067636a2aa5433a04402db4151dce
     sundberg@sundberg-MS-7680 ~/git-repository/ubuntu.iso/Archive/Maps
     % git log --stat --all -SSHA256-s32--4f5ce34d1b0b8d854a315530b2fdcbfa9c3067636a2aa5433a04402db4151dce/SHA256-s32--4f5ce34d1b0b8d854a315530b2fdcbfa9c3067636a2aa5433a04402db4151dce
     commit 51a57a023774ff80408210828f298f5c42a7e0be
     Author: XXXX
     Date:   Sun Dec 9 13:42:40 2012 +0200
        git-annex automatic sync
     Archive/Maps/cities/diskcache/config.cfg |    1 +
     1 file changed, 1 insertion(+)

So how can i deduce what is the remote i should try to clean up ?

Thanks!


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkbv1oKTKhbBp0Ljh_WGU7BFSWWxBr7D3U"
 nickname="Pauli"
 subject="Just to add"
 date="2013-04-02T19:01:09Z"
 content="""

When i do 'git annex get .' on my usb-remote i get these kind of prints:

    get Archive/Maps/cities/diskcache/voices/slovenian_f/slovenian_female/g5ext_000 (not available)
    No other repository is known to contain the file.
    failed

No repository contains the file? Well why cannot i then get it removed from the listing ..
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkbv1oKTKhbBp0Ljh_WGU7BFSWWxBr7D3U"
 nickname="Pauli"
 subject="comment 2"
 date="2013-04-02T19:10:18Z"
 content="""
Just to continue my monolog: doing 'git rm -r cities' followed with git annex sync did the trick. Why didn't i find this anywhere ..
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 3"
 date="2013-04-02T21:21:21Z"
 content="""
git-annex adds large file capabilities to git, but it leaves all the regular git commands there for you to use. So the answer to \"how do I do <something I'd normally do with a git command>\" is always \"run the git command\". You only need to use git-annex to add large files to git, and to move their content around between repositories. Hope that helps clear that up.
"""]]


code:
Thank you for adding import options to handle duplicates. Very handy when consolidating data from various sources.

Can deletion of the source files be decoupled from annex duplication/deduplication options? For example, I would like to import source files without deleting them and at the same time do not import duplicates.

Better yet, since deletion of source files is potentially dangerous, a delete option could be required for deletion to be performed. Example:

git annex import --deduplicate --delete_all_source_files
git annex import --deduplicate --delete_source_duplicates

Also, it would be great to have import "status" option which goes over files to be imported and logs their status ( to be imported, duplicate etc. ) without actually performing any changes. It would be great for testing and trial runs.

I hope the above make sense. It would make import feature more flexible.

Cheers,


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkeJKC5Sy0stmcTWyePOLEVv0G-x1yaT_w"
 nickname="Josef"
 subject="wishlist"
 date="2013-09-26T11:11:19Z"
 content="""
Posted the above yesterday before realizing that it should probably go to wishlist requests. I am sorry about that.

Basically it is a request to extend import options and perhaps make the options easier to use/understand.

Suggested Import Options:

- source directory,
- destination directory,
- deduplicate in annex ( yes, no )
- delete source files ( yes, no )
- trial run ( screen output only )

Many thanks for a great product!

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkeJKC5Sy0stmcTWyePOLEVv0G-x1yaT_w"
 nickname="Josef"
 subject="Additional Comments"
 date="2013-09-30T21:33:31Z"
 content="""
Imported several thousand files to annex and would like to add the following comments:

- it would be great to have an option to exclude hidden dot files from import,

- empty directories should be deleted when files located in the directories are deleted,

- \"git annex add\" seems to process directories and files alphabetically, unfortunately import processes files in a different order, which makes it hard to predict which files are deleted when deduplicating,

Cheers,

"""]]


code:
Consider the following two use cases:

* I have a git-annex repo on a portable medium and carry it around between several machines. I use it on a non-important system with the most current git-annex installed, automagic upgrade happens. I am now forced to upgrade git-annex on all other machines. Bonus points if this happens in the background and I don't even notice it until it's too late.

* My system crashes and I use a rescue CD to access local data, including git-annex. The rescue CD includes a newer version of git-annex and once my system is restored, I am forced to upgrade git-annex locally.

My suggestion would be not to upgrade automatically, but to either ask the user if this is OK or to error out and request that they run git annex update by hand.

Optionally, this could be done via a local config variable which should default to error or ask, not upgrade.


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2011-03-18T00:38:51Z"
 content="""
These are good examples; I think you've convinced me at least for upgrades going forward after v2. I'm not sure we have enough users and outdated git-annex installations to worry about it for v1.

(Hoping such upgrades are rare anyway.. Part of the point of changes made in v2 was to allow lots of changes to be made later w/o needing a v3.)

Update: Upgrades from v1 to v2 will no longer be handled automatically
now.
"""]]


code:
So suppose you put an existing folder under git-annex control, like this:

    cd Archive_of_all_my_files
    git init .
    git annex init "Archive Folder"
    git annex add .

And now you forget the last step:

    git commit -am "init"

Instead, you fool around with git. After all, isn't git designed to keep your data safe and make sure there's no way you can accidentally lose it? The only branch that exists is 'git-annex', so you switch to that branch to see what it is:

    git checkout git-annex

This command switches to the branch git-annex and makes a mess in the working directory. And you can't switch back to master since master does not have any commits. Is there a clean way to go back? I tried 'git checkout --orphan master', but the messy files stayed in the working directory. In the meantime, there should probably be a warning on the walkthrough page.

I think git / git annex should warn you before switching the branch in this rare situation.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.193"
 subject="comment 1"
 date="2013-07-02T05:28:00Z"
 content="""
Yes, any git repository where there has not been an initial commit made is in an unusal situation. I've often felt this is a design flaw of git; it could start off with an empty root 00000 commit and avoid this complexity.

There are probably all kinds of fancy git commands that can be used to get out of the situation you describe, but finessing a perfect exit from a situation that has happened in the first 5 minutes of using a new repository seems like overkill. Just run `git annex unannex` to move all your files out of git-annex; run `git rm -rf .` to delete the files from the git-annex branch; run `rm -rf .git`, and you're back where you started with some files and no git repository.

... And no, neither git nor git-annex make any promises about keeping your data secure until you have both committed it and sent it somewhere else.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.193"
 subject="comment 2"
 date="2013-07-02T05:29:00Z"
 content="""
Oh and no, I don't think it's useful to bog down the walkthrough with warnings and caveats about this type of thing.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnHRhCe3qwVKQ8_NOGGSYJnAMW6FFyKbOc"
 nickname="Holger"
 subject="comment 3"
 date="2013-07-02T05:52:10Z"
 content="""
Thanks for your helpful response. The cleaning procedure that you propose is perfectly fine.

I wasn't suggesting to put up a big warning sign; I just feel like the walkthrough would be a little smoother if it explained something like: hey, 'git annex add .' just moved all of your precious files to '.git/annex/' and renamed them in a way that may scare you. But don't worry: if, at any point, you want to get them out again and put them back in their original place, just run 'git annex unannex'.
"""]]


code:
[[!comment format=mdwn
 username="Xyem"
 ip="87.194.19.134"
 subject="comment 4"
 date="2013-10-22T11:28:52Z"
 content="""
I've had this a few times (or situations very similar).

    git init
is now always followed by

    git commit -m \"CREATE GIT REPOSITORY\" --allow-empty

.. which I should really set as an alias :)
"""]]


code:
Hey,

I've found that git annex works great as a way to publish websites to a web server.  I can edit my website on my computer, `git annex sync` my working directory to the VPS, and then `git annex get files-I-want-to-publish`.  This works great.  I can maintain my normal working directory structure on the VPS and I don't have to worry about people seeing files I DIDN'T want to publish, since the dead symlinks just show up as 404s.

There's one small problem.

Say:

 1) I've already published a file using `git annex get file-to-update`

 2) I update that file on my computer

 3) I do `git annex sync`

 4) I do `git annex get file-to-update`

Between steps 3 and 4, file-to-update goes from being an accessible web resource to being a dead symlink.  It's not really a problem for me, as hardly anyone visits my site. But it would be nice if I could make `sync` leave the old symlink to the old file until I `get`ed the new one.

Is this possible?

PS: For those who might follow in my footsteps, remember that you probably don't want people reading the contents of your .git dir, so make a re-write rule for this!

Timothy


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 1"
 date="2013-08-07T16:18:52Z"
 content="""
I'd recommend that you just copy your files to the repository to publish them before running git-annex sync.

Alternatively, unless your web pages are very large, you can just check them into git directly, still use git-annex sync if you want (and still using git-annex for large files), and avoid the complication of being able to have files that are listed in the repository but whose content is not present.
"""]]


code:
I'd like to have git-annex assistant running so I can use it to sync files, but when I downloaded and ran it the webapp was missing.

I've recently begun using Trisquel, or rather the KDE version Triskel.  The current Trisquel is built from Ubuntu Precise, 12.04, the current LTS version.  Thus it seems that I'm a bit behind the current Ubuntu software, and my impression is that this relates to my problem.

In [a post](http://git-annex.branchable.com/forum/Cannot_launch_webapp_on_ubuntu_12.04_using_ppa/) on this forum I read that a version of git-annex was built without the webapp, and that this had since been corrected.  It appears from my git-annex version data that I have an older version, though I downloaded it only two days ago, and it's older than the corrected version.  Is this because my Ubuntu stuff is out of date?

More importantly, is there any way I can get webapp running on my current OS, or must I wait until Trisquel gets to 13.04?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.90"
 subject="comment 1"
 date="2013-07-17T19:36:27Z"
 content="""
There is a PPA for Ubuntu Precise, which includes the webapp.

Or you can use the standalone tarball, which works nearly everywhere.
"""]]


code:
Hello,

First of all, thanks to Joey for developing git-annex, good job!

I have a small feature request: when running git annex watch, new files are automatically added to the annex. It would be nice to let this depend on an attribute: add a file to annex if an attribute is set, otherwise do a regular git add.

My use-case is the following: I have a repository containing documents I'm working on (mostly LaTeX), which I'd like to be regular files in git (no annex), and a bunch of extra documentation (pdfs) and images, which I'd like to go to the annex. I currently set a git-attribute (addtoannex), and use a shell script to selectively add files to annex as follows:

Content of .gitattributes:

    *.png addtoannex
    *.jpg addtoannex

Snippet of add script:

    git check-attr addtoannex "$FILE" | grep -q ": set$"
    if [ $? -eq 0 ]; then
        git annex add "$FILE"
    else
        git add "$FILE"
    fi

It would be great if the watcher could honour an attribute.

best regards,

Tom


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.3.125"
 subject="comment 1"
 date="2013-02-07T18:10:14Z"
 content="""
This has been requested a couple of times in different places.

The problem with using a git attribute is it only allows basic matching on filenames. Some users would prefer to use the size of the file. So if/when this gets implemented it will probably not use a git attribute, but some other config setting.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-06-11T14:28:56Z"
 content="""
See [[tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant]]
"""]]


code:
Hi,

Suppose I have an annexed file whose content is stored on an extenal hard drive.
When the hard drive is mounted, is it possible to have immediately access to this file without transfering it, by modifying the symlink to point to the file content on the hard drive instead of having a broken link?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.244"
 subject="comment 1"
 date="2014-04-02T18:25:16Z"
 content="""
See [[todo/union_mounting]]
"""]]


code:
I don't think it's a bug or a todo, just a request for clarification.

So, I installed git-annex first on a Mac, and created the initial repository, and started dropping stuff into it. All fine.

Created a second repository on an external (but never disconnected, also does Time Machine) drive. All fine. Everything's there.

Went across the room to my Linux desktop, installed git-annex, and tried to pair it to my Mac's git-annex repo. Hit this bug, installed latest git-annex from tarball and got past that and got the pairing done. Syncing happened, all looked well...

But when I look in the annex dir on that Linux machine, I now just see symlinks to the actual files in the repo's .git/annex stuff. It kinda works, but it's a bit distracting, especially if you open such a file in any desktop app and you can see it's got the target of the link (some sha-value filename.ext) rather than the sensibly-named link. Including files created and coped into that annex on that machine. (In Nautilus they all show with symlink & lock emblems.)

Look again at some documentation, and apparently it's supposed to do this

Which makes me wonder why it *isn't* doing it on the Mac I started the repository on?

Went to another mac (laptop) and installed git-annex, and set up pairing to the first mac - as far as I can tell, doing the exact same procedure as I did under Linux. TBH through the webapp there's not much scope for doing it a different way. Files sync across...

And there they are, the actual files, directly in the annex dir, not symlinks.

(I am by the way confirming this at the commandline, it's not just what Finder/Nautilus may be displaying to me graphically.)

Now, seeing the actual files in the annex directory is what I *prefer*; but I'm a: confused as to why the Macs are doing it that way but Linux isn't, and b: given the "how it works" link, why everything isn't using the links.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.51"
 subject="comment 1"
 date="2013-09-12T17:58:24Z"
 content="""
When you create a git-annex repository using the webapp, it uses [[direct_mode]]. That is not the default if you create a repository by hand at the command line, although you can enable it if you prefer it.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlEhzszkzOIy8-Rx8b2mcr75QcnIc6O_OA"
 nickname="Rachel"
 subject="webapp/direct"
 date="2013-09-12T18:17:51Z"
 content="""
well, i did it using the webapp both times; i've barely scratched at the git-annex commandline so far.

However... I then tried to repeat it on another linux box (running ubuntu saucy and the version of git-annex in that repo) and it worked as expected: Real files in the annex directory, ie: as I understand it, direct mode.

So, went back to the first linux box (running ubuntu raring), and cleared everything off, quit and started again, doing the same thing again, and *it* did the direct mode thing itself this time.

Which I think leaves me with it the way I want it. :-)

My only guess is, maybe it got set up in non-direct mode when I first started using the older git-annex in the raring repos, before failing with the bad comment in ssh public key bug, at which point I switched to the downloaded-today tarball and just tried to resume. Starting from scratch with the newer version worked.

I'm sure more questions to come later; am a Dropbox fugitive by way of ownCloud (won't-fix bug on version upgrades) basically trying to replicate that kind of convenience. But those are for other postings.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.51"
 subject="comment 3"
 date="2013-09-12T20:22:44Z"
 content="""
Yep, relatively ancient (from last year) versions of the webapp didn't use direct mode by default, and have various bugs as well.
"""]]


code:
I have two machines (A and B ) both have a directory called "inbox"
I want to sync these two directories. So I ran git annex assistant on machine A and set up a remote to B with **ssh** and selected **client: a repository on your computer**.

The end result is that I have something that looks like a git bare repo on machine B and I have no files on A from machine B.

What am I doing wrong? What shall I do to achieve my goal?
It is kind of dropbox functionality what I want. (Which I never used for security/policy reasons.)


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-03-19T18:05:58Z"
 content="""
The \"Remote server using ssh\" option in the webapp is intended to set up a bare git repository on a server, not a non-bare git repository on a client.

For what you want, both your computers need to be running the git-annex assistant.

* If the computers are on the same network, pick the \"Local computer\" option on one to start a pairing process with the other.
* If the computers are not on the same network, but can (apparently) access one-another by ssh, then you're a lucky guy. :) But the webapp doesn't cater to this unusual configuration (unless the local pairing option above works). All you need to do though, it manually add a git remote on each that points to the already existing, git-annex assistant managed repository on the other. Eg: `git remote add B ssh://machineB:/~/annex`
* If the computers are not on the same network, and cannot access each other directly using ssh, you will need
to use the XMPP option, and a transfer remote that both can access.
"""]]


code:
I have a git annex repository which I use to store versioned binaries in a regular git repository. We also use submodules in the repository (the sources of the versioned binaries) but the binaries are not in the submodules. This has worked great for us.

Recently, there have been a number of changes (I got a new laptop, moved to fedora 20, renamed the repository, etc...) that happened at once. Now when I checkout a clean working version of the repository and then run git annex get . and then do an unlock and a lock on the file I get the "Locking this file would discard any changes you have made to it. Use 'git annex add' to stage your changes. (Or, use --force to override)" warning.

I have tried to isolate this as much as possible but can't find what caused this. On a co-worker's laptop it seems to work fine so far. A clean repository didn't fix it. An old repository didn't fix it (though I recall it working there previously on this laptop). It did work and just stopped recently.

    $ git annex version
    git-annex version: 5.20140107
    build flags: Assistant Inotify DBus Quvi TDFA
    key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SHA256 SHA1 SHA512 SHA224 SHA384 WORM URL
    remote types: git gcrypt bup directory rsync web glacier hook external
    local repository version: 5
    supported repository version: 5
    upgrade supported from repository versions: 0 1 2 4

    $ git annex lock --debug --verbose vendors/unittest-xml-reporting-1.5.0.tar.gz  # (some names changed to protect the innocent)
    [2014-02-05 12:54:45 IST] read: git ["--git-dir=/home/user/loppa_deuce/loppa/.git","--work-tree=/home/user/loppa_deuce/loppa","diff","--name-only","--diff-filter=T","-z","--","vendors/unittest-xml-reporting-1.5.0.tar.gz"]
    [2014-02-05 12:54:45 IST] chat: git ["--git-dir=/home/user/loppa_deuce/loppa/.git","--work-tree=/home/user/loppa_deuce/loppa","cat-file","--batch"]
    [2014-02-05 12:54:45 IST] read: git ["--git-dir=/home/user/loppa_deuce/loppa/.git","--work-tree=/home/user/loppa_deuce/loppa","diff","--name-only","--diff-filter=T","-z","--cached","--","vendors/unittest-xml-reporting-1.5.0.tar.gz"]
    lock vendors/unittest-xml-reporting-1.5.0.tar.gz git-annex: Locking this file would discard any changes you have made to it. Use 'git annex add' to stage your changes. (Or, use --force to override)


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmfVOIl06X7NKio-NBYZ-cfw5LFVqHMX8o"
 nickname="Daniel"
 subject="Hash is the same"
 date="2014-02-05T11:13:12Z"
 content="""
I forgot to mention that I also checked the hash of the file to make sure that nothing changed the file and nothing changed. Just quick unlock/lock triggers it.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmfVOIl06X7NKio-NBYZ-cfw5LFVqHMX8o"
 nickname="Daniel"
 subject="New git-annex package to blame"
 date="2014-02-05T14:23:40Z"
 content="""
I downgraded my git-annex version, rechecked out the repository and now everything works.
    $ git annex version
    git-annex version: 4.20130827
    build flags: Assistant Inotify

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="206.74.132.139"
 subject="comment 3"
 date="2014-02-06T17:05:45Z"
 content="""
Recent versions of git-annex have tried to extend the --force option to be needed in any operation that can possibly cause data loss. This includes locking a file, since that throws away any changes.

Note that `git annex lock` does not check if the file is unmodified. For a few reasons including

* some backends don't include a checksum
* it would be expensive to check a checksum
* the file could get modified after or during a checksum check, and those modifications would be missed

If you are sure you want to throw away any changes, use --force as suggested. If not, use `git annex add $file`, and assuming you're using a checksumming backend, it will notice the file has not changed and do what you want `git annex lock $file` to have done in this case.
"""]]


code:
Not logging this in the bugs section, but poking at the new webapp feature...

<pre>
laplace:annex jtang$ git annex webapp -d
[2012-07-26 19:02:50 IST] read: git ["--git-dir=/Users/jtang/annex/.git","--work-tree=/Users/jtang/annex","show-ref","git-annex"]
[2012-07-26 19:02:50 IST] read: git ["--git-dir=/Users/jtang/annex/.git","--work-tree=/Users/jtang/annex","show-ref","--hash","refs/heads/git-annex"]
[2012-07-26 19:02:50 IST] read: git ["--git-dir=/Users/jtang/annex/.git","--work-tree=/Users/jtang/annex","log","refs/heads/git-annex..731005d121426a38b206c4544da02cdb3b974974","--oneline","-n1"]
[2012-07-26 19:02:50 IST] read: git ["--git-dir=/Users/jtang/annex/.git","--work-tree=/Users/jtang/annex","log","refs/heads/git-annex..d36d8d88847decc2320f0be22892ad94a8abe594","--oneline","-n1"]
[2012-07-26 19:02:50 IST] read: git ["--git-dir=/Users/jtang/annex/.git","--work-tree=/Users/jtang/annex","log","refs/heads/git-annex..57bcddc14d03b61028f7002e2dabcc5181d74f3d","--oneline","-n1"]
[2012-07-26 19:02:50 IST] read: git ["--git-dir=/Users/jtang/annex/.git","--work-tree=/Users/jtang/annex","log","refs/heads/git-annex..372aceaf49b60ebe31cc3fe2e52ba4fbe99c134f","--oneline","-n1"]
[2012-07-26 19:02:50 IST] chat: git ["--git-dir=/Users/jtang/annex/.git","--work-tree=/Users/jtang/annex","cat-file","--batch"]
[2012-07-26 19:02:50 IST] read: git ["config","--null","--list"]
[2012-07-26 19:02:50 IST] call: open ["file:///Users/jtang/annex/.git/annex/webapp.html"]
The file /Users/jtang/annex/.git/annex/webapp.html does not exist.
git-annex: failed to start web browser on url file:///Users/jtang/annex/.git/annex/webapp.html
</pre>

I would have expected the open command to open up http://localhost:port/ instead of a file on my machine. Anyway, its just an observation on the current state of the webapp feature, not expecting it to work enough for me to test it ;)


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.2.189"
 subject="comment 1"
 date="2012-07-26T18:17:40Z"
 content="""
This could happen if you already had an old version of git-annex assistant running in that repository.

Try:

git annex assistant --stop ; git annex webapp
"""]]


code:
[[!comment format=mdwn
 username="jtang"
 ip="79.97.135.214"
 subject="comment 2"
 date="2012-07-26T20:42:37Z"
 content="""
The last few commits fixed the webapp startup quirk, it doesn't work yet for me as the watch command is probably failing

<pre>
laplace:annex jtang$ git annex webapp  -d
[2012-07-26 21:41:15 IST] read: git [\"--git-dir=/Users/jtang/annex/.git\",\"--work-tree=/Users/jtang/annex\",\"show-ref\",\"git-annex\"]
[2012-07-26 21:41:15 IST] read: git [\"--git-dir=/Users/jtang/annex/.git\",\"--work-tree=/Users/jtang/annex\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
[2012-07-26 21:41:15 IST] read: git [\"--git-dir=/Users/jtang/annex/.git\",\"--work-tree=/Users/jtang/annex\",\"log\",\"refs/heads/git-annex..731005d121426a38b206c4544da02cdb3b974974\",\"--oneline\",\"-n1\"]
[2012-07-26 21:41:15 IST] read: git [\"--git-dir=/Users/jtang/annex/.git\",\"--work-tree=/Users/jtang/annex\",\"log\",\"refs/heads/git-annex..d36d8d88847decc2320f0be22892ad94a8abe594\",\"--oneline\",\"-n1\"]
[2012-07-26 21:41:15 IST] read: git [\"--git-dir=/Users/jtang/annex/.git\",\"--work-tree=/Users/jtang/annex\",\"log\",\"refs/heads/git-annex..57bcddc14d03b61028f7002e2dabcc5181d74f3d\",\"--oneline\",\"-n1\"]
[2012-07-26 21:41:15 IST] read: git [\"--git-dir=/Users/jtang/annex/.git\",\"--work-tree=/Users/jtang/annex\",\"log\",\"refs/heads/git-annex..372aceaf49b60ebe31cc3fe2e52ba4fbe99c134f\",\"--oneline\",\"-n1\"]
[2012-07-26 21:41:15 IST] chat: git [\"--git-dir=/Users/jtang/annex/.git\",\"--work-tree=/Users/jtang/annex\",\"cat-file\",\"--batch\"]
[2012-07-26 21:41:15 IST] read: git [\"config\",\"--null\",\"--list\"]
git-annex: failed to start git-annex assistant
</pre>

it doesn't segfault or anything, it just states that the assistant command failed to start :P
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.2.189"
 subject="comment 3"
 date="2012-07-26T22:07:25Z"
 content="""
I've been improving this code path, so make sure you're current. I just made it wait for longer than the 10 seconds it was waiting before giving up.

If it still fails, try running `git-annex assistant --debug --foreground`
in one terminal, wait for it to start up, and then run `git annex webapp` in another.
If that still fails, check if `.git/annex/webapp.html` exists.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.2.189"
 subject="comment 4"
 date="2012-07-26T22:08:53Z"
 content="""
Oh, wait, you're right, if the assistant process that watch forks off is segfaulting, that error message is about the best we can hope for. You can try the manual startup of the assistant to confirm it's crashing..
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 5"
 date="2012-07-27T15:53:50Z"
 content="""
I can confirm that the watch command is crashing when i start it up manually.
"""]]


code:
[[!comment format=mdwn
 username="jtang"
 ip="79.97.135.214"
 subject="comment 6"
 date="2012-07-27T18:21:17Z"
 content="""
FYI, the webapp starts up and sends me to a web browser with the correct page ;) this is after the other issue of [[Watch command as of commit 6cecc26206c4a539999b04664136c6f785211a41 segfaults]] got fixed.
"""]]


code:
Lyve Home is an interesting device to store personal photos and videos. It is a 2TB hard drive that stores all your photos. It tries to replicate the files on every device you connect (computer, phone, etc.) and keeps at least two copies of each photo. The photos are not stored in the cloud, Lyve only stores metadata in the cloud.

http://www.youtube.com/watch?v=sOE5uWqS6YQ

This is something that could be easily made with git-annex and a small computer, like a Raspberry Pi, and a hard drive.


code:
Hi there,

first of all, great job done on annex! I think I finally found what I was looking for to organize all my data.

My question: when I do git annex whereis Music, I get a detailed list of every single song and its remote locations. I have tens of thousands of songs, so just typing git annex whereis Music gives me way too much information.
Here is my proposition: in my case, every file of the Music folder resides in the same remote. So, wouldn't it be better to show just one line of output telling me that Music is on remote bla and blub, and then I know that all files in Music are in remotes bla and blub?

Is that possible now already or would that be a feature request? What do you think?

Cheers,
Moritz.

PS: I know that I can specify filters, e.g. git annex whereis --in bla --and --not --in blub. That is great!


code:
Firefox has the nasty habit that it will force-dereference symlinks when locally opening files (i. e., opening an annexed document will cause it to be opened in .git/annex/objects/…). Since this will break relative links within HTML files, this would make Firefox pretty useless when working with a git annex containing HTML files. (Apparently this behavior is [desired](https://bugzilla.mozilla.org/show_bug.cgi?id=803999) upstream and might not be fixed.)

Seems I’m not the only one who would like to work with annexed HTML files, though. On the [Debian bugtracker](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=691099), another user shared a handy shim which can be used in LD_PRELOAD and which will force Firefox to open symlinks in-place. Thought I’d share this here in case it’s of use to anyone.


code:
[[!comment format=mdwn
 username="zardoz"
 ip="78.48.163.229"
 subject="comment 1"
 date="2014-07-31T11:43:16Z"
 content="""
Sorry, it escaped my attention there’s a dedicated tips forum. Maybe this should be moved there.
"""]]


code:
With the fundraiser having met its stretch goal of $15,000, I was thinking of how to keep git-annex going beyond another year.  What about selling stuff?  Not just merch (though being able to buy stickers would be neat), but useful things that would complement the software, like selling hardware for a small remote that you could plug into your home network, or even set up at a friend's house to have online, but offsite.

Something like a CubieBoard2 and an HDD for annexes preloaded with all the necessary software and a git-annex apt repo for keeping it updated.  Everything would come in a nice little enclosure with a git-annex logo on it; you plug in the power, the ethernet, and boom it's ready to go, with no terminal magic (but it's there if you want it!).  You could create them on-demand without having to keep much of a stock (except maybe the enclosure), so there's practically no risk.

Aesthetically, I was thinking of something that exposed as little of the insides as possible.  Basically just power, ethernet, and USB ports.

Maybe material for another Kickstarter?


code:
[[!comment format=mdwn
 username="GLITTAH"
 ip="94.126.178.1"
 subject="comment 1"
 date="2013-07-27T01:56:47Z"
 content="""
Other ideas welcome!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.246.110"
 subject="comment 2"
 date="2013-07-28T00:21:39Z"
 content="""
A few people have suggested that idea. I don't know if I want to get into the HW business.
"""]]


code:
http://git-annex.branchable.com/walkthrough/ says "Git wants to first stage the entire contents of the file in its index. That can be slow for big files (sorta why git-annex exists in the first place)."

What is git doing that git-annex isn't, other than copying the file to .git/objects rather than just moving it to .git/annex/objects, prepending it with "blob"+length, and compressing it? If git were changed to store the "blob"+length as part of the object filename rather than as part of the object file content, have a config option to use uncompressed objects for large files (and not try to pack them when creating pack files), and were used on a filesystem such as zfs or btrfs which does COW so the copy would be as fast as a move, then what speed advantage would git-annex still have over git? I realize git-annex has more features than just big file handling, and has the worm backend for even faster handling, but I'm just talking about the case with the default sha backend.

Have such changes been proposed for git? It seems that for anybody already familiar with the git codebase, adding the config option for uncompressed objects and moving the storage location for "blob"+length would be easy changes to make, and I see no downside to them. It wouldn't break backwards compatibility because the object filename being hash."blob".length rather than just hash would indicate that the new object format is in use, and a ".raw" filename extension could be used for uncompressed objects (or more sensibly, in the new format, no additional extension for uncompressed, and ".compressed" for compressed).

This would also eliminate the need for a git-annex object store separate from the git object store, and the complexities involved with having them separate, and the need for symlinks, and the complexities they cause. I don't think that relying on COW for speed is unreasonable once btrfs becomes the default in major Linux distros (the bsds already have zfs and hammerfs); right now part of what git-annex is doing is just working around the functional deficiency of non-COW filesystems.

P.S. I recommend a "plain" option for the page type when submitting comments on your wiki, so I don't have to put HTML line break markup at the end of my lines.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.2.245"
 subject="comment 1"
 date="2012-05-08T18:22:12Z"
 content="""
git's code base makes lots of assumptions hardcoding the size of the hash, etc. (grep its source for magic numbers 40 and 42...) I'd like to see git get parameratised hashes. SHA1 insecurity may evenutally push it in that direction. However, when I asked the git developers about this at the Gittogether last year, there were several ideas floated that would avoid parameterisation, and a lot of good thoughts about problems parameterised hashes would cause.

Moving data into git proper would still leave the problems unique to large data of not being able to store it all on every clone. Which means a git-annex like thing is needed to track where the data resides and move it around.

(BTW, in markdown, you separate paragraphs with blank lines. Like in email.)
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/IAg3idYGk.joxsJb2WCxl20gig.0.8hS#d5165"
 nickname="Kelly"
 subject="comment 2"
 date="2012-05-10T15:01:15Z"
 content="""
I think my comment a couple days ago got caught in the spam filter, so I'm reposting.
What were the ideas to avoid parameterisation? What were the problems of parameterisation, other than just the current hardcoded assumptions?

Speaking of hash insecurity, http://static.usenix.org/events/hotos03/tech/full_papers/henson/henson_html/node8.html says compare-by-hash is a bad idea. As I understand, git doesn't have an option of verifying content matches when the hash matches when adding data to the object store (like zfs's \"dedup=verify\" option, which you can use even when using sha256), because the assumption is that the risk of collision (or at least just the risk of accidental collision) is negligible. Would it be worthwhile to add this option to git-annex?

"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/IAg3idYGk.joxsJb2WCxl20gig.0.8hS#d5165"
 nickname="Kelly"
 subject="comment 2"
 date="2012-05-09T01:22:13Z"
 content="""
What were the ideas to avoid parameterisation? What were the problems of parameterisation, other than just the current hardcoded assumptions?

Speaking of hash insecurity, http://static.usenix.org/events/hotos03/tech/full_papers/henson/henson_html/node8.html says compare-by-hash is a bad idea. As I understand, git doesn't have an option of verifying content matches when the hash matches when adding data to the object store (like zfs's \"dedup=verify\" option, which you can use even when using sha256), because the assumption is that the risk of collision (or at least just the risk of accidental collision) is negligible. Would it be worthwhile to add this option to git-annex?

"""]]


code:
[[!comment format=mdwn
 username="http://christian.amsuess.com/"
 nickname="chrysn"
 subject="comment 2"
 date="2012-05-11T17:40:20Z"
 content="""
from my layman's standpoint, i think it would be feasible. i've suggested this previously, but not pushed it too much. quoting from [[my user page|users/chrysn]]:

* **would like git-annex to**: not be required any more as git itself learns to use cow filesystems to avoid abundant disk usage and gets better with sparser checkouts (git-annex might then still be a simpler tool that watches over what can be safely dropped for a sparser checkout)

*concerning hash sizes or parameterized hashes*: the problems with hash sizes could be avoided if instead of putting the objects in the \"normal\" object dir, barefiles would be managed in a similar way as packs are. when a new files gets added, they'd be cow-copied to ``.git/objects/bare/${HA}/${SH}``, and ``.git/objects/bareprefix/${HA}/${SH}`` would contain the \"blob ${SIZE}\0\" prefix that gets concatenated to the object body to form the object itself.

(maybe it'd even be sufficient to *just store the size* in the bareprefix, as all those objects would be blobs, but then again, some flexibility won't hurt.)

if the *pack file format* is flexible enough, the bareprefix files can get packed too. for the adventerous user who modifies bigfiles, the pack file mechanisms should be made aware of their presence, and be able to store deltas between them. the operations for applying those deltas would be difficult to optimize, and could be added at a later stage. a typical example could be storing a pdf file -- the pdf file format is designed for appending, so chances are the new version is just the old version plus several k at the end.

neither of that would affect git's *wire protocol*, so no compatibility problems. (it would be advisable to find a reasonable way to do sparse checkouts, though; something like \"server, pack and send your master, but make it sparse and don't include blobs >1mb\").
"""]]


code:
I have come up with a moderately complex solution to a particular use case that I have and am posting it here in case it is useful to someone else, and to get suggestions on how to improve it.

#The problem:

I have a large number of files that are accessed infrequently and stored off-line on DVD-Rs.  I need to keep track of which files are on which disc so that when I want a file I can find it.

#The solution:

I currently keep a text file to track which files are on which discs.  I would like to organize all the files in a proper filesystem using git annex, allowing me better organization and the ability to keep some smaller related files online near the annexed large files.

#Requirements:

1) Easily locate the DVD-R containing any specific offline file

This is easily taken care of with git annex whereis

2) Automatically de-duplicate stored files with the same contents

This is taken care of with one of the hash backends (E.G. SHA256)

3) The DVD-Rs still need to be usable without git or git-annex (E.G. The stored files should retain their normal human readable names)

This requirement rules out dir and rsync special remotes, they store the files named according to their hash.  I have settled on making each disc a separate repo which will satisfy this requirement.

#Future goals:

4) Easily incorporate the current DVD-Rs into the new system

I haven't found a way to fulfill this goal yet.  I have some convoluted ideas, but nothing so easy as mount disc, run git annex command.

#The solution in detail
Suppose you have the following tree:

~/mainrepo/thing1/file1.bin
~/mainrepo/thing1/description1.txt
~/mainrepo/thing2/file2.bin
~/mainrepo/thing2/description2.txt

You want to store thing1 on disc1 and thing2 on disc2, but you'd like to keep the descriptions online because they are small and useful for figuring out which thing you want later.

1) Create the main repo and annex the files:

cd ~/mainrepo
git init
git annex init mainrepo
git annex add .
git commit -m 'added files'

2) Create two new unrelated repos and populate them with their respective data and annex:

cd /tmp
mkdir disc1repo disc2repo
cd disc1repo
cp ~/mainrepo/thing1/* .
git init
git annex init disc1
git annex add .
git commit -m 'added files'
cd ../disc2repo
cp ~/mainrepo/thing2/* .
git init
git annex init disc2
git annex add .
git commit -m 'added files'

3) This is optional, but after annexing the files in these new repos, I replace the symlinks pointing into to the .git/annex/objects/ directory with hard links.  This makes the DVD-Rs usable from operating systems that can't deal with symlinks.  (mkisofs handles hard links correctly)

cd /tmp
find disc1repo/ disc2repo/ -type l -execdir sh -c "mv -iv {} {}.symlink && ln -L {}.symlink {} && rm {}.symlink" \;

4) Burn these repos onto DVD-Rs:

cd /tmp
#make isos
mkisofs -volid disc1 -rational-rock -joliet -joliet-long -udf -full-iso9660-filenames -iso-level 3 -o disc1.iso disc1repo/
mkisofs -volid disc2 -rational-rock -joliet -joliet-long -udf -full-iso9660-filenames -iso-level 3 -o disc2.iso disc2repo/
#burn the isos (untested command)
cdrecord -v -dao disc1.iso
cdrecord -v -dao disc2.iso

5) Mount the DVD-Rs and add as a remote and fetch, then drop from the mainrepo:

cd ~/mainrepo
#disc1
mount /mnt/cdrom
git remote add disc1 /mnt/cdrom
git fetch disc1
git annex drop thing1/thing1.bin
umount /mnt/cdrom
#disc2
mount /mnt/cdrom
git remote add disc2 /mnt/cdrom
git fetch disc2
git annex drop thing2/thing2.bin
umount /mnt/cdrom

6) Enjoy!  You can now find out what disc things are on simply using git annex whereis, and you can git annex get them or simply use them directly from the disc.

I'd appreciate any comments and helpful suggestions.  Especially how to simplify the process or easily integrate all the things I already have stored on discs.

Maybe it would be possible to create a special remote using the hooks for the DVD-Rs.

Even though it is a bit tedious and complicated, the current process could be automated using a script.


code:
[[!comment format=mdwn
 username="Steve"
 ip="92.104.175.136"
 subject="Web remote works"
 date="2012-11-25T22:30:46Z"
 content="""
Thanks for the suggestion Joey, I found a way to make the web remotes work for adding the files from existing discs.  I wound up adding a symlink farm to the repo with a link for each disk pointing at the mount point.  This way when I try to retrieve a file, I see the URL which contains the name of the disc:

    $ git annex get bigfile.bin
    get bigfile.bin (from web...)
    curl: (37) Couldn't open file /var/tmp/repo/storage/dvd13/bigfile.bin

      Unable to access these remotes: web

      Try making some of these repositories available:
            00000000-0000-0000-0000-000000000001 -- web
    failed
    git-annex: get: 1 failed

It took me a while as the version of git-annex in portage was rather old and I just didn't get around to updating git-annex on Gentoo for while.  If anybody wants to get git-annex 3.20121112 running under Gentoo I detailed the process I used at <http://git-annex.mysteryvortex.com>

Now I'll have to try out the assistant!  (Though I didn't get the webapp to compile due to a shakespeare-js error)
"""]]


code:
[[!comment format=mdwn
 username="Steve"
 ip="92.104.175.136"
 subject="Bad ebuilds"
 date="2012-12-01T02:45:37Z"
 content="""
Just an update for anybody that used the ebuilds I created from the link above.  They did not create the git-annex-shell symlink which can cause git-annex to start ignoring your remotes.  More details and fixed ebuilds are now on the page.
"""]]


code:
[[!comment format=mdwn
 username="https://launchpad.net/~arand"
 nickname="arand"
 subject="comment 12"
 date="2013-03-11T10:34:42Z"
 content="""
Without having read this, I've reported a very similar wishlist item at:
[http://git-annex.branchable.com/todo/wishlist:_recursive_directory_remote_setup__47__addurl](http://git-annex.branchable.com/todo/wishlist:_recursive_directory_remote_setup__47__addurl)

combining a recursive addurl (in my case using --fast) script with the suggestions regarding symlinks here, it's somewhat workable:

    ln -s /media/cdrom /var/tmp/mycdrom123
    ~/utv/scripts/annex-importdir /var/tmp/mycdrom123

Ideally though, for optical media it would have a couple of more features (some already noted above):

* Ability to form a (reasonably) unique identifier from a disc, using the label and the date of creation
  * Ability for Annex to identify discs using this and ask for the correct disc if the file does not match (accomodating RW discs where label and date might change, or simply disc copies)
    * Example: `not the original disc... trying anyway... file hash mismatch... please enable the remote disc with \"MYLABEL\" and creation date \"2001-01-01\"`
* Option to checksum without importing the actual objects into the annex
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/2grhJvAC049fJnvALDXek.6MRZMTlg--#eec89"
 nickname="John"
 subject="Have you seen the dar utility?"
 date="2012-10-20T19:03:37Z"
 content="""
http://dar.linux.free.fr/doc/index.html

Would be nice to have this as another remote option for git-annex, since I too would like to have static (and possibly incrementally extended) remotes that span multiple DVDs
"""]]


code:
[[!comment format=mdwn
 username="Steve"
 ip="92.104.175.136"
 subject=""free-form" special remote / dar utility"
 date="2012-10-20T22:11:23Z"
 content="""
dar looks familiar, I'm sure I have run across it in the past.  However, it is not suitable in this case; see requirement #3 above that the DVD-Rs be usable without git or git-annex.

What would work we be some sort of special remote that allows free-form data.  Imagine that you create the DVD-R with the files on it, then you mount it and add the mount directory as a free-form special remote.  git-annex checksums all the files under the specified directory and stores the relative path to each file somewhere.  Then, when you want to fetch a specific hash from the remote it looks up the relative path, adds it to the base directory and transfers it into the local .git/annex/objects/ store.

"""]]


code:
[[!comment format=mdwn
 username="http://lj.rossia.org/users/imz/"
 ip="79.165.57.104"
 subject="Yes, I agree, such a special remote for free-form read-only media would be convnient."
 date="2012-10-20T23:58:45Z"
 content="""
I have already stored a lot of large files on DVDs. I did that for arhiving, so I cared that there are several copies. But I want this to be more automated.

I take my disc (or one created by someone else, without any knowledge of Git), checksum its contents in git-annex, and in the projects where I'm using this content, I can check that the file is archived on at least N discs.

Also, I might enhance the content -- this would be refected in a Git commit, so then I want also to be able to check that the new version has also ben archived on severeal discs.

A special remote for such free-form read-only media would be very convenient.
"""]]


code:
[[!comment format=mdwn
 username="Steve"
 ip="92.104.175.136"
 subject="Some free-form remote ideas"
 date="2012-10-21T02:07:40Z"
 content="""
This is starting to get interesting.  A free-form remote would definitely simplify my use case, and also solve the \"future goal\" of easily incorporating my already existing DVD-Rs.

I haven't really looked into the git-annex internals up to this point, but looking at the [[special_remotes/hook]] page there doesn't seem to be a hook for init which would be needed to populate git-annex's index of files in the remote.  (git-annex seems to assume that new special remotes are empty)

Another problem is where to store the hash to path relation information.  On a RW remote it would be stored in the remote, but here we need to keep it in the repo somehow.  This could be in the git-annex branch, or possibly another branch created specifically for this purpose.

1) initremote needs to:

* hash the contents of all the remote's files
* update git-annex's index of the remote's contents
* store the paths to the hashes in the repo

2) store and remove should just fail.

3) retrieve and check present seem straight forward.

The assistant blog mentions adding support for read only remotes but I don't know anything about it: [[design/assistant/blog/day_65__transfer_polish]] (I'm still on 3.20120605)

Let me know if there is anything I haven't thought of yet.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.23"
 subject="comment 5"
 date="2012-10-21T05:36:36Z"
 content="""
I encourage playing around with the hook special remote and see how far you can make it go.

I may be doing something vaguely like this for [[design/assistant/desymlink]], although I'm pretty sure it would still have a git repository associated with the directory of regular files.

One option is to use the web special remote, with file:// urls. Assuming a given disc will always end up mounted somewhere stable, such as /media/dvd1, /media/dvd2, etc, you could then just `git annex addurl file:///media/dvd1/$file`. `git annex whereis` will show the url, which has enough info to work out the disk to mount.

The web special remote did not support file:// urls, but I've just fixed that. The only downside is that, while it will identify files duplicated across disks, and `whereis` will show multiple urls for such files, there's only one web special remote, and so it only counts as 1 copy. This could perhaps be improved; git-annex may eventually get support for remotes reporting how many copies of a file they contain.
"""]]


code:
[[!comment format=mdwn
 username="http://www.openid.albertlash.com/openid/"
 ip="74.96.185.87"
 subject="It works!"
 date="2012-10-24T22:00:31Z"
 content="""
This works great! I first tried it with WORM, no-go. I can see why the SHA backends are so powerful, they appear to circumvent the commits which git usually uses for merging. When I first do the merge, it reports this:

warning: no common commits

Compared to how I've managed CD/DVD backups in the past, this is a quantum leap forward, and I don't find it convoluted in comparison. Yes, there is dar, but I prefer this method. In my case, its the perfect solution for original files, which in generally are treated as immutable, and not accessed very often. They are usually large, too! I'm using them for digital pictures.

"""]]


code:
[[!comment format=mdwn
 username="Steve"
 ip="92.104.175.136"
 subject="web and hook special remotes"
 date="2012-10-24T23:26:53Z"
 content="""
Hi Joey,

Thanks for the advice.  I had thought of the web special remote; but as you may have noticed from my example, I don't use automount so my DVDs and CDs all get mounted in the same place. (/mnt/cdrom) so the web special remote won't work for me.

I'll try to play around with the hook special remote this weekend.  I had a thought it might be interesting to have it search for the DVDs in some common places or even by parsing the mounted file systems, and allow an override or augmentation through git config.

"""]]


code:
[[!comment format=mdwn
 username="Steve"
 ip="92.104.175.136"
 subject="no need to merge"
 date="2012-10-24T23:52:30Z"
 content="""
Albert,

Thanks for feedback!  I'm glad that somebody else found the method I worked out useful.  As I'm going to try and turn it into a proper special remote, let me know if there is any particular use case or feature you'd like me to address.

Note that in my testing, I found that you don't actually need to merge the DVD's branch into the local branch you are using for git annex to be able to find the files on it that are identical to files in your local branch.

I haven't played around with cloning the repo, but I will try that this weekend.  I'm thinking it *might* be necessary to create local branches from the DVD remotes so that they'll get carried along when you clone the repo.

As far as the repos on the DVD's not having a shared ancestry with main repo, that was a conscious choice that I made.  I wanted to add as little extra data to the DVDs as possible since I usually fill them to the brim anyway.  I didn't feel that it would be beneficial for the DVD's to know about the history of the main repo and other files that they don't contain.  Furthermore, besides all the links and history, you'd be replicating all the files in the main repo that aren't annexed.

If you want to avoid the error, but still have a local branch for the DVD repos you should be able to do something like the following:

WARNING: these commands are untested!

git checkout -b disc1 disc1/master
git checkout -b disc2 disc2/master

Working from the original example, you should then get local branches for the DVDs that don't have a common ancestor with your master local repo.  I haven't actually tested that though.  Testing will have to wait for this weekend.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.138"
 subject="comment 9"
 date="2012-10-25T03:33:29Z"
 content="""
@Steve, it seems to me you could still use the web special remote, just pointing it at an url that goes through a symlink to the mount point.
"""]]


code:
Is it possible to run more than one instance of the assistant in an account?

A particular example is that I might have two annexes:

1. for my music, and
2. for some personal documents.

I would like to have my music annex paired with my work laptop and have all the automagical power of git-annex-assistant working for me.  However I don't want to put my personal documents on a work machine, hence the second annex.

(I think) An ideal world would have the assistant managing both annexes.  Is this possible?

I know my question is similar to [one annex versus many annexes?](http://git-annex.branchable.com/forum/one_annex_versus_many_annexes__63__/) but I think it is different enough to warrant a new thread...


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.23"
 subject="comment 1"
 date="2012-10-19T17:08:39Z"
 content="""
If you go to the upper-right menu in the webapp, and select \"New repository\", you can add a separate independant repository. The assistant's autostart will run it on both of them when you log in.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkk3K0AUduAybbBO_LRRGKOe2zcGeezbzI"
 nickname="Nathan"
 subject="Webapp spins when adding a second annex"
 date="2012-11-20T04:55:05Z"
 content="""
I was trying to add a second annex (repository) via the webapp, but it seems like the process never completes: the browser just spins (\"connecting...\"). If I kill the webapp process and restart, I can see the second annex in the list of repositories, but clicking on it gives the same behaviour. Interestingly, the path *is* added to ~/.config/git-annex/autostart, and the webapp does create the path, if needed, and initializes it as a git annex. I've reproduced this on two different machines, both running Ubuntu 12.10 and git annex 3.20121112.

Is there anything I can do to help debug?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.6.49"
 subject="comment 3"
 date="2012-11-25T18:59:37Z"
 content="""
@Nathan at this point, you should have two separate git-annex processes running, one for each repository. You can try opening a shell, changing to the new repository, and running `git annex webapp --debug` to get debugging information on the new one.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.6.49"
 subject="comment 4"
 date="2012-11-28T20:13:11Z"
 content="""
@Nathan, I have just noticed and fixed a problem that could cause this to happen if you were using the standalone tarball (or OSX app) builds. It would not affect git-annex installed by other methods.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkk3K0AUduAybbBO_LRRGKOe2zcGeezbzI"
 nickname="Nathan"
 subject="comment 5"
 date="2012-12-11T04:15:49Z"
 content="""
Thanks, Joey; I was using the standalone build, and it seems to be behaving better now.
"""]]


code:
I have the webapp setup to manage 4 different repositories.

One repository had many gigabytes to upload to a remote. As this was moving along, I wanted to check on the status of one of the other repositories. I knew that this second repository had a lot of data to download. When I switched to the second repo in the webapp, I noticed in my network monitor that the amount of upstream bandwidth I was using dropped and the amount of downstream bandwidth increased. I switched back to the first repo, and my downstream bandwidth dropped and the upstream bandwidth increased.

It looks like git-annex stopped uploading data from the first repo when I switched to the second, and stopped downloading data to the second repo when I switched to the first. Is this correct? I was under the impression that switching repositories in the webapp simply changed the view that I was looking at, but that the assistant would still be managing all the repositories I had setup -- uploading/downloading/syncing concurrently.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-03-17T18:53:26Z"
 content="""
Each repository gets its own git-annex assistant daemon, which is entirely separate from the daemons used by other repositories.

The only affect that switching which repository is in view in the webapp can have is it can start a daemon running on a repository that for some reason did not already have the daemon running. This could happen if it was not set to autostart on login, for example.
"""]]