code:
[[!comment format=mdwn
 username="annexuser"
 ip="77.247.182.246"
 subject="comment 2"
 date="2013-03-17T22:25:58Z"
 content="""
If a special remote was setup using shared encryption, can I change it to using keys by simply doing `initremote myremote encryption=keyid`? Or should I remove the remote and start over?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 3"
 date="2013-03-18T15:09:53Z"
 content="""
git-annex won't let you change a remote's encryption scheme, as this would make any data stored in the remote using the old scheme be no longer accessible. You need to make a new remote. It should be fine to point the new remote at the same location as the old one, though.
"""]]

repo_name: kakafarm/git

license: unknown


code:
Hey Joey,
    Is there a way to see when the last fsck was?  I know about --incremental and the related options, but sometimes I'd like to know when the last time I fsck'd a file (or even the whole repo) was.  There doesn't seem to be a command line option for it, but I know that info is saved somewhere...maybe this could be part of 'git annex status'?

$(words of gratitude and encouragement)


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-05-19T18:50:06Z"
 content="""
You're quite right. that info is stored in `.git/annex/fsckstate`. The timestamp of the file is when the last incremental fsck was started. The contents of the file is the same date, represented as seconds from epoch.

That is only written to when doing an incremental fsck though. A non-incremental fsck does not touch the file.
"""]]


code:
I would like to use git-annex with [S3 reduced redundancy storace (RRS)](https://aws.amazon.com/s3/faqs/#rrs_anchor), a cheaper S3 service with an higher chance of seeing your files disappear overnight. This means that git-annex may think that a certain files has been copied to a S3 remote while, in fact, it is no longer there.

I would like to check that the files S3 are really there and, if some files are not, record that we lost that copy. In the case of S3, either the data is there and it is complete or it is not there at all, so there is no need to check the data itself, just the presence of the files.

Is there a way to check that all the files that are supposed to be in a S3 remote are still there?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.114"
 subject="comment 2"
 date="2014-04-24T18:12:59Z"
 content="""
git annex fsck --fast --from $yours3remote
"""]]


code:
Hi,

I really like git-annex for sharing contents with all my devices. But i have a problem with the assistant on my phone, i don't know how to git annex get some file but not all.
For example i have this on a remote:

repository:
 - big file1
 - big file2
 - big file3

I have this repository on my phone. I would like to be able to checkout out only "big file2" directly within the assistant.
How can i achieve that?

Thank you.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.114"
 subject="comment 1"
 date="2014-04-24T18:15:15Z"
 content="""
[[preferred_content]] expressions can be used to do that. Works best if all the files you want have some extension or are in a specific directory.

There are some [[preferred_content/standard_groups]] that might do what you want, particularly the \"manual\" one.
"""]]


code:
[[!comment format=mdwn
 username="elfangor"
 ip="176.57.244.101"
 subject="comment 2"
 date="2014-04-28T13:12:07Z"
 content="""
Thanks for your reply.

The Standard group \"manual\" seems to be what i need. The last thing i'd like to now is, is there a way to do git annex get on the assistant(on the webui) on my phone? (I know i can do this on the shell opened, but it's not that easy to write from the phone.) I guess it's not implemented, and i understand that it should be a lot of work to add a file browser in the ui where you can git annex get, and git annex add from it.

Thanks again for your answer, really helpfull: I have a better idea on how i will structur my git annex repositories.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="216.145.95.162"
 subject="comment 3"
 date="2014-05-19T15:45:57Z"
 content="""
There is not currently any UI to do git-annex get/drop.

There is some recent work on [[tips/file_manager_integration]] which lets you use a file manager and pick files you want and files to drop. Nobody has gotten it working with android file manager yet AFAIK.
"""]]


code:
I recently began experimenting with direct mode on a repository where only a very limited number of commands (`git annex sync`, `git annex add .`, etc.) are being used (through desktop buttons).

Things were working well, and then on another repository in indirect mode I moved a large folder and synced that data with a bare repo, and synced the repo in direct mode with the same bare repo (to get everything in sync).

At that point, git annex seemed to hang, taking forever to complete, and from looking a processes and files it seemed like it was going nowhere, so I killed it. I wish I had done a bit more diagnostics before that, but unfortunately I didn't.

At that point, I noticed that the old directory was still there and that there were still many files in it, but the new folder had also been created and there were files there, too. So I thought the transfer was done, and for whatever reason `git annex sync` had just not cleaned up properly.

In fact, the sync was only half done and the remaining files in the old directory were the only (local) copies (on the direct mode repo). I removed them and then synced again, which actually told git to delete those symlinks.

When I updated another (indirect mode) repository I noticed this, so I reverted the commit in question and got the symlinks back, no data lost. Then I went back to the direct mode repo, switched to indirect mode because I was worried about direct mode, `git annex sync`ed, then `git annex get` to get the files again from a usb and everything was back to normal.

Except that when I tried to go back to direct mode in that original repo, I got an error saying that git could not stat a file which is in the old (deleted) folder. Searching around, I noticed that in the `.git/annex/objects` directory, there are many remaining `.map` files with lines referring to the old (deleted) directory.

Is there any way to "reset" this somehow? I would like to switch back to direct mode, and I'd prefer not to recreate the repo.

Oh and I'm using version: 4.20130501-g4a5bfb3.

Thanks!


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnkBYpLu_NOj7Uq0-acvLgWhxF8AUEIJbo"
 nickname="Chris"
 subject="comment 1"
 date="2013-05-10T03:50:26Z"
 content="""
Maybe a simpler question: if I delete all the `.map` and `.cache` files that are leftover in the `.git/annex/objects` directory, will that effectively \"reset\" the status so that I can then switch to direct mode?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnkBYpLu_NOj7Uq0-acvLgWhxF8AUEIJbo"
 nickname="Chris"
 subject="comment 2"
 date="2013-05-10T04:01:38Z"
 content="""
After reading [this](http://git-annex.branchable.com/internals/):

> Also in direct mode, some additional data is stored in these directories. .cache files contain cached file stats used in detecting when a file has changed, and .map files contain a list of file(s) in the work directory that contain the key.

I decided that indeed, deleting the `.map` and `.cache` files should do the trick, so I deleted them and direct mode works again.

Still not sure what went wrong with the original sync, but at least I got things back on track.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 3"
 date="2013-05-13T18:00:42Z"
 content="""
This sounds like one or more bugs that I should address, but I'm having trouble understanding exactly what you did, from your description, at the level of detail I need to replicate a problem.

Do you still have a copy of the corrupted repository? Or any error messages to show me?

It sounds like at least one problem might have been this bug: [[bugs/Unable_to_switch_back_to_direct_mode]]
At least that is fixed in current git.
"""]]


code:
So i am coming from git world, All documentation and posts seemed to be for media etc. I have a simple question.

Can i use git annex to manage large files , Which i think i can. But will it work with branching/tagging etc.

e.g. We currently use git have some large files in git repo.... I am planning to move the large files to use annex. But would like to maintain branches, i.e. each branch might have different version of files and maybe tagged, So i can get back some old version from a branch.

Is this possible ? I did not get any explicit  answer or examples on how this works or even if its supported.

Also can i use same folder for git and git-annex ?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.235"
 subject="comment 10"
 date="2013-07-10T17:05:49Z"
 content="""
It has its own ssh server written in java too, which hardcodes the set of commands it supports. AFAICS it would require some java coding to add a command to this, it does not seem to be designed for easy extensability.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn5RcmefXjrl1vmbHIiOWQyXGXVKxlm3rg"
 nickname="Kavin"
 subject="comment 11"
 date="2013-07-10T20:14:52Z"
 content="""
:( Ahh... So i guess this might not be possible in near future.

Ill go to plan B, Setup another server which users will connect to for large files which will have git-annex installed. (So a central Repository for git annex). They will need to add another remote in git config.

Does git annex have a way to manage ssh keys ?  (Please can you point me to the location where i can read how git-annex-shell ? )
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.235"
 subject="comment 12"
 date="2013-07-10T21:10:03Z"
 content="""
Yes, a separate repository is one way to go, although I do not think it would be very hard to get gerrit's internal ssh to run git-annex-shell.

[[git-annex-shell]] is a low-level restricted shell like git-shell, not a login manager, so it has nothing at all to do with ssh keys. Things like gitosis and gitolite can be built on top of git-shell and git-annex-shell and handle ssh key management.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.235"
 subject="comment 1"
 date="2013-07-08T20:09:54Z"
 content="""
Yes, git-annex would not do very well at adding large file support to git if it did not handle tagging, branches, etc! So of course it does. It's in a sense too obvious a thing to get much mention. And so people sometimes get confused about it.

The only thing to need to be aware of coming from git is that not every repository will have every version of every file locally available. When you check out a branch, you may need to run `git annex get` to retrieve those versions from origin or elsewhere.

And, `git annex unused` can be used to find versions of files that no existing tag or branch refers to, and `git annex dropunused` can then delete those versions. If you want to ensure every revision in your git repo is accessible, you should avoid using those two commands; otherwise git-annex will never delete old versions of files.

The unreleased git master adds a new feature, a --all switch that makes git annex commands operate on all versions of files. While normally `git annex get` will only do what it needs to to get all files in the currently checked out branch, `git annex get --all` will pull down every version of every file in the whole history. Similarly, `git annex copy --all --to origin` will ensure that every locally available version of every file is sent to origin.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.235"
 subject="comment 2"
 date="2013-07-08T20:18:27Z"
 content="""
And yes, you can use the same git repository for files added to git `git add file` and added to git annex `git annex add file`
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn5RcmefXjrl1vmbHIiOWQyXGXVKxlm3rg"
 nickname="Kavin"
 subject="comment 3"
 date="2013-07-08T23:27:45Z"
 content="""
Awesome!! thanks, As i am kicking the tires, Based on my current scenario, How can i deploy this in my company , So each developer to do the bare minimum, Like just install git annex and it should work.

i.e. is it possible to avoid the step git remote add backup ? Then they can just do

$ yum install git-annex
$ cd repo
$ git checkout staging
$ git pull
$ git annex get large_file

And it then just works.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.235"
 subject="comment 4"
 date="2013-07-09T18:14:07Z"
 content="""
I don't know where you got the \"git remote add backup\" step from. Obviously this is not necessary unless you want to add a remote named \"backup\".

To use git-annex in a centralized git environment, which it sounds like you have here, you just need to install git-annex on the central git server, and arrange for all the developers to have ssh access to it. Then any git repository that is cloned from that server using ssh as the transport can support git-annex without the user needing to do anything special to set it up.

This assumes that users have shell accounts on the server. git-annex includes the git-annex-shell program, which is similar to the git-shell in git. User accounts can be locked down to use this restricted shell if giving them full shell access to the server is not desired.

If the server is using a git repository manager like gitolite or gitosis, those can also be adapted to use git-annex-shell. I got gitolite patched to support it earlier, see [[tips/using_gitolite_with_git-annex]].

PS: I'm available for consulting on deploying git-annex in production environments. ;)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn5RcmefXjrl1vmbHIiOWQyXGXVKxlm3rg"
 nickname="Kavin"
 subject="comment 5"
 date="2013-07-09T20:32:51Z"
 content="""
Thank you very much!!!!! Yes, Planning to use Centralized repo. Basically what i understood was there will be a git server and separate server for hosting the large files. Its great to know i can use the same server and do not need to explicitly point to the server.

As we are using Gerrit, What is the general best practice for it. (After i am done with it, I will write a blog post, As i am sure many people with centralized repo's might need this)
Thanks!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.235"
 subject="comment 6"
 date="2013-07-09T20:39:25Z"
 content="""
Yes, the default for a repository accessed using ssh is to also store the large files in that repository. Of course you can set up a second remote to hold the large files if that works better.

I don't have any personal experience with Gerrit. All I can say is that files managed by git-annex will appear as symlinks.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn5RcmefXjrl1vmbHIiOWQyXGXVKxlm3rg"
 nickname="Kavin"
 subject="comment 7"
 date="2013-07-09T23:48:17Z"
 content="""
Thanks, So i made some progress, In the gerrit repo server,  (which has git-annex installed)

I ran:
    git annex init origin,

Did a clone of the repo, it had the branch git-annex, So i assumed all is working.

I did a
    git annex add
and
    git push

Then when i try to
    git annex copy --to origin

I get error , But git-annex-shell is installed on the machine:

    $ echo $PATH
    /usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/gerrit2/bin
    $ which git-annex-shell
    /usr/bin/git-annex-shell

    Gerrit Code Review: git-annex-shell: not found
    rsync: connection unexpectedly closed (0 bytes received so far) [sender]
    rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]

    rsync failed -- run git annex again to resume file transfer failed

Any idea what i  need to do to make it work ?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.235"
 subject="comment 8"
 date="2013-07-10T16:42:18Z"
 content="""
If gerrit is handling incoming ssh connections, it's probable running its own restricted shell. That shell would need to be configured to spawn git-annex-shell when asked to do so in order for git-annex to transfer files to that repository.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="comment 9"
 date="2013-07-10T16:57:29Z"
 content="""
AFAIK Gerrit relies on jgit implementation of git in java; allowing it to execute helper programs like git-annex-shell may be tricky.
"""]]


code:
# Comparison

Based on my understanding of the http://git-annex.branchable.com/not/ page and of some research,
I have made a draft of an [Online excel document][1] that compares different git file solutions.

Feel free to edit the document and help the community to understand the different solutions out there.

[1]: https://onedrive.live.com/redir?resid=C66C90783BD2C185!1200&authkey=!ACF-Ol_mG3DwQ2k&ithint=file%2c.xlsx

Thank you very much,
Alex


code: I use git-annex mostly to archive my photos.  They're generally large raw files and I am encrypting them going to the S3 special remote.  Are there any plans to compress objects before encrypting them?  It would save on S3 costs, only add some overhead on get and copy, and I don't think there would be any problems looking up data later.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.8.117"
 subject="comment 1"
 date="2012-12-14T18:53:04Z"
 content="""
gpg compresses encrypted content by default, so all encrypted special remotes get compression for free
"""]]


code: Wepapp only allows me to setup consistency checks for the local repo. Non of my computers has full copy of repos. How can I setup annex so that every night for one hour it checks consistency of s3 data?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 1"
 date="2013-12-31T19:30:00Z"
 content="""
The webapp does not let you set up a consistency check for data in a S3 remote because the only way to check it would be to download the data, which is expensive in bandwidth and potentially, Amazon S3 fees.
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/FHnTlSBo1eCGJRwueeKeB6.RCaPbGMPr5jxx8A--#ce0d8"
 nickname="Hamza"
 subject="comment 2"
 date="2014-01-01T01:50:57Z"
 content="""
Is it possible to do it from command line? I was thinking with a time limit I would spread the fsck over a mount run it once a while to make sure all data is actually there when I need it.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 3"
 date="2014-01-01T21:05:20Z"
 content="""
Adapting an example from the man page:

`git annex fsck --incremental-schedule 30d --time-limit 1h --from S3`
"""]]


code:
I have some questions about how to best manage content on mobile
devices.  This is probably more of a wishlist discussion, although I
haven't had enough experience with the assistant and preferred content
features enough to know how much of this might already be possible.

(While I did donate to both campaigns, I also didn't donate at the
"personal git-annex assistant" level, so I also know not to expect
personal undivided attention :) . Any tips would be appreciated.)

In short, I don't think git-annex really fits with how I'd like to use
it on my Android phone.  My current setup is as follows:

- I have a large repository with various media files.  This is stored
  on several computers, at least one of which generally has a copy of
  everything.  I use a centralized bare repo to automatically
  synchronize between computers with "`git annex sync origin`" using
  cron, and manually control what is on each one using "`copy --to
  host`", "`get`", "`drop`", etc.  It is *awesome*.

What I would like for Android is:

- The phone keeps in sync with the origin repository.

- On the phone, I'd like to browse through and choose content to get
  or drop.  I could use the command-line "`get`" and "`drop`" for
  this, of course, but being able to do it through the webapp would be
  better, as typing on a phone is no fun.

- From _off_ the phone, I'd like to be able to choose which content
  should be _on_ the phone.  For example, using the command line or
  webapp on my desktop, I'd like to be able to choose media files that
  should be downloaded to (or removed from) the phone.  However, I
  don't want to have to move files between directories in the repository
  to achieve this.

- The only useful content that gets created on the phone is photos and
  videos.  Those should get uploaded to the main server, but only the
  most recent files should remain on the phone by default (say,
  anything older than 2 weeks gets dropped).  Both on the phone, and
  off the phone, I'd be able to override that using the previously
  described features, so that certain photos remain on the phone
  forever, or get dropped earlier.

Thoughts?  I feel like git-annex is close to being able to do this.

-jim


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl_EUsflI4UOGtx0-2yrRJuGAzNyN_5UYY"
 nickname="Chris"
 subject="comment 1"
 date="2014-01-03T16:33:51Z"
 content="""
Jim,

I have nothing in terms of a solution, but wanted to say thanks for writing this up.  This describes precisely the use case I am also seeking on my Android phone.

-- Chris
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnBJ6Dv1glxzzi4qIzGFNa6F-mfHIvv9Ck"
 nickname="Jim"
 subject="Metadata?"
 date="2014-02-21T18:44:10Z"
 content="""
I suspect the new [metadata](http://git-annex.branchable.com/design/metadata/) support could be used to help implement this.
"""]]


code: Is it possible to convert a regular git annex repo (git clone then git annex init in the folder), to an rsync remote. I have an annex with alot of remotes which makes the sync operation take a really long time. I would like to convert some of those remotes to rsync. This particular repo has a TB of data so I would like to avoid dropping content from the remote than re download everything.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.172"
 subject="comment 1"
 date="2014-02-23T18:57:42Z"
 content="""
This is doable. It works best if the remote repo is a bare git repository, because then the filenames line up 100% with the filenames used in a rsync special remote. If the git repo is not bare, the rsync special remote will first try the paths it expects, and only then fall back to the right paths, so a little extra work done. (If this became a big problem, it would not be infesable to move the files around with a script.)

Anyway, if it's a bare repo, then repo.git/annex/objects is where you want to point the rsync special remote at. With a non-bare repo, repo/.git/annex/objects/ is the location. I'd recommend moving the objects directory out to a new location, and pointing the rsyncurl at that. This way, there's no possibility of git-annex thinking one files accessed 2 ways is 2 copies.

Of course, you can't use encryption for the rsync special remote.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.172"
 subject="comment 2"
 date="2014-02-23T19:07:59Z"
 content="""
However, if the only problem is that pushing and pulling with a git repository makes `git annex sync` take too long, another option is setting `git config remote.$foo.annex-sync false`. You can still then use git-annex commands to get and push data to the remote, and can even `git annex sync $foo` from time to time, but it won't slow down the normal `git annex sync`.

However, this also prevents the assistant from uploading new files to the remote automatically.
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/FHnTlSBo1eCGJRwueeKeB6.RCaPbGMPr5jxx8A--#ce0d8"
 nickname="Hamza"
 subject="comment 3"
 date="2014-02-23T19:39:28Z"
 content="""
Thanks for the reply, just to make sure I got you right,

It is indeed a non bare git repo. So I will move the folder repo/.git/annex/objects/ to repo/

then run,

git annex initremote myrsync type=rsync rsyncurl=ssh.example.com:~/repo

and enable the remote on other annexes (disks are connected to an ssh server there is no encryption setup right now so I do not mind not having it.). And everything should be setup correctly.
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/FHnTlSBo1eCGJRwueeKeB6.RCaPbGMPr5jxx8A--#ce0d8"
 nickname="Hamza"
 subject="comment 4"
 date="2014-02-23T19:45:09Z"
 content="""
and as a follow up do I have rename the repos or can I reuse the same names for the repos?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.172"
 subject="comment 5"
 date="2014-02-24T19:07:33Z"
 content="""
That looks all-right, although initremote will ask you to tell it what encryption to use, and you'll need to specify `encryption=none`

One thing I forgot to mention is that the UUID of the new rsync repository won't be the same, so git-annex won't know about the files in there. This can be fixed by `git annex fsck --fast --from myrsync`. Which doesn't re-download all the files, but you still may want to run it on a repository close to or on the server for speed.

You can re-use the name you're currently using for the git remote for the new rsync special remote if you like.
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/FHnTlSBo1eCGJRwueeKeB6.RCaPbGMPr5jxx8A--#ce0d8"
 nickname="Hamza"
 subject="comment 6"
 date="2014-02-25T09:34:16Z"
 content="""
assuming the remote I am converting is called some-repo should mark it as dead before converting and reinitting as rsync some-repo again?
"""]]


code:
One of my repositories got corrupted. I am not exactly sure how it happened (was running a series of commands) but I think I accidentally ran regular mv instead of git mv. To fix it I deleted the moved file then checkout the original link however this did not fixed the problem. I ended up with a corrupted repo. Now running any command ends with the following error,

    ga sync
    (merging origin/git-annex origin/synced/git-annex into git-annex...)
    (Recording state in git...)
    error: invalid object 040000 6ad564920e3d78d31c9456f5be3869a0319f9f08 for'3fd/d44'
    fatal: git-write-tree: error building trees
    git-annex: failed to read sha from git write-tree

Was wondering how to fix this? I did run git fsck and git annex fsck but non fixed the problem.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.191"
 subject="comment 1"
 date="2014-04-17T19:47:54Z"
 content="""
git is complaining about one of the files in `.git/objects` being missing or corrupt.

It's not likely that some accidental command you ran caused this. More likely a disk error or an unclean shutdown could have left the repository in this state.

You can try running `git annex repair` which should be able to repair git repositories with this kind of damage.

Or you can just `git clone` the repository from any other place you have it and move over .git/annex/objects and .git/config to the new repository, discarding the damaged one.
"""]]


code:
Hey, I hope somebody can help me:

I recently moved a bunch of files to another location shortly after I renamed them. The assistant was running (repo: indirect mode) and I guess the assistant was confused about whether to commit the rename changes or to fix the symlinks because of the location move (and to commit these changes). I don't know much about the internal design, but after that my repository was kinda broken. Every git command failed and also "git fsck" gave me some error messages. However, I mounted my external drive (backup repo) and tried a "git annex repair". It took quite a while, but it could fix the repo....somehow. This is the current status:

    ~/private [master●] » git annex info
    repository mode: indirect
    trusted repositories: error: refs/remotes/extern/git-annex does not point to a valid object!
    error: refs/remotes/extern/git-annex does not point to a valid object!
    error: unable to resolve reference refs/heads/git-annex: Datei oder Verzeichnis nicht gefunden // (english: No such file or directory)
    fatal: Cannot lock the ref 'refs/heads/git-annex'.
    git-annex: git [Param "update-ref",Param "refs/heads/git-annex",Param "ea4d4b5e09bfd10c714c24ca76ab5af0625bb6a7"] failed

    ~/private [master●] » git fsck
    Checking object directories: 100% (256/256), done.
    Checking objects: 100% (654755/654755), done.
    error: refs/remotes/extern/git-annex does not point to a valid object!
    error: bad ref for refs/heads/git-annex
    Checking connectivity: 654760, done.
    dangling blob cb2a012e65b15baf0bc51cb8ac6ab05c4ac4e543
    dangling blob 6a48216d54be6fbf0539c86501ce9e4567ebf678
    dangling blob 3f8f21304c6dcfc6ace92093425d7f057138b5e8
    dangling blob 4943840264ad7c06071f510261151c28b4fb5168
    dangling blob d67e845b06b17c3ba5b44a454d3128dea6ebe8cc
    dangling blob 38b7e4ce06a84c03b9a0f7f3a84872b2ed202421
    dangling blob ca0a65e010c69e02953d63f33c5f2030f793c8ff
    dangling blob aeba65035f566ec9f18ad2ba71b155b0efd2193a
    dangling blob d3cea542db4c2904f5c8c6dd98ca3868074e6d36
    dangling commit fe1026edee5dcc6306f04ebe1a7762b86678db8a
    [... some more dangling objects ...]

However, git annex repair does not want to repair my repo a second time:

    ~/private [master●] » git annex repair
    Running git fsck ...
    No problems found.
    ok
My directory seems to be fine, also the data (symlinks) I renamed & moved are also fine. Just the "git-annex" branch seems to be broken now and I don't know how to fix that.

Cheers,
Stephan


code:
[[!comment format=mdwn
 username="sts"
 ip="134.147.73.126"
 subject="comment 1"
 date="2014-07-18T09:46:01Z"
 content="""
I forgot, this might be interesting

    git show ea4d4b5e09bfd10c714c24ca76ab5af0625bb6a7

    commit ea4d4b5e09bfd10c714c24ca76ab5af0625bb6a7
    Author: Stephan Schindel <sts@tp1.rub.de>
    Date:   Fri Jul 18 11:28:07 2014 +0200

        branch created

and

    ~/private [master●] » git annex fsck
    error: refs/remotes/extern/git-annex does not point to a valid object!
    error: refs/remotes/extern/git-annex does not point to a valid object!
    error: unable to resolve reference refs/heads/git-annex: Datei oder Verzeichnis nicht gefunden
    fatal: Cannot lock the ref 'refs/heads/git-annex'.
    git-annex: git [Param \"update-ref\",Param \"refs/heads/git-annex\",Param \"199fbe3e7ca1dbce99ffa04e440209eb93446374\"] failed

and

    git show 199fbe3e7ca1dbce99ffa04e440209eb93446374

    commit 199fbe3e7ca1dbce99ffa04e440209eb93446374
    Author: Stephan Schindel <sts@tp1.rub.de>
    Date:   Fri Jul 18 11:46:29 2014 +0200

        branch created

"""]]


code:
[[!comment format=mdwn
 username="sts"
 ip="134.147.73.126"
 subject="comment 2"
 date="2014-07-18T11:17:49Z"
 content="""
Ok, I could fix this issue by doing a fresh clone of one of my remotes:

    git clone path/to/remote
    [move objects folder from broken repo .git/annex/objects to new clone]
    git annex info (look for UUID or DESC)
    git annex reinit \"DESC\"
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 3"
 date="2014-07-21T22:57:32Z"
 content="""
There was a bug in the repair code that could leave the repair incomplete like this; I've fixed it.

> I recently moved a bunch of files to another location shortly after I renamed them. The assistant was running (repo: indirect mode) and I guess the assistant was confused

No, I don't think that this was caused by the assistant somehow getting confused. Your git repository was missing objects that had been previously committed to git. The only way this can happen is if your computer lost data. Most commonly due to crashing or being powered off or the drive the repository is in being removed while git is in the middle of committing changes.
"""]]


code: I tried to compile the assitant branch on Ubuntu 12.04. But i depends on the DBus libraryw hich does not compile with some glibberish errors. Is there a way to solve this?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.246.119"
 subject="comment 1"
 date="2012-08-25T13:06:31Z"
 content="""
Hmm, let's see...

If the gibberish error is ouyay orgotfay otay otay elltay emay utwhay ethay
roreay asway, then we can figure it out, surely..

If the gibberish error looks something like Ḩ̶̞̗̓ͯ̅͒ͪͫe̢̢̳̦͔̬ͦ̊ͭͭͤͣ̂ͅ ̣̘̹̄̕͢Ç̛͈͔̹̮̗͈͓̞ͨ͂͑ͅo̧̹̗̪͇̫̿ͥͮ̿͢m̷̢̞̙͑̊̔ͧ̍ͩ̇̚ę̜͑̀͝s̖̱̝̩̞̻͐͂̐́̂̇̆͂

.. your use of cabal
has accidentually summoned Cthulu! Back slowly away from the monitor!

Otherwise, you might try installing the `libdbus-1-dev` package with apt,
which might make cabal install the haskell dbus bindings successfully. Or
you could just install the `libghc-dbus-dev` package, which contains the
necessary haskell library pre-built. But I don't know if it's in Ubuntu
12.04; it only seems to be available in quantal
<http://packages.ubuntu.com/search?keywords=libghc-dbus-dev>

Or you could even build it with the Makefile, rather than using cabal.
The Makefile has a `-DWITH_DBUS` setting in it that can be removed to build
the fallback mode that doesn't use dbus.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.246.119"
 subject="comment 2"
 date="2012-08-25T13:11:37Z"
 content="""
I fnordgot to mention, cabal can be configured to not build with dbus too. The relevant incantation is:

cabal install git-annex --flags=\"-Dbus\"
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmU_2tE75oyG0h2ZPN4lcroIKEMC8G-otE"
 nickname="Michael"
 subject="Shame on me..."
 date="2012-08-25T15:43:19Z"
 content="""
The build error is:

$ cabal install DBus
Resolving dependencies...
Configuring DBus-0.4...
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for DBUS... yes
configure: creating ./config.status
config.status: creating DBus.buildinfo
Building DBus-0.4...
Preprocessing library DBus-0.4...

DBus/Message.hsc:1:14:
    Warning: -XPatternSignatures is deprecated: use -XScopedTypeVariables or pragma {-# LANGUAGE ScopedTypeVariables #-} instead

DBus/Message.hsc:2:12:
    Warning: -fglasgow-exts is deprecated: Use individual extensions instead
[1 of 5] Compiling DBus.Shared      ( dist/build/DBus/Shared.hs, dist/build/DBus/Shared.o )
[2 of 5] Compiling DBus             ( dist/build/DBus.hs, dist/build/DBus.o )

DBus.hsc:26:49:
    Warning: In the use of `mkTyCon'
             (imported from Data.Typeable):
             Deprecated: \"either derive Typeable, or use mkTyCon3 instead\"
[3 of 5] Compiling DBus.Internal    ( dist/build/DBus/Internal.hs, dist/build/DBus/Internal.o )

DBus/Internal.hsc:12:27:
    Module `Control.Exception' does not export `throwDyn'
cabal: Error: some packages failed to install:
DBus-0.4 failed during the building phase. The exception was:
ExitFailure 1

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.246.119"
 subject="in the language of the ancients, small details matter"
 date="2012-08-25T15:50:47Z"
 content="""
It seems that to cabal, \"DBus\" and \"dbus\" are different packages, and it seems you should install the latter, which has a current version of 0.10 which works and not some old 0.4 version.
"""]]


code:
After testing around git-annex on my mac using ntfs-3g to access the removable harddrive, I conclude that this is very dangerous.

Here are some example error snippets:

    add man/man1/git-tar-tree.1
    git-annex: /Volumes/SeagateExpansion/annex/testdirect/.git/annex/objects/0K/JJ/SHA256E-s3805--948fd508a364e036ec28f5287d11825062aea2c0ed86a3a89ed871edc6ddc7a8.1/: openTempFile: interrupted (Interrupted system call)
    failed

    add man/man1/git-bundle.1
    git-annex: /Volumes/SeagateExpansion/annex/testdirect/.git/annex/tmp/e3f_804_SHA256E-s8739--569a4b4fe1072e17ff0aa9a2f67d0130b9ec3ed23f96c5a4ea38ada82f50b4aa.1.log: rename: does not exist (No such file or directory)
    failed

    add git-core/git-completion.bash failed
    add man/man7/gitglossary.7 failed
    (Recording state in git...)

    git-annex: /Volumes/SeagateExpansion/annex/testdirect/.git/annex/journal/.fuse_hidden00005dd50000010a: removeLink: does not exist (No such file or directory)
    failed
    git-annex: add: 3 failed

It's interesting that after each git-annex add ., it would got fewer failures. But in the end it stuck at 2 files. After 60 times doing: git-annex add .

It went from this:

    git-annex: add: 248 failed

to this:

    git-annex: add: 2 failed

Now the repository has 246 objects, let's try to convert it into an indirect one:

    git-annex indirect
    ...
    git-annex info
    ...
    annexed files in working tree: 9

Only 9 files made into the indirect repository! Remaining files stay as original state. Now it would be a nightmare to try to recover form this mess.

I discourage anyone from using ntfs-3g on mac for annexing. Especially direct & indirect mode. Bare repo seems to be fine however.


code:
The "git annex add" command adds new file to the annex. However ".DS_Store" files are ignored by git-annex. Is there a list of files that are being ignored?

Maybe sometimes it's useful to add .DS_Store extended attribute data to the annex to ensure a complete sync of Mac files...


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 subject="comment 1"
 date="2012-08-02T23:29:38Z"
 content="""
git-annex ignores whatever git does. See `git-ls-files` and the gitignore configuration.
"""]]


code:
Hi,
May I know, how can I debug git-annex code.
I am new to Haskell Platform, I would like to know which IDE can be used to debug haskell code.
Thank You.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 subject="This is not an easy question to answer..."
 date="2012-06-04T19:49:46Z"
 content="""
Do you have a bug in git-annex that you need fixed, or are you just curious?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlYu7QmD7wrbHWkoxuriaA9XcijM-g5vrQ"
 nickname="Royal"
 subject="comment 2"
 date="2012-06-05T17:19:16Z"
 content="""
Hi,

I want to replace rsync with aspera-rsync. Whenever there is file transfer between 2 repositories which are in two different hosts, git-annex will use rsync protocol. I am trying to replace that rsync call with aspera-rsync so that transfer can be more faster. Since I am new to Haskell I am finding difficulties to understand the flow of execution. Is there any way I can debug so that I can get the flow?

Thanks
"""]]


code: I've had to change my default `text/html .html` handler from a text editor to a browser to support the opening sequence of `git annex webapp`. any other way around this? perhaps to have it open the page in the default browser rather than the default text/html handler?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.252.11.120"
 subject="comment 1"
 date="2012-11-13T17:25:07Z"
 content="""
Not without exposing the secret token to other users in the system by passing it as a parameter to the browser command. Which is why it uses the method it does.

But, you can configure it to use a specific browser in the standard git way:

git config web.browser chromium
"""]]


code:
After moving some files (about 1G, some big and some small files) in and out the annex, I noticed that the size of the repository has grown quite a bit. My empty repository now is over 100 MB (even after  "git annex dropunused ..." and "git gc").

Most of this size is not Git metadata but many small files in the git-annex branch that seem to hold information about files I deleted (even in the other known repositories).

So is there a way to get rid of these useless but space consuming information?

(Maybe there is no (elegant) way to remove the symlink versions from the Git history (is there?), but it would already be nice if those small git-annex metadata files could be removed)


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 subject="comment 1"
 date="2012-08-07T20:06:26Z"
 content="""
You must have quite a number of files and/or location tracking churn to get the branch that large.

Removing location tracking files for vanished files from the git-annex branch would not save much space.. Only a very little bit of tree object size going forward. The way the git-annex branch is merged does not really allow deleting files from it. And of course git uses space for even deleted files.

It'd be possible to delete the git-annex branch entirely, first backing up git-annex:uuid.log  git-annex:remote.log and git-annex:trust.log and then making a new, clean branch that those files are added back to. Then run git annex fsck on every clone to repopulate the location tracking info for files that still exit. (Note that this would also lose urls stored by `git-annex addurl`.)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 2"
 date="2012-08-08T14:54:42Z"
 content="""
I'm sort of getting into the same issue as I run 'git-annex get .' and 'git annex sync' with one of the annexes that I have on a nightly basis to collect data for back up reasons. But it's good to know there is a work around for it.
"""]]


code:
Does git-annex (safely) handle detached work trees?

That is, in git I can set `GIT_WORK_TREE=/dir/A` and `GIT_DIR=/dir/B` in the environment and have all my .git stuff in /dir/B and all my files in /dir/A.

I can see this coming in useful for a few situations, but in particular for difficult file systems - like SMB or old implementations of NFS.

In my particular case I have a Drobo (something like a proprietary NAS).  The Drobo is linux based, but by default mounts as a samba share or if you install `unfsd` it can be mounted via NFS.  Unfortunately, the nfs is v3 and doesn't allow locks, so git-annex barfs.  :-(

What I'd like to be able to do is have a direct mode annex on the drobo, with the git directory sitting on one of my linux machines.  That machine would be the only one that would directly access the drobo data as an annex but other systems that look at the drobo would see what looks like a normal directory structure; for example my media centre - `mythtv` naturally! - would see "normal" names for my music collection, not SHA256 hashes...

I guess there would be an issue if there were different `GIT_DIR`s pointing to the one `GIT_WORK_TREE`, but that is a caveat emptor IMHO.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlRThEwuPnr8_bcuuCTQ0rQd3w6AfeMiLY"
 nickname="Alex"
 subject="comment 10"
 date="2013-02-24T12:04:21Z"
 content="""
Been using it a few hours now and it works like a charm - first thing I did was get a sitemap of the LCA and FOSDEM talks and use addurl --fast on them, since I've been wanting to do that since I saw the lightning talk last year at FOSDEM.

Thanks again for making this awesome utility!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.238"
 subject="comment 1"
 date="2013-01-14T16:32:51Z"
 content="""
Recent versions of git-annex support these variables. I haven't tested it extensively, but AFAIK it works.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlRThEwuPnr8_bcuuCTQ0rQd3w6AfeMiLY"
 nickname="Alex"
 subject="Having issues here"
 date="2013-02-23T14:44:15Z"
 content="""
I'm trying to use git-annex with vcsh, and it doesn't seem to be respecting the GIT_WORK_TREE and GIT_DIR variables vcsh sets. If I try it manually, the same happens.

Any time I try to invoke a git-annex subcommand (init, add) via vcsh (vcsh $repo annex init/add/etc) it spits out an error that \"git-annex: Not in a git repository.\"

I'd really like this to work, so that I can manage the bigger files in my home with topic-based annexes alongside plain-git repos for dotfiles.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.253.75"
 subject="comment 3"
 date="2013-02-23T15:01:57Z"
 content="""
@Alex, you forgot to say what version of git-annex you are having trouble with.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlRThEwuPnr8_bcuuCTQ0rQd3w6AfeMiLY"
 nickname="Alex"
 subject="comment 4"
 date="2013-02-23T15:07:52Z"
 content="""
3.20130216.1 from Hackage - I packaged it for Exherbo specifically to try this, and installed it about two hours ago.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.253.75"
 subject="comment 5"
 date="2013-02-23T16:44:48Z"
 content="""
So, I found and fixed that bug. It really seems to be working now. Although I've thought that at least twice before!

There is an important caveat though, with using `GIT_DIR` and/or `GIT_WORK_TREE`.

git-annex needs to check in symlinks that point at the git repository. If using `GIT_DIR`, those symlinks do not look like \"-> .git/annex/objects/\"; they instead point off to some git repository elsewhere, as a relative path. For example they could look like \"-> ../gitrepo/annex/objects/\". Similar when using `GIT_WORK_TREE`.

But this is a problem if you want to use the same git repo elsewhere, with a git work tree and repo that are not set up the same relative to one-another, because the links will all be wrong. So, if you decide to use `GIT_DIR` or `GIT_WORK_TREE` with git-annex, any clone of the repository will need to place the git directory in the same place relative to the working tree.

If you later want to change the relative paths between git directory and work tree, it would need to be changed in all clones of the repository, and then you could use `git annex fix` to update the symlinks.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlRThEwuPnr8_bcuuCTQ0rQd3w6AfeMiLY"
 nickname="Alex"
 subject="comment 6"
 date="2013-02-23T21:23:18Z"
 content="""
Cool, thanks! The caveat shouldn't be much of a problem for my use case - with vcsh, the workdir is always $HOME, and I'm using the recommended mr-based workflow so the gitdir will always be ~/.config/vcsh/repo.d/$name.git in each clone.

EDIT: fix erroneous path
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlRThEwuPnr8_bcuuCTQ0rQd3w6AfeMiLY"
 nickname="Alex"
 subject="Further possible issues"
 date="2013-02-23T23:17:43Z"
 content="""
Mm, looking at the actual code changes it may still not work with vcsh - since multiple GIT_DIRS share one GIT_WORKTREE, some symlinks will point to ~/.config/vcsh/repo.d/foo.git and some will point to ~/.config/vcsh/repo.d/bar.git. When git annex is invoked with GIT_DIR='~/.config/vcsh/repo.d/bar.git', with the changes you made files annexed by ~/.config/vcsh/repo.d/foo.git will still match isLinkToAnnex.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.210"
 subject="comment 8"
 date="2013-02-23T23:23:01Z"
 content="""
That should not be a problem, because git-annex only acts on files that `git ls-files` lists.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlRThEwuPnr8_bcuuCTQ0rQd3w6AfeMiLY"
 nickname="Alex"
 subject="comment 9"
 date="2013-02-24T00:52:52Z"
 content="""
Alright then, sounds like it should be perfect! Thanks for the quick response!
"""]]


code:
I'm starting to experiment with git annex.  I'd like to use it for a centralized git repo that will be checked out often, but the clones will rarely need some large binary files (used for testing).  Therefore, I've set up a centralized/bare git repo and a clone of that repo using the instructions at [centralized_git_repository_tutorial](http://git-annex.branchable.com/tips/centralized_git_repository_tutorial/) and [bare_repositories](http://git-annex.branchable.com/bare_repositories/).  I've added some files to the annex in the clone.

I'm struggling to understand the difference between copy, move, and get.  Here's a sequence of commands:

    >> git annex add shared/1bel.maegz
    >> git commit -m "added first file"
    >> git push
    >> git annex move shared/1bel.maegz --to origin
    ## Now it no longer exists in my local repo
    >> git annex get shared/1bel.maegz
    fails.
    >> git annex get shared/1bel.maegz --from origin
    fails.
    >> git annex copy shared/1bel.maegz --from origin
    fails.
    >> git annex move shared/1bel.maegz --from origin
    succeeds! Now I have the file in my clone.

Each failure message is:

    fatal: Could not switch to '../.git/annex/objects/W8/gZ/SHA256-s99196--62874e9b58e652c9c01e796c2bf38b2234a80e0cef95c185bb7f0857d9765df2': No such file or directory
    git-annex: <file descriptor: 6>: hGetLine: end of file

How are copy, move, and get different? Which one *should* I be using to move my large data into the central (bare) repo?  Will it then be available to other clones?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-03-09T02:00:11Z"
 content="""
There is no particular reason to use any of copy, move, or get. Use which ever command makes sense at the time.

The problem you're encoutering is that you have an old, and broken version of git-annex installed. Upgrade and the error message will go away and both get and copy, in your example, will work.
"""]]


code:
Is there likely to be any problem in pointing different annexes to the same special remote (i.e. rsync/box.com/etc.) ?

As the objects are stored based on their SHA256 key the expectation is that the chance of collision is is small.

The only problem I can foresee is where the same content is stored in more than one annex and it is deleted in the remote in one annex, but not the other - there won't be any protection against that, but for non-overlapping content this risk should be negligible.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.210"
 subject="comment 1"
 date="2013-02-24T21:51:19Z"
 content="""
You're basically right. Additionally, if you use encryption, the keys used on the remote are themselves encrypted and so even the same content will result in different keys.

With this said, it's easy to use a subdirectory of a rsync or box.com remote, and I'd rather recommend doing that, rather than jumbling multiple repositories data into a single directory.
"""]]


code:
I have a NAS at home which I access both as AFP/SMB shares and thru ssh/rsync.  Now, I'd like to keep on using the shares as before, i.e., browsable via AFP/SMB with conventional file names.  Ideally, I'd also like to git-annex some NAS shares, preferably, in direct mode. However, it seems out of the question to install git-annex on the NAS (hopefully, I'm wrong about this in the long run).

Two **non-special remote setups** would be:

1. Mount the share and turn it into a direct mode git annex repo.  Does anybody have experience with this?  I'd suspect this to be very inefficient due to the use of all the files in .git over AFP/SMB.  Configuration as a WORM backend seems to be advised? (Edit: Well, I just tried this and 'git annex init' failed as described in this [forum post](http://git-annex.branchable.com/forum/Can__39__t_init_git_annex/).  So AFP/SMB seem to be non-starters. :-(  Furthermore, AFP/SMB are immediately detected as crippled and set to direct mode automatically.)
2. Same as 1. but with a local GIT_DIR.  This should work by having .git on the NAS link to the local GIT_DIR.

Alternatively, I could treat my NAS as a [**web special remote**](http://git-annex.branchable.com/tips/using_the_web_as_a_special_remote/).  Some URL schemes come to mind:

* **file:**  This would benefit from some wish list items ([recursive directory remote setup/addurl](http://git-annex.branchable.com/todo/wishlist:_recursive_directory_remote_setup__47__addurl/)).
* **rsync:** AFAIK not implemented (yet?) as an option for web special remotes.
* **sftp:** (Might also include ssh access.)

The problem is that the "web semantics" don't really work in my use case:

* Files might change/move on the NAS.
* I'd like changes (e.g., renamed files) in my local repo to propagate to the NAS.  Currently, git-annex would use git push for this purpose IIUC, however that's not available on the NAS...
* the web semantics seem to imply that there is exactly one "web" repository (and the URL is fixed)

All of these indicate a mismatch between my use case and web special remotes.

Hence my question: **Would something like a "direct special remote" make sense?**

As a starting point I'd look at a setup similar to 2. above, i.e., a remote "working copy" with local GIT_DIR.  Except that instead of a whole local .git directory a branch in an existing .git dir might be more appropriate...

--Chris


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmVICFY2CDP08xdsPr3cgmScomy9HA-1sk"
 nickname="Andrew"
 subject="comment 1"
 date="2013-04-04T00:33:15Z"
 content="""
\"Dumb NAS in direct mode, without git-annex installed\" would greatly expand the possible use cases IMO - it would transform git-annex from a distributed filesystem to a general sync tool. Or maybe I'm dreaming.
"""]]


code:
[[!comment format=mdwn
 username="https://pradermecker.myopenid.com/"
 ip="81.244.179.73"
 subject="Local rsync ssh remote (I am confused)"
 date="2013-04-13T17:54:06Z"
 content="""
If I understand this right, it is not possible to use git-annex as a simple wrapper around rsync.

My use case is this: I have a music folder on a plug computer running debian squeeze. The plug computer acts as a streaming music server. My wish is to create a local ssh remote to have my laptop git-annex music repo in sync with the plug computer. So ideally I don't want to install git-annex on the plug computer (but git is there so a git bare repo can be used).

According to the UI, if I use \"Pairing with a local computer\", both computers needs to have git-annex installed. If I try \"Remote server\" either I need both machines git-annex aware or I need the content of the plug computer to be encrypted (which is obviously not what I want in this case).

Is the use case supported by the UI ? Should I try with the CLI ?

Thanks for your help

"""]]


code:
[[!comment format=mdwn
 username="https://pradermecker.myopenid.com/"
 ip="81.244.190.181"
 subject="comment 4"
 date="2013-04-14T09:01:27Z"
 content="""
Thanks !

Before trying the CLI, I will install have a go with installing git-annex on the plug computer.

Do you know by any chance if version 3.20120629~bpo60+2 (the available version on squeeze-backport) will be compatible with the current latest release (4.20130405) ?

I guess I am asking if version 3 is compatible with version 4.

"""]]


code:
[[!comment format=mdwn
 username="andy"
 ip="99.48.75.171"
 subject="comment 4"
 date="2013-04-14T01:07:19Z"
 content="""
@pradermecker: I think that the CLI would be able to do what you're describing (to some extent), but the issue is that the filenames it creates will be hashes, and not in the folder pattern that is on your laptop. Of course, if the plug computer uses file metadata to determine file info, and will search a directory tree, then it probably won't care that the files are named strange things.

As a quick example:

I have a repo with three files in it:

    $ ls -R
    .:
    File 1  File 2  File 3

I copy those files to a directory remote:

    $ git annex copy --to=test-dir *
    copy File 1 (to test-dir...)
    ok
    copy File 2 (to test-dir...)
    ok
    copy File 3 (to test-dir...)
    ok
    (Recording state in git...)

Then I look at what's in my directory remote:

    $ cd ../remote
    $ ls -R
    .:
    0d4  d35  f15

    ./0d4:
    d37

    ./0d4/d37:
    SHA256-s7--a940d8aadc02f798331b2d46f1a8ad2c9821783060f4a0810da42bf785855c1c

    ./0d4/d37/SHA256-s7--a940d8aadc02f798331b2d46f1a8ad2c9821783060f4a0810da42bf785855c1c:
    SHA256-s7--a940d8aadc02f798331b2d46f1a8ad2c9821783060f4a0810da42bf785855c1c

    ./d35:
    55c

    ./d35/55c:
    SHA256-s7--ab1ad6a49c022416008887464b8dc03b523b9e81530cf47d1f6f7712c1b30955

    ./d35/55c/SHA256-s7--ab1ad6a49c022416008887464b8dc03b523b9e81530cf47d1f6f7712c1b30955:
    SHA256-s7--ab1ad6a49c022416008887464b8dc03b523b9e81530cf47d1f6f7712c1b30955

    ./f15:
    3f2

    ./f15/3f2:
    SHA256-s7--d1993f1115215aa6389e33fa9979fb39bb29e5bed14661baf1cf6af3182f0164

    ./f15/3f2/SHA256-s7--d1993f1115215aa6389e33fa9979fb39bb29e5bed14661baf1cf6af3182f0164:
    SHA256-s7--d1993f1115215aa6389e33fa9979fb39bb29e5bed14661baf1cf6af3182f0164

I haven't checked, but I think that the assistant will be able to use a special remote that you create from the CLI once you create it.

To create a remote without encryption using the CLI, try a command of the form `git annex initremote remote-name blah blah blah encryption=none`
Also see the information at [[special_remotes/directory]], [[special_remotes/rsync]], and [[Repo accessible from \"dumb\" client without git-annex]].
"""]]


code:
[[!comment format=mdwn
 username="andy"
 ip="99.48.75.171"
 subject="comment 4"
 date="2013-04-16T07:57:05Z"
 content="""
Not I. Sorry. You might want to make a forum post about this, if you haven't already: [[forum]]
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 6"
 date="2013-04-16T17:03:36Z"
 content="""
Version 3.2012* is compatible with the current version. You can install that version anywhere you don't need the full assistant.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkI9AR8BqG4RPw_Ov2lnDCJWMuM6WMRobQ"
 nickname="Dav"
 subject="Curious to know if there's progress on this"
 date="2013-12-09T00:49:45Z"
 content="""
The above scenario is also quite interesting to me. Similarly running and old OS (Debian 4.x) on a NAS, and it would be great to be able to access files directly (via direct mode) and manage via git annex, along with a true backup elsewhere. Mostly commenting so I'll get emails for further discussion here.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmNu4V5fvpLlBhaCUfXXOB0MI5NXwh8SkU"
 nickname="Adam"
 subject="comment 1"
 date="2014-05-08T16:56:34Z"
 content="""
I recommend rewriting your comment.  Use paragraphs and be more clear and specific.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="216.145.95.162"
 subject="comment 2"
 date="2014-05-17T17:00:19Z"
 content="""
git-annex doesn't allow bonding repositories like that, but it does have repository groups which can accomplish the same thing.

For example, if you put B C and D into the archive group, and set the [[preferred_content]] expression to \"standard\", then each repository will only want files that have not yet reached some other archive repository.

Or, you can make up your own group name, like \"BCD\" and set the preferred content of each repository to something like \"not (copies=BCD:2)\"  -- and now it will try to have 2 copies of each file on one of the drives.

Once it's configured, using either the git-annex assistant, or `git annex sync --content` will copy files around according to the configuration.

This will tend to fill up the first drive, or the first 2 drives, and only use subsequent drives if files don't fit on the first drive. You can use [[cost]] settings to control which drives files are put on.
"""]]


code: Do both friends need to be online in order for Jabber syncing to work? Or will the pushed changes be stored on the Jabber server, so that when the remote machine does come online it can pull them even if the original machine is now offline?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-02-27T17:54:47Z"
 content="""
Both friends need to be online at the same time to sync with Jabber.

There are some extensions that store and forward jabber messages, but:

* They are probably not widely available.
* AFAIK they only forward chat messages. But we try not to use visible messages, because that would prevent
  reusing your jabber account with git-annex.
* Most importantly, the git protocol lets the sender and receiver communicate with each other to work out what data to transfer. If one if offline, the best that could be done is to upload the whole git repository!

If you want the ability for off-line clients to sync up, you should add a shared bare git repository.
"""]]


code:
[[!comment format=mdwn
 username="annexuser"
 ip="24.16.193.140"
 subject="comment 2"
 date="2013-03-11T19:40:30Z"
 content="""
How would I add a shared bare git repository in the web app?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 3"
 date="2013-03-11T19:45:34Z"
 content="""
That's done by the \"Remote server\" option on the Repositories page.

You need to have git-annex (any old version will do) installed on the server for this to work.

Of course, a regular git remote will do as well, although the assistant does not set this up. You could even use github for that remote, or a similar service; any remote added with `git remote add` will be used by the assistant.
"""]]


code:
Hi

I am trying to understand how git-annex works. Does it version big files at all?

thanks


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.27"
 subject="comment 1"
 date="2012-11-09T18:56:59Z"
 content="""
Yes. Have you read [[How_It_Works]]?
"""]]


code:
[[!comment format=mdwn
 username="karhun"
 ip="99.20.248.32"
 subject="comment 2"
 date="2012-11-09T21:37:19Z"
 content="""
Hi

Thanks for the pointer.

The reason I asked about is that it says that \"git-annex allows managing files with git, without checking the file contents into git. \". So I was just trying to understand the mechanic behind the description. If it is not check into the git then probably it is not versioning the files. I assumed that it must be  merely a sync tool.

"""]]


code:
[[!comment format=mdwn
 username="karhun"
 ip="99.20.248.32"
 subject="comment 3"
 date="2012-11-12T05:58:16Z"
 content="""
Hi

I have one more follow up question.

So git-annex versions files as I understand. Is it possible to use it without file versioning and just use it as a sync tool. I realize that there are example scenarios to demonstrate it however I would like to disable versioning if possible because I am intending to use it on large folders with big binary type files.

thanks

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.27"
 subject="comment 4"
 date="2012-11-12T15:17:53Z"
 content="""
Let's suppose you delete a file from git, and sync the deletion. Later, you can run `git annex unused` and it will find content not needed by any branch of the git repository. And if you choose, you can use `git annex dropunused` to delete that content.
"""]]


code:
Out of simple curiosity, how does 'git annex migrate' work?  I'm mostly wondering how file integrity is ensured.

Let's say you want to migrate foo.txt from (say) md5 to sha256.  Does git annex simply sha256sum foo.txt and rename it in the .git/objects folder to the new sum?  Or does it md5sum foo.txt, verify that it's the not corrupt, then sha256sum it and rename to the new sum?

You could run into problems if you migrate without first verifying; if the file is corrupt and you simply sha256sum and rename it, then the file wouldn't seem corrupt at your next fsck.

I'm sure you've considered this during the basic design phase of git-annex, but I'd just like to be sure.  I'm kind of paranoid when it comes to data integrity. =P


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-05-13T18:28:01Z"
 content="""
I actually didn't originally consider this, but it turns out I have wonderful users, and you're at least the second to consider this case.

So the released git-annex version does already verify the checksum when doing a migration. However, as I was checking this, I noticed
I had left a window between the verification and the generation of the new key for the migration. It's pretty unlikely a file would get corrupted just as it was being migrated, but that's no excuse. I've committed a change that reverses the order; so it generates a new key and then verifies the old one.
"""]]


code:
[[!comment format=mdwn
 username="GLITTAH"
 ip="154.51.136.37"
 subject="comment 2"
 date="2013-05-14T15:54:08Z"
 content="""
An even better solution.  Thanks for your time and effort joey!
"""]]


code:
My Use Case:

I try basicly to use annex as a raid-like tool (at least thats the first step)

so I added some big files to it, and then I synced it to a usb-remote. So far all did go well...

No I startet annex watch daemon... because I thought it would then watch the files and checkin and out at least for the "origin" copy of the files.

Then I thought lets delete some files I dont need anymore... so... rm.... ^^

Maybe I just wanted to see what happens or if that would magicly do what I wanted him to do... ok I knew that it would not delete the file on the usb-drive (backup...) ok... but maybe at least localy...

Now what did happen instead of that... the links are gone yes... the files it self are in the .git objects tree, so they did not get deleted, so ok not the way I wanted... have to unlock it first, would make maybe sense...

So I tried first to get the links back... tried fix, tried unused, tried get... but the links doesnt show up again...

is there a way to first bring back the links?
is it save or the right way to just git rebase HEAD~3 to bring the links back?

and then when I want to delete files from all places whats the way to do that... annex unlock -> then delete? or git drop filex --copies=0 or something?

like I said I try to use is like a more flexible raid thing, this files are to big to really back em up with history... I watch them... and then soon I will often delete them (from everywhere). but other parts stay... I dont delete them then... ^^

So maybe I missuse annex for that usecase... try to find that out ;)


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.60"
 subject="comment 1"
 date="2012-09-16T00:14:49Z"
 content="""
You can use any git command you like like `git rebase` or `git revert` to operate on the links.

git-annex, like git, keeps data for files that have been deleted.

The `git annex unused` command will find that data and let you remove it. See [[walkthrough/unused_data]].

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn4bbuawnh-nSo9pAh8irYAcV4MQCcfdHo"
 nickname="Stefan"
 subject="thanx for fast help ;)"
 date="2012-09-16T16:09:29Z"
 content="""
git-annex version: 3.20120629

git annex unused did not show me anything... dont know why... I did only \"rm Directory\" not git rm... but \"git annex watch\" did check it in... I sadly have reverted and droped the file now... but if you want I can try to delete other files the same way to see if annex unused shows them... then I post the hole commands... k I try it now:

oh I see the directory I wanted to drop did not get dropped because I set numcopies to 2 ;) so I try it again with that file:

1. rm -r directory
2. ls it is gone (should be ;) )

user@host:~/annex/SOFTWARE$ git annex unused
unused . (checking for unused data...) (checking master...) (checking synced/master...) (checking usbdrive/master...) ok
user@host:~/annex/SOFTWARE$

Is that like it should be? If so, should unused only show stuff that I delete with git rm?

-----------------------------------

Ok I did read the unused doku you pointed me. So I thought I got it, but I did not:

If I

1. rm File
2. in the usb-repos, git pull
both repositories dont have a link to the file.
3. git unused shows nothing
4. it seemed that the link AND the File it links to got deleted:

-../.git/annex/objects/pM/vq/SHA256-s1066518528--00c5e1b1610c0c2dfab05c7a55aaad

but the file size is 1gb, both directories dont get smaller... also if I use git gc after that it does not get smaller too.

Is there a way to really delete files from both repositories.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.60"
 subject="comment 3"
 date="2012-09-16T17:08:49Z"
 content="""
> unused . (checking for unused data...) (checking master...) (checking synced
/master...) (checking usbdrive/master...) ok

It still considers a file's content used while some branch contains the file. So one of the branches it's checking there still has your file in it. Once you get all the branches synced up, the file will become unused.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn4bbuawnh-nSo9pAh8irYAcV4MQCcfdHo"
 nickname="Stefan"
 subject="comment 4"
 date="2012-09-16T22:51:00Z"
 content="""
hmm Ok I tried now something that seems to did it...  after I did checkin and pull the deletion of the soft-link of the file... I addionaly used git annex sync, and then again pushed it from the usb drive... now the unused command finds stuff...

So am I right now to make that happen:

1. delete the file in the \"main\" repos
2. git pull from the other repository(s)
3. git annex sync

4. git annex unused (now it founds something to delete)

5. git annex dropunused 1-1000 (or some other range) do that in both repositories to delete in on both... and do that maybe as cronjob (from point 2 to 5) or do the last last step only if a drive is nearly full (automaticly or manually...)

ok so maybe I could use that.

the advantages over something like rsync are...

1. I have a history and can reverse some operations (till I do step 5 ^^)
2. I am more flexible in repo/remote types
3. checksums get saved
4. I can have repo with all links but 0 data on a 3rd pc and can \"get\" stuff dynamicly (what is the cheapest transfeir) ot them... aka partial checkouts

ok I think the 4. advantage buys me ;)

hope I got it now right with the 5 step instruction.

keep up your great work, would have tried kickstarter here maybe too for other stuff... but I am no american ;) and eu-guys are not allowed, and I also have no relative in usa ;) but that only as a site note ;)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
 nickname="Justin"
 subject="comment 5"
 date="2012-09-17T14:12:13Z"
 content="""
That looks right except you shouldn't have to `git pull from the other repository(s)`.  The first thing `git annex sync` does is a `git pull`.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn4bbuawnh-nSo9pAh8irYAcV4MQCcfdHo"
 nickname="Stefan"
 subject="comment 6"
 date="2012-09-17T23:00:09Z"
 content="""
ok I now at least got it that it is nice to run watch on both repositories...

but what I dont get is, is the git annex watch commando not a automatic way so that this deamon makes sync command whenever needed? so I have to use git annex sync first on the repos where I delete the file and then again on the repos I want to receive that change... that seemed to work, but why did watch not do that for me???

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn4bbuawnh-nSo9pAh8irYAcV4MQCcfdHo"
 nickname="Stefan"
 subject="comment 7"
 date="2012-09-18T00:11:50Z"
 content="""
cd /mnt/data/annex/ &&
/usr/bin/git annex sync &&
/usr/bin/git annex copy --auto --to usbdrive &&
cd /mnt/usb1/annex/ &&
/usr/bin/git annex sync &&
echo \"synced\" ||
echo \"backup of annex failed\" | sendxmpp -t name@server.country

ok suprisingly that did not only sync the files but also did remove the file, so you dont need a pull... dont know what I did wrong the other times I tried, does maybe the  copy command also \"copies\" the delete commit?

but annyway that works for me... (for now I do the dropunused command manuelly...)

that should work as cronjob...
"""]]


code:
I just created a new Annex by doing the following:

 1. git init
 2. git annex init
 3. git annex add .
 4. git commit -m "Added files"
 5. git annex status

I see the following:

    local annex keys: 224
    local annex size: 41 gigabytes
    known annex keys: 235
    known annex size: 49 gigabytes
    bloom filter size: 16 mebibytes (0% full)
    backend usage:
            SHA256: 459

Why is there an 8 gigabyte difference here?  What/where are those files?  What is a bloom filter?


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
 nickname="Justin"
 subject="comment 1"
 date="2012-10-17T12:54:42Z"
 content="""
Those are duplicate files.

see http://git-annex.branchable.com/tips/finding_duplicate_files/ for how to easily display them.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 2"
 date="2012-10-17T17:32:52Z"
 content="""
The local keys are files whose content is locally present.
The known keys are annexed files in the current branch, whose content may or may not be present.

Justin is correct -- if you have the same file in the tree twice, it will be counted twice as known keys. Since git-annex deuplicates, only one local key is needed to store it.

The bloom filter is a technical implementation detail that allows the potentially expensive status scan to run in constant space. You can read about it on Wikipedia if interested. :)
"""]]


code:
Hi,

how can I handle *drop* with the assistant? There are some files in my annex that I don't need on my laptop. so i just *drop* them on the command line bevcause I have copies on my server and an external usb drive. But after a while the assistant pulls all the files back in. Can I avoid this and make the *drop* somehow "sticky" until i *get* them over cli?

Best, Michael


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn7gQ1zZDdWhXy9H51W2krZYShNmKL3qfM"
 nickname="Karsten"
 subject="comment 1"
 date="2013-04-11T07:32:50Z"
 content="""
Hi,

you can set your repository type to 'manual'.

Best
Karsten
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmU_2tE75oyG0h2ZPN4lcroIKEMC8G-otE"
 nickname="Michael"
 subject="comment 2"
 date="2013-04-11T08:26:01Z"
 content="""
@Karsten But then I would loose all replication features? Or can I mark subfolders as manual? Or would I mix different annexes?
"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 3"
 date="2013-04-11T13:18:04Z"
 content="""
The trivial way to do this is to set your repository to \"client\" and then when you want to drop a file, create a folder called \"archive\" and drag the file into that folder.  The assistant will drop it (if numcopies is satisfied that there are enough copies elsewhere).  If you want it back, drag it out of the archive and the assistant will re-fetch it.

This is how I use git-annex: I have an ssh remote and a usb drive which are both \"backup\" and my laptop is \"client\", and my numcopies is sent to 2.

Files are always copied to the ssh remote and the usb drive because they're \"backup.\"  That fulfills my numcopies=2.  At that point, anything in an \"archive\" folder on my laptop will be dropped automatically.

It means my annex is littered with \"archive\" folders, but it achieves the desired effect.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmU_2tE75oyG0h2ZPN4lcroIKEMC8G-otE"
 nickname="Michael"
 subject="comment 4"
 date="2013-04-11T13:39:13Z"
 content="""
A nice feature would be, to  be able to put a .annex-manual file into a subfolder and from that point the files would require manual git-annex management.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 5"
 date="2013-04-11T15:00:10Z"
 content="""
Setting your repository to manual doesn't prevent it syncing changes out to other repositories, it just prevents it from downloading the contents of files it doesn't already have.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmU_2tE75oyG0h2ZPN4lcroIKEMC8G-otE"
 nickname="Michael"
 subject="comment 6"
 date="2013-04-12T06:31:17Z"
 content="""
@joey But a *manual* node will still be the middleman between say a remote host and a local usb connected to the node?
"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 7"
 date="2013-04-12T13:30:24Z"
 content="""
As I understand it, nothing will \"be the middleman\" in the sense of, if annex A has remotes B and C, grabbing content from B in order to give it to C without regards for whether or not A wants it.  Annex A doesn't think in terms of moving content from one remote to another remote, only in terms of pulling and pushing to remotes individually.

I get the \"move content from remote B to C\" effect by setting numcopies=2 and making both remotes \"backup.\"  If content is only on remote B, it will migrate to A in order to fulfill numcopies.  Then it will migrate to C because of C's preferred content settings.

(Respecting minimum numcopies is higher priority than respecting preferred content settings, so even if A wouldn't normally \"want\" the content, it will receive it if it needs to in order to fulfill numcopies.)

I've got A set to \"client\" so content in archive directories disappears once the content is on B and C (because numcopies is fulfilled, and so A's preferred content can take effect.)

If you had A set to \"manual,\" then it wouldn't automatically drop content once it moved from B to A to C, though.  Because it would only drop things manually.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 8"
 date="2013-04-16T20:46:09Z"
 content="""
@edheil, you used to be right about the middleman. But I'm continually improving things. :)

So, consider this situation:

> A (client) --- B (client) ---- C (archive)

If a file is created on A, inside an `archive` directory, B wants a copy, since it's not archived yet. Once B gets the copy, it sends it on the C. At that point, B notices that hey, this file was archived and is in an archive directory, and so it no longer wants its copy and drops it.

(At this point A will also want to drop the file. However, it cannot! This is because git-annex requires positive, direct verification that some other repository has a file before dropping it, and A cannot talk to C to check.
However, if you set C to be trusted, this verification is bypassed, and then A will be able to drop the file as well.)

This support for middlemen is a new feature, which will be in the next release. You can get it in any recent nightly build.

Amusingly this feature was built without writing any haskell code.. just fine-tuning the preferred content expressions!

We can also consider what happens if B is set to manual. In this case, it won't automatically get the file from A. But if you manually get it, then B will send it on to C. And A will drop the file once it hears that C has it. Due to the manual mode, you'll have to manually drop it from B of course.
"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="99.54.57.201"
 subject="comment 9"
 date="2013-04-17T03:37:48Z"
 content="""
Nice!  It sounds like there would be another way to accomplish what I'm currently accomplishing by using numcopies=2 and standard client/backup directories. (backup <--> client <--> backup)

In order to get double backups, I *could* tweak the content settings for my laptop's annex, which is currently set to standard \"client\" mode, so that it's exactly like \"client\" mode except that \"archive\" directories want to drop content which is archived *twice*. (backup <--> tweaked-client <--> backup)

Then I'd get content automatically copying itself out to two archives, and dropping it when it has done so, without touching numcopies.

No reason not to keep doing it the way I'm doing it, since it's simpler, but it's cool to see the assistant getting smarter. :)

"""]]


code:
I currently use duply/duplicity to back up my networked computers to my home server. I have two external HDDs and, every week or so, I bring one of these home and copy the backup files to the hard drive (I leave them on the server to easily restore files and because I have a large hard drive in that). I use some hand-written scripts to keep a copy these backup files in the cloud (Ubuntu One) until they have been copied to both hard drives, ensuring that there are always two copies of the files somewhere offsite. Out of paranoia, I also have some "standalone backups" that are just huge encrypted archives of important folders (say, my entire Photos directory) as at a certain date - in case for some reason duplicity ever stops working or I need to roll something back to a version years earlier. I am less worried about these standalone backups and manually keep one copy of each somewhere.

It sounds like Git-Annex could automate things quite nicely (and give me some neat extras, like knowing where they were). This is how I understand I should do it, but please let me know if it is the right approach or if you have any suggestions:

1. Create a folder on my server called "annex" and make a Git-Annex "large backup" repository in there.
2. Create a folder within that called "archive" and put a "backup" folder within that. I understand that having the backups within an archive folder will mean that they aren't automatically copied to my desktop machines etc.
3. Within that "backup" folder, create two folders, one called "duplicity" and one called "standalone". Put the backups in the respective folders.
4. Set up gcrypt Git-Annex repositories on my two external HDDs as "small backups". This seems to just start copying files across. That surprised me, as the files are in the archive folder and I thought the default was numcopies=1. Is there some autosync option that I need to turn off? Ideally, I would like it to encrypt/decrypt primarily with my server GPG key (which I'm not worried about copying around my computers), but also encrypt to my personal GPG key (where I'd only put my public key on the server, but I know I will not lose the secret key for that). Am I right that to do that I would need to set the repos up manually with:

    git init --bare /mnt/externalHDD1

    git annex initremote externalHDD1 type=gcrypt gitrepo=/mnt/externalHDD1 keyid=$serverkey keyid=$personalkey

    git annex sync externalHDD1

    Or should the gitrepo be the location of my main Git-Annex repository? How do I make it sync up with my other repos?

5. I understand that I would then need to set numcopies=3 in a .gitattributes file in the "archive/backup/duplicity" directory and, say, a numcopies=2 in the "archive/backup/standalone".
6. I could then add a cloud repository as a "transfer" repository and Git-Annex should only keep files on that that are not already in the right number of places (similar to what my scripts are doing now).
7. I have recently upgraded my hard drive, so I have my old 1TB internal hard drive that I will be putting in a cupboard somewhere. I was thinking that I could make this an archive drive for things like one copy of my duplicity/standalone backups. I wouldn't want it to be the only copy of anything. If I just set it as an archive drive, would this work?
8. Are there more clever ways of doing this? I consider my external HDDs and the cloud repo as "offsite" repositories and ideally there would always be one copy of my backups offsite (in addition to at least three overall). There would also ideally be one of each of my files "live" (in most cases my server) that could instantly push files into a cloud repo and then to wherever I am. Is there any ability to put repositories in groups and write rules like that?

Any thoughts greatly appreciated!

Aaron


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 1"
 date="2013-11-07T16:42:14Z"
 content="""
This all sounds reasonable to me.

gitrepo= is the location of the repository you are setting up with gcrypt, not your existing git-annex repository.

numcopies configures the lower bound of the number of copies, not the upper bound. There's no \"small backup\" type, you must mean either small archive, or incremental backup. Either type of repository is going to want to have files that have not previously been archived, or backed up.

You might eventually want to write your own [[preferred_content]] expressions to handle offsite repositories. I'd recommend starting simple and building up.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnFjuvfPpi1kf6l54bxfFUm0Aw_Gf_IO0o"
 nickname="Aaron"
 subject="comment 2"
 date="2013-11-07T19:46:32Z"
 content="""
Thanks Joey!

> gitrepo= is the location of the repository you are setting up with gcrypt, not your existing git-annex repository.

Great, thanks. If I set these up outside of the WebApp, what is the best way to link them up to my main repositories?

> numcopies configures the lower bound of the number of copies, not the upper bound. There's no \"small backup\" type,
> you must mean either small archive, or incremental backup. Either type of repository is going to want to have files
> that have not previously been archived, or backed up.

Yes, I meant incremental backup. Thanks, understood. The page on preferred content ( http://git-annex.branchable.com/preferred_content/ ) made things a bit clearer. I didn't realise that the incremental backups etc want a copy even if it isn't necessary to satisfy numcopies. Does that mean that once it is on one backup drive, it will not be sent to the other unless I increase numcopies?

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 3"
 date="2013-11-08T17:06:02Z"
 content="""
The webapp can use remotes that you have configured at the command line just as well as it can use remotes you configure using the webapp.

Each file only gets backed up to one incremental backup repository. If you have a full backup repository it will (try to) get another copy of every single file.
"""]]


code: I am currently using rsync.net to keep my notes in sync, both git repo and the content (gpg encrypted) is stored there. Instead of creating two other repos on a server that I have which has git-annex installed where I can actually use remote daemon. Is it possible to create a repo containing both the git repo and the content encrypted from the command line?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.114"
 subject="comment 1"
 date="2014-04-27T00:31:18Z"
 content="""
Why not go all the way and encrypt the git repository stored on the remote server too? [[special_remotes/gcrypt]]

When you have git-annex-shell installed on the remote server, the remote daemon should work for gcrypt repositories as well as normal non-encrypted git repositories.
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/FHnTlSBo1eCGJRwueeKeB6.RCaPbGMPr5jxx8A--#ce0d8"
 nickname="Hamza"
 subject="comment 2"
 date="2014-04-28T09:14:12Z"
 content="""
I would love to use gcrypt and I tried it but the problem is at every sync it will ask for my pass phrase even though the key used to encrypt the repo is pass phrase less it goes through the keys in my keychain and asks me to unlock all my keys one by one.
"""]]


code:
Hello,

I hope my understanding of the git-annex webapp is correct, and if not, please correct my wrong thought-pattern.

I have 2 VPS servers and 5 computers. On the 5 computers, I want to sync one folder (with subfolders and files). All the computers are full-disk encrypted, and it is important for me that when the data leaves my computer, it stays encrypted in one way or another. I want to use the two VPS's as an encrypted ssh full backup repository.

The computers are rarely on at the same time, some are off for two weeks, others are on more frequent. However, the VPS's are always on.

Now, is it possible to set up git-annex to support this behaviour? Using an encrypted ssh remote to make sure the folder is in sync on all computers?

I've set git-annex up on one box, with the two servers as encrypted full backup remotes (without git-annex installed there), and configured my jabber account. That all works, and the files get synced.

Then I've set up a second computer, while the first one is still on. There I've added my jabber account, and created a local repository. Then I selected "Share with your other devices". It now shows my jabber account on both machines, and I saw a few syncing messages, but then there came errors, "Unable to download files from your other devices". I've attached the log at the bottom of this page.

It gives me the option to add a Cloud Repository. If I add the encrypted rsync full backup remote VPS there, will it work and keep my files in sync? is there something I'm doing wrong?

Note that the computers I've set up now are two OS X machines running Mountain Lion, and the servers are CentOS 6.

Log:

    [2013-07-15 07:44:17 CEST] main: starting assistant version 4.20130709-g18e5f43

    (scanning...) [2013-07-15 07:44:17 CEST] Watcher: Performing startup scan
    (started...) [2013-07-15 07:45:08 CEST] main: starting assistant version 4.20130709-g18e5f43

    (scanning...) [2013-07-15 07:45:08 CEST] Watcher: Performing startup scan
    (started...) [2013-07-15 07:54:31 CEST] XMPPClient: Pairing with stnl in progress
    rercrrrreveeeec:ccccv vvvv:r:::: e    rsrrrreoeeeesussssorooooucuuuurerrrrc cccceveeee a    vnvvvvaiaaaansnnnnihiiiisesssshdhhhhe eeeed(dddd C    (o((((CnCCCConoooonennnnncnnnneteeeecicccctottttiniiiio oooonrnnnn e    rsrrrreeeeeestsssse eeeetbtttt y    b bbbbypyyyy e    peppppereeeee)eeeer
    rrrr)))))

    recv: resource vanished (Connection reset by peer)
    recv:r errceevcs:vo :ur rercseeos uovruacrneci esv havenadin si(hsCehoden dn( eC(coCtnoinnoennce tcriteoisnoe ntr  erbseyes tep teb eybr y)p
    epeere)r
    )
    [2013-07-15 07:55:00 CEST] XMPPSendPack: Syncing with janwxmpp
    Already up-to-date.
    recv: resource vanished (Connection reset by peer)
    [2013-07-15 07:55:04 CEST] XMPPReceivePack: Syncing with janwxmpp
    To xmpp::janwxmpp@gmail.com
     * [new branch]      git-annex -> refs/synced/0a41c397-09e8-4957-a6f4-2a6846a4f9d8/cmVsc3RubEBnbWFpbC5jb20=/git-annex
     * [new branch]      master -> refs/synced/0a41c397-09e8-4957-a6f4-2a6846a4f9d8/cmVsc3RubEBnbWFpbC5jb20=/master
    [2013-07-15 07:55:25 CEST] XMPPSendPack: Unable to download files from your other devices.
    [2013-07-15 07:55:25 CEST] XMPPSendPack: Syncing with janwxmpp
    recv: resource vanished (Connection reset by peer)
    fatal: Could not read from remote repository.

    Please make sure you have the correct access rights
    and the repository exists.
    [2013-07-15 07:57:25 CEST] XMPPSendPack: Unable to download files from your other devices.
    recv: resource vanished (Connection reset by peer)
    recv: resource vanished (Connection reset by peer)
    recv: resource vanished (Connection reset by peer)
    recv: resource vanished (Connection reset by peer)
    recv: resource vanished (Connection reset by peer)

    (Recording state in git...)

    (scanning...) [2013-07-15 08:01:36 CEST] Watcher: Performing startup scan
    (started...) recvr:rrrrrrre eeeeeeecrcccccccvevvvvvvv:s::::::: o       rurrrrrrrereeeeeeescsssssssoeooooooou uuuuuuurvrrrrrrrcaccccccceneeeeeee i       vsvvvvvvvahaaaaaaanennnnnnnidiiiiiiis sssssssh(hhhhhhheCeeeeeeedoddddddd n       (n(((((((CeCCCCCCCocooooooontnnnnnnnninnnnnnneoeeeeeeecnccccccct tttttttiriiiiiiioeooooooonsnnnnnnn e       rtrrrrrrre eeeeeeesbssssssseyeeeeeeet ttttttt p       bebbbbbbbyeyyyyyyy r       p)pppppppe
    eeeeeeeeeeeeeeerrrrrrrr))))))))

    [2013-07-15 08:44:42 CEST] XMPPSendPack: Syncing with janwxmpp
    recv: resource vanished (Connection reset by peer)
    recrvre:ec cvrv:e: s roreuesrsocoueur rcvceae n vivasanhnieisdsh he(edCd o (n(CnCoeoncnntneiecoctnti ioronen s reretes sebetyt   bpbyey e prpe)ee
    err))

    fatal: Could not read from remote repository.

    Please make sure you have the correct access rights
    and the repository exists.
    [2013-07-15 08:46:42 CEST] XMPPSendPack: Unable to download files from your other devices.
    [2013-07-15 08:46:42 CEST] XMPPSendPack: Syncing with janwxmpp
    fatal: Could not read from remote repository.

    Please make sure you have the correct access rights
    and the repository exists.
    [2013-07-15 08:48:42 CEST] XMPPSendPack: Unable to download files from your other devices.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.90"
 subject="comment 1"
 date="2013-07-16T19:40:20Z"
 content="""
Yes, you need to add the rsync remote to the second computer. Once it's set up, they will be able to exchange file contents using it, and using XMPP as a control channel.

If the first computer already has that rsync remote configured, it should push some notes about the existance of that remote to the second one, over XMPP. Then in the webapp, you should just need to click on \"enable\" next to the rsync remote, and it'll prompt for the necessary info to set it up.
"""]]


code:
I'm trying to add a ssh remote in the web app, but I receive the following error:

    Internal Server Error

    ssh-keygen ["-F","router.eisenacher81.org"] exited 1

The console complains about the known_hosts file:

   /home/michael/.ssh/known_hosts is not a valid known_hosts file.
   line 44 missing key:  AAAAB3NzaC1yc2EAAAABIwAAAIEAtnX75Qa8YVR...
   key_read: uudecode AAAAB3NzaC1|1|veTakKhYY3OSqCepiq7WAUK8cxQ=|suoi0YU/lgg781Vz9O7yTao5exY= ssh-rsa                            AAAAB3NzaC1yc2EAAAABIwAAAIEA4bTnEFFiKz4+i3S9DWHJVoV8CD1DsVRJRodWdTA86+x58S0l0rUyJgXbGwScMI+xFCrqzpd4Hgoc4ElRykj5SL+7IuB5ZAe+4ILQPeiL9ck/1Q7uoh7vWiURXr92Hz5tjuEe3QI9H   iKauXj5yj5mGq1VVXLN1CdfQ99G4zSxK7c= failed
   line 73 invalid key: |1|k9vzo7Ftuxf235It5LbrX36p2f0=|rNHdygip...
   line 93: invalid hashed name: |1||1|aptgcuUg4ITWhZQLxzRHH8gUIOQ=|Qe2JAlAw++SZosE9ZhQW+fA3twE=...
   line 120 missing key:  AAAAB3NzaC1yc2EAAAABIwAAAQEAsHa75NfjyB1...
   /home/michael/.ssh/known_hosts is not a valid known_hosts file.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.14.141"
 subject="comment 1"
 date="2012-09-22T12:40:55Z"
 content="""
Looks like you have a corrupt known_hosts file, but I'm pretty sure git-annex did not do it, as it only lets ssh update that as normal. If you delete the problem lines it'll probably work, though you might want to save a copy to investigate what happened.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.14.141"
 subject="comment 2"
 date="2012-09-27T15:29:56Z"
 content="""
I've made the assistant ignore ssh-keygen -F failing.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmU_2tE75oyG0h2ZPN4lcroIKEMC8G-otE"
 nickname="Michael"
 subject="comment 3"
 date="2012-09-27T15:42:19Z"
 content="""
Okay, cleaning up the file worked for me, but I suspect that these invalid entries were created by a valid program. So ignoring errors in the file is \"good enough\".
"""]]


code: I have an external HDD which I setup as a remote manually before I started using the assistant.  When I plug it in files sync fine and the git-annex branch gets synced, but the master branch doesn't; git annex sync doesn't do the job and I have to run git pull.  This is annoying because it means that portable drive is useless as a backup, since it doesn't have the required metadata; I have to manually git pull to make it a useful backup.  How can I make the assistant sync the metadata in--if not the master branch, at least origin/master so I could merge later without the remote available.  Thanks.


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 1"
 date="2012-12-08T21:25:49Z"
 content="""
apologies if I misunderstand your question, but I believe that git-annex puts the metadata in the repository's sync/master branch, which you can merge into the repository's local master either manually or by running \"git annex sync\" in the repository.

"""]]


code:
[[!comment format=mdwn
 username="spwhitton"
 ip="163.1.166.255"
 subject="comment 2"
 date="2012-12-08T21:57:37Z"
 content="""
It does, which is what one would have expected I guess, but I didn't look.  Thanks so much.
"""]]


code:
Is there any fast way to rsync a direct-mode (assistant) git/annex directory from one machine to another?

Scenario: I was already using unison to synchronize 40GB between my desktop and laptop. I've added git annex to my desktop, but is there a way to bootstrap on my laptop without needing to re-transfer all the files? (I don't have a .git on my laptop yet but all the files are there already.)


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="24.159.78.125"
 subject="comment 1"
 date="2014-05-23T15:30:23Z"
 content="""
I don't recommend moving direct mode repositories between filesystems, because git-annex relies on the inode as one way to detect if files in a direct mode repo have changed. It will notice that all the inode numbers have changed, and fall back to a mode where it only uses mtime.

Anyway, your situation is better handled by initializing a repository on the desktop, and a separate one on the laptop. Then add remotes tying the repos together, and git-annex sync will just do the right thing, including not transferring files that have already been synced to both places.
"""]]


code:
[[!comment format=mdwn
 username="jamieson"
 ip="71.11.161.246"
 subject="Thanks!"
 date="2014-05-23T16:11:35Z"
 content="""
I'll do it that way, thanks!!
"""]]


code:
hi,
i just did git annex unused in 2 repos, one with WORM and one with SHA1.
with WORM i instantly could see which file it was, with SHA1 not.

so would it be possible to just add the first seen filename to the hashes identifier? the name could just be ignored but it would help with unused to see which file i am searching for.
thanks!