code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkJafmCf-sg9_OM0pynFYM3AO4WCgJiaMI"
 nickname="Michele"
 subject="possible explanation"
 date="2013-10-29T11:56:21Z"
 content="""
now, i tried to understand what happens. Instead of issuing the *git annex sync*, I relied on *git pull origin, git merge origin/master*, (I red [[http://git-annex.branchable.com/forum/Help_Windows_walkthrough/]] and I assume that pull origin / merge origin/master would work similarly to the \"download\" part of sync, *except for losing all my direct content*) just to understand what was going on, with a clarifying result:

while git annex sync fails on push, git pull origin fails on pull:

    M:\win>git pull origin
    Updating 5408d6f..c566a69
    error: Your local changes to the following files would be overwritten by merge:
            myfile
    Please, commit your changes or stash them before you can merge.
    Aborting

note that the file has not been modified locally (just got it through git annex get).
issuing a git diff, reveals:

    M:\win>git diff myfile
    diff --git a/myfile b/myfile
    index beaf3e8..dc5b4ff 120000
    --- a/myfile
    +++ b/myfile
    @@ -1 +1 @@
    -.git/annex/objects/z5/v7/SHA256E-s8--6090923ed0931dcc6699f32fb66fa4ba32c10924088b12c66fb4ce35a91ba98c/SHA256E-s8\ No newline at end of file
    +linux.1
    (END)

ok, i follow suggestion, and I perform a git stash. that still wouldn't suffice for git annex sync:

    M:\win>git annex sync
    commit
    ok
    pull origin
    ok
    push origin
    To ssh://michele@home/home/michele/homebase
     ! [rejected]        master -> synced/master (non-fast-forward)
    error: failed to push some refs to 'ssh://michele@home/home/michele/homebase'
    hint: Updates were rejected because a pushed branch tip is behind its remote
    hint: counterpart. Check out this branch and merge the remote changes
    hint: (e.g. 'git pull') before pushing again.
    hint: See the 'Note about fast-forwards' in 'git push --help' for details.
    failed
    git-annex: sync: 1 failed

now, i can perform instead a *git pull origin*, since I am confident my content is stashed.

    M:\win>git pull origin
    Updating 5408d6f..c566a69
    Fast-forward
     myfile => myfile.renamed | 0
     1 file changed, 0 insertions(+), 0 deletions(-)
     rename myfile => myfile.renamed (100%)

merge is not doing anything more: at this stage content has gone (file is a direct-mode symlink nad it cannot be fixed by fsck).
But i can recover it from stash (and I must do it unless I want to get the annex to think i still have content).

    git stash apply

voilà: the content is there! and the repos seems in good order.
this only adds up that this is possibily a bug in the fact that git reports direct content as modified when indeed it hasn't been modified: but this affects git annex sync only when merging renaming files.
git annex sync now works perfectly .

to sum it up, I have two questions:

1) does using stash to circumvent the problem expose me to any risk ?
2) would the behaviour on receiving renames in the abovementioned situation worth to be signaled as a bug ?

"""]]


code:
hi,

i think of use git-annex as the backbone of a archival systems. at first point no distributed storage, just 1 node.
but now i run into the topic below ( deleted the "named" symlink of the "object" -- how to recover ?)

maybe someone can enlighten me...

thanks,
.ka

// about the version. ( debian-squeeze, bpo )

$ git-annex version
git-annex version: 3.20120629~bpo60+2
local repository version: 3
default repository version: 3
supported repository versions: 3
upgrade supported from repository versions: 0 1 2

// building up a testcase.

$ git init
Initialized empty Git repository in ...test2/.git/

$ git annex init
init  ok
(Recording state in git...)

$ echo "aaa" > 1.txt

$ echo "bbb" > 2.txt

$ git-annex add .
add 1.txt (checksum...) ok
add 2.txt (checksum...) ok
(Recording state in git...)

$ git commit -a -m "added 2 files"
fatal: No HEAD commit to compare with (yet)
fatal: No HEAD commit to compare with (yet)
[master (root-commit) fc2a5d7] added 2 files
 Committer: userhere user <bla@bla>
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
...
 2 files changed, 2 insertions(+), 0 deletions(-)
 create mode 120000 1.txt
 create mode 120000 2.txt

// ok, so far standard. i have now 2 files - lets delete one.

$ rm 2.txt
$ ls -l
lrwxrwxrwx 1 xp xp 176 24. Okt 22:55 1.txt -> .git/annex/objects/Z6/7q/SHA256-s4--17e682f060b5f8e47ea04c5c4855908b0a5ad612022260fe50e11ecb0cc0ab76/SHA256-s4--17e682f060b5f8e47ea04c5c4855908b0a5ad612022260fe50e11ecb0cc0ab76

// eek, delete of 2.txt was a bad idea (it was just the symlink) -- try to recover...

$ git-annex fix
$ git-annex fsck
fsck 1.txt (checksum...) ok
$ ls
1.txt

// still not here.. how to recover the link to 2.txt ???
// i still see the content of the file in the object folder
// if I want to use git-annex as the backend of a archival system, this is important.

repo_name: kakafarm/git

license: unknown


code:
[[!comment format=mdwn
 username="http://ciffer.net/~svend/"
 ip="2001:1938:81:1ff::2"
 subject="git checkout"
 date="2012-10-24T22:50:05Z"
 content="""
You can use `git checkout -- file` to restore the link. `git status` will display information for restoring the file.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.138"
 subject="it's a git repository..."
 date="2012-10-25T03:29:20Z"
 content="""
So any git stuff can be used. If you deleted a file, committed it, and want to undo that, you can `git log --stat` to find the commit, and `git revert` it. If you deleted a file, haven't committed yet, and want one more look at it, you can `git stash` to get it back, and `git stash apply` to re-stash the change. Or yeah, just `git checkout` to get back a deleted file you have not committed yet.
"""]]


code:
[[!comment format=mdwn
 username="ka7"
 ip="2001:7b8:155d:0:222:64ff:fe16:dc52"
 subject="ok, that worked."
 date="2012-10-25T20:15:26Z"
 content="""
i think of a kind of \"WORM-library\", so basically just add, not allow to remove content. (at least not for the user thru the mounted device)
- so a script to add/commit -- but as stag-1 check for delete files and get them back. some git magic needed, but should be doable.
- or thru \"samba\" parameters set to add but not delete/overwrite files.  (read yes, write yes, delete no) -- to be proved thats possible, but not your job :) ( annex-ing via cron every /5 or via inotify)
so yea, will play for a while and maybe come back with new. thanks to everybody.
<3 git-annex <3
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnXlmnJKsPulm2S_hwwRm3Ky27Zyf-wKMw"
 nickname="Chad"
 subject="After the fact..."
 date="2013-12-22T03:51:36Z"
 content="""
I've run into a situation where I've inadvertently deleted a file from my Andriod-driven annex repo rather than dropping the file. (A set of very large *.flac files, to be exact.) These are now gone from the repo on my laptop and the repo on my workstation. When looking at the `git --log stat' call, I see the addition of these files, but I do not see their deletion. When running `git annex unused', I found references to the *.flac files, their SHA key, and indeed found the commit they were associated with. I have not dropped these unused files. How to I recover them?
"""]]


code:
In my current annex config, I have 4 computers with "traditional" git annexes as well as an external drive that is a git annex, an rsync'd backup annex, and a glacier archive.  Today, one of the computers got a corrupted git repo. It was complaining that a pack file was invalid. In my attempts to fix it, a commit was logged that deleted every file in the annex. I didn't find this out until I did 'git annex sync' and watched git delete everything, then send all those commits to my other 3 systems and the external drive. *facepalm*

Fortunately, I had one of those other systems in direct mode and I copied everything from the annex as a backup. Now, when I try to re-add files to the annex, I'm running into some errors. These appear to be "collisions" within the annex part of the .git folder:

    %  › git annex add House.netspd
    add House.netspd (checksum...)
    git-annex: /Users/akraut/Desktop/annex/.git/annex/objects/31/Gw/SHA256E-s167433--41e68ea0adb5a4086a0b7b39d0556b9b86523ffb6b498d58f12f96460da315e9/SHA256E-s167433--41e68ea0adb5a4086a0b7b39d0556b9b86523ffb6b498d58f12f96460da315e9.map.tmp62699: openFile: permission denied (Permission denied)
    failed
    git-annex: add: 1 failed

Any ideas on what's going on here? Perhaps how to get things added back in or recovered? It seems all the actual file contents are here, but annex doesn't seem to know they're there anymore.


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="99.54.57.201"
 subject="comment 1"
 date="2013-01-26T07:11:35Z"
 content="""
speaking strictly from the git point of view, you can create a new commit that reverses an old commit completely.

    git revert <commit>

creates a new commit which does the exact opposite of the commit you specify.  Theoretically it bring everything back to you quick and clean!

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmRFKwny4rArBaz-36xTcsJYqKIgdDaw5Q"
 nickname="Andrew"
 subject="comment 2"
 date="2013-01-26T08:13:34Z"
 content="""
Oh! Good call. That worked, though I did have to sync and revert alternately a few times to get things resolved.  I still haven't figured out what exactly happened, though. In two separate instances, the commit message was \"git-annex automatic sync\"
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 3"
 date="2013-01-26T09:11:00Z"
 content="""
I've fixed the bug that led to the permissions denied error.
"""]]


code:
Starting the assistant version 5.20140613 on my  repository (~60GB), it performed a merge
with an offline-repository, deleting a large part of the files.
Since the repo is in direct mode, I cannot do a git revert. The other repo is not available anymore

Any way of getting them back?
Also, du shows me that the .git/annex/objects folder has approximately the size of my repo before the incident.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 1"
 date="2014-06-16T17:58:22Z"
 content="""
All your files are still there..

You can either use `git annex indirect` to switch to indirect mode and then `git revert`, or you could git clone the repo to someplace else, do the revert there, and then run `git annex sync` in first the clone and then in the direct mode repo to sync the reversion back.
"""]]


code:
Hi,

I switched distribution and cloned my annex from the backup afterwards. getting the files from my backup worked fine, but i have a remote on an external disk and git-annex keeps saying it's unavailable despite it being there. I have the same username and hostname as on my old system so and the remote repo still has the uuid 3d661c5e-c84e-4902-b52e-decbafadfa4f. The only thing that changed is the path, it's know mounted unter /var/run/media/phaer/95... but a symlinked to the path below did not help. Any suggestions?

    3d661c5e-c84e-4902-b52e-decbafadfa4f -- phaer@kassiopeia:/media/phaer/9502bc14-b83e-471a-95a8-80ff9072a4ab/annex

and thank you for this amazing piece of software.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="64.134.31.139"
 subject="comment 1"
 date="2013-10-21T22:24:35Z"
 content="""
It might help if you pasted the actual error message that git-annex shows when you try to use this remote. Along with the part of .git/config that configures this remote.

In general, git-annex does not are if you've reinstalled your OS, moved things around, etc. As long as the git remote's path points to a repository, git-annex will see it and be able to use it.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmBhQgaA5QrFwT67-Bo0qPIx0HD9roDrso"
 nickname="Paul"
 subject="comment 2"
 date="2013-11-20T11:20:13Z"
 content="""
The problem was just a missing git remote. Sorry & thanks
"""]]


code:
This falls into the category of "noob questions" I think.

The one piece of the git-annex assistant puzzle I've never messed with is XMPP pairing.  I'm wondering how well a pair of repos can keep in sync with each other if their only connection is via XMPP.  Will things go badly if changes are made to one while the other is offline?  Do messages get queued up to deliver when they're both online?  (Or do they get queued on the server side so they can be delivered even if one of them is online, makes changes, then goes offline, and the other one comes online later?)

If some xmpp messages don't go through for whatever reason, will the remotes be able to "catch up" with each other later on and make up for lost time?

Just hoping for a general sense of the limitations of XMPP pairing.   TIA.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.90"
 subject="comment 1"
 date="2013-07-16T19:27:07Z"
 content="""
Both clients have to be online at the same time for XMPP push to work. Once they're able to see each other, they'll sync up, even if they've diverged since the last sync.

If you tend to only have one client or the other online, you should set up a git repo on a ssh server. Then clients will drop off their changes there, and the other one will check it when it comes online. (You don't really need to use XMPP at all in this case.)

The most robust and fast combo is to use XMPP pairing, and also have a git repo on a ssh server. This way, when both clients are online, they'll use XMPP to instantly propagate changes, and when not the server is there to fall back to.
"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 2"
 date="2013-07-16T23:36:22Z"
 content="""
Thanks, that was exactly the kind of explanation I needed.
"""]]


code:
[[!comment format=mdwn
 username="Flo"
 ip="141.76.46.31"
 subject="Git repositery?"
 date="2013-07-19T08:56:04Z"
 content="""
I'm not sure to know what you mean with \"git repository on a SSH server\". I tried to use the assistant to add this git synchronization fall-back if XMPP is not available. However, it synchronizes all files too with this new repository. I tried to make it manual, unwanted, etc. But it stills send all my files on the transfer repository on rsync.net. I don't want that...
"""]]


code:
**<out-of-date-warning>**The main problems this is supposed to solve are addressed in a different way with [[todo/hidden files]] and the `--fast` option introduced in [[batch check on remote when using copy]], so while this is not technically obsolete, the main reasons for it are gone. --[[chrysn]]**</out-of-date-warning>**

This is a rough sketch of a modification of git-annex to rely more on git commit semantics. It might be flawed due to my lack of understanding of git-annex internals. --[[chrysn]]

Summary
=========

Currently, [[location tracking]] is only used for informational purposes unless a repository is [[trust]]ed, in which case there is no checking at all. It is proposed to use the location tracking information as a commitment to keep track of a file until another repository takes over responsibility.

git's semantics for atomic commits are proposed to be used, which makes sure that before files are actually deleted, another repository has accepted the deletion.

Modified git-annex-drop behavior
==========================

The most important (if not only) git-annex command that is affected by this is `git annex drop`. Currently, for dropping a large number of files, every file is checked with another (or multiple, if so configured) host if it's safe to delete.

The new behavior would be to

* decrement the location tracking counter for all files to be dropped,
* commit that change,
* try to push it to at least as many repositories that the numcopies constraints are met,
* revert if that fails,
* otherwise really drop the files from the backend.

Unlike explicit checking, this never looks at the remote backend if the file is really present -- otoh, git-annex already relies on the files in the backend to not be touched by anyone but git-annex itself, and git-annex would only drop them if they were derefed and committed, in which case git would not accept the push. (git by itself would accept a merged push, but even if the reverting step failed due to a power outage or similar, git-annex would, before really deleting files from the backend, check again if the numcopies restraint is still met, and revert its own delete commit as the files are still present anyway.)

Implications for trust
==============

The proposed change also changes the semantics of trust. Trust can now be controlled in a finer-grained way between untrusted and semi-trusted, as best illustrated by a use case:

> Alice takes her netbook with her on a trip through Spain, and will fill most of its disk up with pictures she takes. As she expects to meet some old friends during the first days, she wants to take older pictures with her, which are safely backed up at home, so they can be deleted on demand.
>
> She tells her netbook's repository to dereference the old images (but not other parts of the repository she has not copied anywhere yet) and pushes to the server before leaving. When she adds pictures from her camera to the repository, git-annex can now free up space as needed.

Dereferencing could be implemented as `git annex drop --no-rm` (or `move --no-rm`), freeing space is similar to `dropunused`.

A trusted repository with the new semantics would mean that the repository would not accept dropping anything, just as before.

Advantages / Disadvantages
=====================

The advantage of this proposal is that the round trips required for dropping something could be greatly reduced.

There should also be simplifications in the `git annex drop` command as it doesn't need to take care of locking any more (git should already do that between checking if HEAD is a parent of the pushed commit and replacing HEAD).

Besides being a major change in git-annex (with the requirement to track hosts' git-annex versions for migration, as the new trust system is incompatible with the old one), no disadvantages of that stragegy are known to the author (hoping for discussion below).


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2011-02-22T18:44:28Z"
 content="""
I see the following problems with this scheme:

- Disallows removal of files when disconnected. It's currently safe to force that, as long as
  git-annex tells you enough other repos are belived to have the file. Just as long as you
  only force on one machine (say your laptop). With your scheme, if you drop a file while
  disconnected, any other host could see that the counter is still at N, because your
  laptop had the file last time it was online, and can decide to drop the file, and lose the last
version.

- pushing a changed counter commit to other repos is tricky, because they're not bare, and
  the network topology to get the commit pulled into the other repo could vary.

- Merging counter files issues.  If the counter file doesn't automerge, two repos dropping the same file will conflict. But, if it does automerge, it breaks the counter conflict detection.

- Needing to revert commits is going to be annoying. An actual git revert
  could probably not reliably be done. It's need to construct a revert
  and commit it as a new commit. And then try to push that to remotes, and
  what if *that* push conflicts?

- I do like the pre-removal dropping somewhat as an alternative to
  trust checking. I think that can be done with current git-annex though,
  just remove the files from the location log, but keep them in-annex.
  Dropping a file only looks at repos that the location log says have a
  file; so other repos can have retained a copy of a file secretly like
  this, and can safely remove it at any time. I'd need to look into this a bit more to be 100% sure it's safe, but have started [[todo/hidden_files]].

- I don't see any reduced round trips. It still has to contact N other
  repos on drop. Now, rather than checking that they have a file, it needs
  to push a change to them.
"""]]


code:
[[!comment format=mdwn
 username="http://christian.amsuess.com/chrysn"
 nickname="chrysn"
 subject="comment 2"
 date="2011-02-23T16:43:59Z"
 content="""
i'll comment on each of the points separately, well aware that even a single little leftover issue can show that my plan is faulty:

* force removal: well, yes -- but the file that is currently force-removed on the laptop could just as well be the last of its kind itself. i see the problem, but am not sure if it's fatal (after all, if we rely on out-of-band knowledge when forcing something, we could just as well ask a little more)
* non-bare repos: pushing is tricky with non-bare repos now just as well; a post-commit hook could auto-accept counter changes. (but pushing causes problems with counters anyway, doesn't it?)
* merging: i'd have them auto-merge. git-annex will have to check the validity of the current state anyway, and a situation in which a counter-decrementing commit is not a fast-forward one would be reverted in the next step (or upon discovery, in case the next step never took place).
* reverting: my wording was bad as \"revert\" is already taken in git-lingo. the correct term for what i was thinking of is \"reset\". (as the commit could not be pushed, it would be rolled back completely).
    * we might have to resort to reverting, though, if the commit has already been pused to a first server of many.
* [[todo/hidden files]]: yes, this solves pre-removal dropping :-)
* round trips: it's not the number of servers, it's the number of files (up to 30k in my case). it seems to me that an individual request was made for every single file i wanted to drop (that would be N*M roundtrips for N affected servers and M files, and N roundtrips with git managed numcopies)

all together, it seems to be a bit more complicated than i imagined, although not completely impossible. a combination of [[todo/hidden files]] and maybe a simpler reduction of the number of requests might though achieve the important goals as well.
"""]]


code:
[[!comment format=mdwn
 username="http://christian.amsuess.com/chrysn"
 nickname="chrysn"
 subject="relation to [[todo/branching]]"
 date="2011-02-23T21:48:14Z"
 content="""
the non-bare repository issue would go away if this was combined with the \"alternate\" approach to [[todo/branching]]. (with the \"fleshed out proposal\" of branching, this would not work at all for lack of shared commits.)
"""]]


code:
so i am wondering how better to remember the state of a repository.

the use case is this: i have this part of my music collection which i consider to be my "favorites". i would like to have only those on an external hard drive, but also on my portable music player. so i cloned my main music collection to the external hard drive and did `git annex get` on the files i wanted there. now i want to replicate that to the music player - how would i go around doing that? if i clone that repo again, it will all have empty symlinks and i would have to reproduce the same process all over again, which i can script, but it seems a little quirky...

ideas? --[[anarcat]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlQeMMI82sfgZ9lim_9GqLkc4fsd0Z3Qt8"
 nickname="Ronald"
 subject="comment 1"
 date="2014-05-14T09:08:22Z"
 content="""
I'm still learning git-annex myself, but I think you can make a new branch 'favorites'. Sort out your favorite music and make a checkout on your external HDD and music player. In the walkthrough you can see how that works.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkYmMFDdf3GJ9Oba6NCVkzGc4JyB9WavMs"
 nickname="Xinruo"
 subject="comment 2"
 date="2014-05-15T02:07:06Z"
 content="""
I think a better solution is to use http://git-annex.branchable.com/tips/metadata_driven_views/
You can tag the files as favorites, then use get --metadata tag=favorites to get only favorites files.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 3"
 date="2014-05-15T20:01:14Z"
 content="""
While you can certianly use views for this, I tend to do the same with my sound files without view, but just using location tracking info. I have a set of music I'm currently listening to, and want to have more or less the same files on multiple computers as I have on my laptop. With maybe a bit more on some computers with more space. So, I do:

    git annex sync
    git annex get --in darkstar

Where \"darkstar\" is the name of the repository on my laptop.

Another nice trick that also takes advantage of the location tracking logs is this to get back some files you dropped temporarily to free up space:

    git annex get --in='here@{yesterday}'

"""]]


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="70.83.139.100"
 subject="comment 4"
 date="2014-05-21T18:02:51Z"
 content="""
i like @yesterday, that's a neat trick that would have basically resolved my problem here. thanks!
"""]]


code:
I've gone through most of the screen casts and lots of documentation,
but my first attempt produced an unexpected result.

I wanted to use git-annex is a Dropbox-simplistic setting - syncing
a directory between my notebook (OSX) and my desktop (OpenSUSE 12.3).
Running the assistant on OSX and adding a "Remote server" repository,
changing the repository group to client produced a bare repository
on the desktop.

Am I missing something obvious?

I've tried a few different version of git-annex, most recently

   git-annex version: 4.20130727-g9399845
   build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP

but it remains the same.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmg8DSRA425Aik2clIuge8Jj-s7VJltsBI"
 nickname="Tommy"
 subject="comment 1"
 date="2013-07-28T06:05:48Z"
 content="""
I found a partial answer on http://git-annex.branchable.com/forum/Add_a___34__local__34___remote :
\"The webapp does not set up remote non-bare repositories because unless the user is comfortable with the cli, nothing is going to keep them up-to-date.\"

So, clearly \"Remote server\" isn't what I want, but my \"desktop\" is headless and using jabber seems a silly complication when I have a perfectly fine ssh connection.  This isn't as trivial as I had expected.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.21"
 subject="comment 2"
 date="2013-07-30T18:56:00Z"
 content="""
Make a non-bare git repository on your server. Run `git annex init` in it. Run `git annex assistant` in it.

Now on your notebook, you can tell the assistant to use a remote ssh server, and point it at that directory. It will see the repository already exists, and use it, rather than making a new bare repository. The assistant daemon you've run on the server will notice when changes are pushed to the directory, and update it.

You can further set up XMPP, so that the server can tell the notebook if you make any modifications to the server. You can do this by running `git annex webapp --listen=IPADDRESS` on the server, and opening the url it prints out. But this is optional, only needed if you're going to be editing files on the server.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkftzaCvV7EDKVDfJhsQZ3E1Vn-0db516w"
 nickname="Edward"
 subject="One snag"
 date="2014-07-28T19:37:04Z"
 content="""
I setup a non-bare repo on a server by following the above steps (git init, git annex init, then add it as a Remote Server from elsewhere and combine repos). It worked, but I hit a snag and needed to add another step.

After git init, you're not sitting on any branch yet, and that seems to have prevented the assistant from doing anything to synchronize the working tree on the server. After I did \"git checkout synced/master\", it started working.
"""]]


code:
I'm sorry to ask this again, but I'm not being able to find my previous message regarding this issue. I'm finding very hard to find older posts in forums. Is there a way to search for all my posted messages?
Anyway, after some automount problems (changed mount point name), my usb repo (direct) ended in a weird state.
See for instance the screenshot: <http://picpaste.com/git-annex-problem-eZvOl90y.jpg>

In addition making a git-annex info gives the following:

 * repository mode: direct
 * trusted repositories: error: refs/heads/master does not point to a valid object!
 * error: refs/heads/synced/git-annex does not point to a valid object!
 * error: refs/heads/synced/master does not point to a valid object!
 * error: refs/remotes/sapo/annex/direct/master does not point to a valid object!
 * error: refs/remotes/sapo/git-annex does not point to a valid object!
 * error: refs/remotes/sapo/master does not point to a valid object!
 * error: refs/synced/1912d5a7-3929-47f0-8e25-a071d7079cc4/git-annex does not point to a valid object!
 * error: refs/synced/1912d5a7-3929-47f0-8e25-a071d7079cc4/master does not point to a valid object!
 * error: refs/heads/master does not point to a valid object!
 * error: refs/heads/synced/git-annex does not point to a valid object!
 * error: refs/heads/synced/master does not point to a valid object!
 * error: refs/remotes/sapo/annex/direct/master does not point to a valid object!
 * error: refs/remotes/sapo/git-annex does not point to a valid object!
 * error: refs/remotes/sapo/master does not point to a valid object!
 * error: refs/synced/1912d5a7-3929-47f0-8e25-a071d7079cc4/git-annex does not point to a valid object!
 * error: refs/synced/1912d5a7-3929-47f0-8e25-a071d7079cc4/master does not point to a valid object!
 * fatal: index file smaller than expected
 * git-annex: failed to read sha from git write-tree

What should I do? delete the usb repo and start again. Thank god this is just a trial.

What happens if a file gets corrupted. Lets say it is corrupted in repo A, and fine in repo B. Will the good copy be overridden?

git-repair takes a long time and seems to be stuck there, or maybe it just takes a long time (10Gb repo). Is this normal?
    > git-annex repair
    Running git fsck ...
    Initialized empty Git repository in /tmp/tmprepo.1/.git/
    Trying to recover missing objects from remote sapo.
    Unpacking all pack files.
    Unpacking objects: 100% (348848/348848), done.

Thanks in advance.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 1"
 date="2014-07-15T18:50:29Z"
 content="""
It's normal for git-repair to take a long time.

git-annex uses checksums to detect if a file gets corrupted.
"""]]


code:
Can diskreserve be done with directory special remotes?
Where should such per directory remote setting be put?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.235"
 subject="comment 1"
 date="2013-07-10T23:50:53Z"
 content="""
Not currently supported. I've added a todo: [[todo/free_space_checking_for_local_special_remotes]]
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="comment 2"
 date="2013-07-10T23:56:08Z"
 content="""
I was looking at git-annex git and saw this and thought it might be supported.

commit 5cc76098ca7b702772ccf37a47f03da088148003
Author: Joey Hess <joey@kitenet.net>
Date:   Fri Apr 20 16:24:44 2012 -0400

    Directory special remotes now check annex.diskreserve.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.235"
 subject="comment 3"
 date="2013-07-11T16:17:36Z"
 content="""
Shows what I know! It is supported. Repurposed the bug report to allow configuring a different diskreserve for the remote than the main annex.diskreserve..
"""]]


code:
Hi,

This might be a stupid question, but I did not find any information about it.
Can I retrieve previous versions of a file?
Let's say, I wanna do a "git annex get file", but considering a specific commit id. Is it possible? Are all the versions of the files kept inside .git/annex/objects?

Thanks!


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2012-04-24T21:14:15Z"
 content="""
To get to a specific version of a file, you need to have a tag or a branch that includes that version of the file. Check out the branch and `git annex get $file`.

(Of course, even without a tag or branch, old file versions are retained, unless dropped with `unused`/`dropunused`.
So you could even `git checkout $COMMITID`.)
"""]]


code:
When trying to setup an rsync.net repo I always get the following error:

Permission denied, please try again.
Permission denied, please try again.
Received disconnect from 114.xxx.xxx.xxx: 2: Too many authentication failures for 2*****

I can ssh into the account without any problems and couldn't find anything, which would have helped me any further.
Any ideas? Is the problem sitting in front of the computer? Is it a bug?

Thanks.
David

This is happening on Mavericks (10.9)


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn_3tllXjSmtgm__aGr9Z4gVNFgJgGyJ30"
 nickname="David Alan"
 subject="comment 1"
 date="2014-04-02T15:58:57Z"
 content="""
This problem also exists, when connecting to other ssh remote servers.
The error msg would be:

 Failed to ssh to the server. Transcript: Permission denied, please try again. Received disconnect from 80.xxx.xxx.xxx: 2: Too many authentication failures for ssh-xxxxxx-git-annex-assist
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.244"
 subject="comment 2"
 date="2014-04-02T19:41:29Z"
 content="""
What version of git-annex?

Do you have ssh-askpass installed?

You might try running git webapp from a terminal, in case ssh is unable to prompt for the password in a window using ssh-askpass.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn_3tllXjSmtgm__aGr9Z4gVNFgJgGyJ30"
 nickname="David Alan"
 subject="comment 3"
 date="2014-04-03T00:31:14Z"
 content="""
I'm running Version: 5.20140318-gdcf93d0. Do I have to install a ssh-askpass manually? I only found one thread referring to ssh-askpass, but didn't really know what to do with that information. So, no I don't have it installed as far as I know... _I_ never did anyway...

Do I need it? Did I miss something in the docs?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="216.145.95.162"
 subject="comment 4"
 date="2014-05-19T16:46:55Z"
 content="""
I suggest you upgrade to yesterday's release. It moves ssh password prompting during repository setup into a field in the webapp, so should avoid this class of problems.
"""]]


code:
Hello,

I am on OS X, using version 5.20140613.  I installed using brew.

I have a really simple example where I'm trying to use local rsync to test transfer between two repos.  It keeps asking me for a password over and over.  I do not know what password it is.

I'm literally doing just this:

    $ cd annex-tests
    $ mkdir target
    $ mkdir source
    $ cat > source/text
    hi this is a test
    $ cd source
    $ git init
    Initialized empty Git repository in /Users/me/Documents/annex-tests/source/.git/
    $ git annex init
    init  ok
    (Recording state in git...)
    $ git annex initremote rsync-login type=rsync rsyncurl=me@localhost:/Users/me/Documents/annex-tests/target encryption=none
    initremote rsync-login ok
    (Recording state in git...)
    $ git annex describe rsync-login "test rsync url with a login"
    describe rsync-login ok
    (Recording state in git...)
    $ git annex add text
    add text ok
    (Recording state in git...)
    $ git annex sync rsync-login --content
    commit  ok
    copy text copy text (checking rsync-login...) Password:
    Password:
    Password:
    Password:
    Password:

I try a blank password, I try my login password, I tried 'none' (the encryption type), I try everything..  no dice

So right off the bat let me explain a few things.

1 -  Why not use rsa keys?  Because I was having the exact same problem and I wanted to make it even easier and use login/password

2 -  Why not use the webapp since a few forum posts say you can set passwords in there?  because my annex doesn't seem to recognize the command.

    $ git annex webapp
    git-annex: unknown command webapp

3 -  I verified my rsync command/url appears ok by doing the following.  The password for that command is as expected, it is my username password:

    $ cat > test2
    hi test
    $ rsync test2 me@localhost:/Users/me/Documents/annex-tests/target
    Password:
    $ cat /Users/me/Documents/annex-tests/target/test2
    hi test

Any help will be appreciated!


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 1"
 date="2014-07-11T18:41:32Z"
 content="""
Re brew not being built with support for the webapp, I suggest you file a bug on brew about this. IIRC they were, or were planning to include the webapp in the build. They may have lost track of that.

Now, what does rsyncurl=me@localhost:/Users/me/Documents/annex-tests/target say? It says to use rsync with the \"url\" me@localhost:/Users/me/Documents/annex-tests/target. What does rsync do with that url? It tries to ssh into localhost, as user \"me\". There's no magic here. I don't know why your login password isn't working, but you can verify by passing --debug to git-annex that it just runs rsync with the url you've given it.

BTW, are you sure you want to use a rsync special remote at all, rather than a regular git remote using ssh?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawk3JJvaarUKd7xShCMze_kpHVcuCzFrWTI"
 nickname="Lucas"
 subject="comment 2"
 date="2014-07-15T16:18:01Z"
 content="""
Well I got the password issue straightened out (was me, don't ask).

Both the rsync and git via ssh are working well, thanks!
"""]]


code:
[Walkthrough](http://git-annex.branchable.com/walkthrough/using_ssh_remotes/) says that when using ssh remotes rsync is used for transfering files. Is rsync used via ssh or unsecure?
-- Michael K.


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2011-03-06T15:59:37Z"
 content="""
Everything is done over ssh unless both repos are on the same system (or unless you NFS mount a repo)
"""]]


code:
[[!comment format=mdwn
 username="http://m-f-k.myopenid.com/"
 ip="92.194.43.135"
 subject="comment 2"
 date="2011-03-06T16:33:19Z"
 content="""
Great! This was the only thing about git-annex which could have kept me from using it. --Michael
"""]]


code:
[[!comment format=mdwn
 username="hannes"
 ip="130.226.142.243"
 subject="original repo git annex version"
 date="2012-07-08T11:55:42Z"
 content="""
is 3.20120629
"""]]


code:
In short: is it safe to rsync .git/annex/objects/ to other clones?

I'm in the process of migrating a lot of my files to a new backend.  Most them
still use the old non-file-size-tracking SHA1 backend from when WORM was the
default.  I have multiple clones that each use about 460 GB of their 500 GB
drives.

I've noticed that git-annex creates hardlinks to migrated content in
.git/annex/objects, so I'm able to migrate all content despite only having 40
GB of free space.  Excellent.

Now I'm planning to `rsync -a --hard-links .git/annex/objects/
${CLONE}/.git/annex/objects/` to recreate the hardlinks and save space on the
clones as well.  If I `fsck` afterwards, this should be fine, right?

I've tried this with a test repository and it works but I'd like to be extra
sure that I'm not missing something crucial.

The alternatives that I'm aware of are:

 * making space first: `git annex dropunused "1-20000"` and `git annex get .`
 * running `git annex migrate` in each clone.

I expect rsync to be safer and faster than these alternatives.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.246"
 subject="comment 1"
 date="2013-11-16T17:36:48Z"
 content="""
As long as you fsck afterwards, this is fine.

You might want to `git annex fsck --all` in case you have objects for old versions of files.
"""]]


code:
[[!comment format=mdwn
 username="gernot"
 ip="89.0.237.214"
 subject="comment 2"
 date="2013-11-18T15:09:59Z"
 content="""
Thanks for the confirmation, Joey!  I also wouldn't have thought of `--all`.
"""]]


code:
I backup/sync my music using annex. I used to have one repo with 3 clones, one full copy on my vps, one full copy and one partial copy on my laptops. I decided to move all data to s3. Declared my vps repo dead, purged history (I do not care about history for this particular repo) pushed the git repo to a different computer (bare repo no data) and data to s3 (gpg encrypted). I've just finished uploading all files (around 200gb) couple files failed during the initial upload so I did a `git annex copy --quiet --to mys3 --fast` this command used to take 15 20 secs on my laptop when sending data to vps using ssh but now it took 2 hours to complete (pushing mem usage to 90%).

I have one other repo (1.5gb in size 42k files using indirect mode, data gpg encrypted on s3) using the same setup except this repos content has always been on s3  i had the same behavior on this repo too. adding a file and running a copy to push content to s3 took couple hours even if I add a single 1kb file. I used to blame my hard drive, damn thing is slow but now I think this is related to s3. is there any workaround for this?

Both machines are using,

    git-annex version: 4.20130913-gd20a4f2
    build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP Feeds Quvi


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
 nickname="Justin"
 subject="comment 1"
 date="2013-11-21T04:09:15Z"
 content="""
Is

    git annex copy --not --in mys3 --to mys3 .

any faster?
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/FHnTlSBo1eCGJRwueeKeB6.RCaPbGMPr5jxx8A--#ce0d8"
 nickname="Hamza"
 subject="comment 2"
 date="2013-11-21T17:47:57Z"
 content="""
No difference. Both take roughly the same time but I did time a couple of runs with both commands they usually take 3 to 5 minutes to complete. Maybe git did something behind the scenes like gc? But still slower than it used to be. My other repo (one with 42k files) still takes a 2 hours.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
 nickname="Justin"
 subject="comment 3"
 date="2013-11-22T20:21:11Z"
 content="""
Well it's not related to s3... that copy command won't even do any network traffic if there is nothing to copy.  I have a similarly configured annex with 4500 files and that command takes 10 seconds to run.

I do remember there being a recent fix that reduced the algorithmic complexity of an operation, but I forget which.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 4"
 date="2013-12-12T20:14:01Z"
 content="""
You mentioned something about high memory usage when copying. How much memory are we talking about?

Have you run `git annex forget` in this repository before? It kind of sounds like you have, and it might be possible that it's repeatedly trying to forget old history for some reason.
"""]]


code:
the git-annex branch of a repository i've had running since 2010 has grown to unmanagable dimensions (5gb in a fresh clone of the git-annex branch, while the master branch has merely 40mb, part of which is due to checked-in files), resulting in git-annex-merges to take in the order of magnitude of 15 minutes. getting an initial clone of the git-annex branch (not the data) takes hours alone in the "remote: Counting objects" phase (admittedly, the origin server is limited in ram, so it spends its time swapping the git process back and forth).

is there a recommended way for how to reset the git-annex branch in a coordinated way? of course, this would have to happen on all copies of the repo at the same time.

the workflow i currently imagine is

* rename all copies of the repository (the_repo → the_repo-old, the_repo.git → the_repo-old.git)
* clone the old origin repository to a new origin with --single-branch. (this would be *the* oportunity to ``git filter-branch --prune-empty --index-filter 'git rm --cached --ignore-unmatch .git-annex -r' master`` as well, to get rid of commits of pre-whatever versions)
* ``git annex init`` on the master repository
* clone it to all the other copies and ``git annex init`` there
* set all the configuration options (untrusted repos etc) again
* either
  * ``git annex reinject`` the files that are already present on the respective machines, or
  * move the .git/annex/objects files over from the original locations, and use ``git annex fsck`` to make git-annex discover which files it already has, if that works. (i have numcopies=2, thus i'd dare to move instead of copy even when trying this out the first time. complete copies, even of partially checked out clones, will exceed the capacities of most clients)

my questions in that endeavor are:

* is there already a standard workflow for this?
* if not, will the above do the trick?
* can anything be done to avoid such problems in future?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.140"
 subject="comment 10"
 date="2013-07-20T20:03:20Z"
 content="""
Since you seem to have found a way that works, I'm only answering for completeness: To resolve a conflicted merge on the git-annex branch, you can just add all lines present in either side of the merge, in any order. This won't necessarily be the most minimal resolution, but it is guaranteed to always be a valid one. There is a git-union-merge program in the git-annex source (not built by default) that can do that when merging any set of branches.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="propagating squashed history to other remotes"
 date="2013-08-12T01:44:59Z"
 content="""
The easiest method seems to be to force-push git-annex and master to other remotes, e.g.

    git push -f myremote git-annex:refs/heads/git-annex

Before doing this, make sure location logs etc had a chance to propagate across all remotes.

It's a good idea to remove synched/ branches before doing git-annex sync on the repos with rewritten history, too:
    git branch -D synced/master
    git branch -D synced/annex

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4830:1600:187::2"
 subject="comment 12"
 date="2013-09-04T06:38:00Z"
 content="""
`git annex forget` automates this now, without needing to force-push or have a flag day. Needs a version of git-annex supporting it installed on *all* the computers you use the repo on. Repos notice they need to forget when git annex is run in them, and do, automatically.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.194"
 subject="comment 1"
 date="2012-10-31T16:03:55Z"
 content="""
Yes, you can use fsck like that. I outlined a similar approach [here](http://git-annex.branchable.com/forum/Delete_unused_files__47__metadata/#comment-e2bd822214df96606a60e8aee2c8637a), and I think you don't even need to make new git repositories, just delete the old branch and git gc it -- but I've not heard of anyone doing this yet.

So, since 2010, your repo must have gone through at least one and probably two repository format changes, which bloated the git branch. Hopefully we'll have no more of those. My largest repo that also went through that is under 150 mb however.

There was a recent bug fix where `git annex copy` unnecessarily updated location log even when the file was already copied. That kind of thing can bloat the repository, especially if you had that in a cron job... You might find `git annex log` useful to look through history of files and see if there have been a lot of location changes logged for whatever reason.
"""]]


code:
[[!comment format=mdwn
 username="http://christian.amsuess.com/chrysn"
 nickname="chrysn"
 subject="worked well"
 date="2012-11-04T12:23:56Z"
 content="""
the procedure i outlined originally worked well for me; the method chosen for reinjection was moving over the .git/annex/objects directory and doing a ``git annex fsck``.

special care had to be taken of the special remote (rsync+gpg) -- i guess that's why they are called special ;-) . as described in the forum post you linked, i had to copy over remote.log and the uuid.log line from the old git-annex branch -- otherwise, a ``git annex initremote`` would have generated a new hmac, effectively resetting the remote repo.

the formerly 5gb git-annex branch (admittedly not ``git gc``'d recently, but that just wasn't feasible any more) shrunk down to around 25mb of current location information. i'll keep an eye on how it's growing to see if the problem is inherent or if it was just old bugs causing trouble.
"""]]


code:
[[!comment format=mdwn
 username="http://vjt.myopenid.com/"
 nickname="vjt"
 subject="drop "content removed from annex" history"
 date="2013-06-18T02:12:01Z"
 content="""
Joey,

dropping the git-annex branch and subsequent fsck worked. Moreover, as I turned my repository in containing over 700k objects due to a silly cycle of `git annex add` / `git annex unannex`, bloating *both* `git-annex` and `master` history, to clean up I successfully performed a squashed rebase of master onto itself.

Here's what I did, in detail:

    $ git checkout git-annex
    $ cp *.log ..
    $ git checkout master
    $ git br -D git-annex
    $ git br -D synced/git-annex
    $ git checkout <first commit>
    $ git checkout -b git-annex
    $ cp ../*.log .
    $ <remove the changes done in the first commit, my case just adding a .gitignore>
    $ git add *.log
    $ git commit --amend -m 'Init'

With this, I got rid of the many `update` commits. Now, the fun part:

    $ git checkout master
    $ git rebase -i <first commit>
    <In the git-rebase-todo, I squashed almost everything, except a few commits I wanted to preserve>
    $ :wq

Rebase went fine, and I was left with a clean master. I brought also `synced/master` up to date:

    $ git checkout synced/master
    $ git reset --hard master

Now I re-created all the location links with fsck:

    $ git annex fsck

And eventually, got rid of the redundant history:

    $ git reflog expire --expire=now --expire-unreachable=now --all
    $ git gc --prune=now
    $ git repack
    $ git prune

yay, 500k objects less ^_^'.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="would git annex fsck also recreate location tracking info for special remotes?"
 date="2013-07-17T17:02:50Z"
 content="""
Suppose I want to drop history like discussed in this entry. How well does this deal with special (e.g. directory on an offline disk) remotes?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="comment 5"
 date="2013-07-17T17:06:39Z"
 content="""
Also, would simply squashing git-annex branch history (without fsck etc) work?
This seems easier.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.90"
 subject="comment 6"
 date="2013-07-17T18:57:12Z"
 content="""
You can use `git annex fsck --from remote` to verify that every file location tracking thinks is on the remote still is. It's innefficient though -- it has to download the whole file to check the special remote still has the right content! That transfer can be avoided by adding --fast.

This is documented in the man page. :)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.90"
 subject="comment 7"
 date="2013-07-17T18:59:03Z"
 content="""
I don't see any reason why squashing git-annex branch history would not work. If you squash it to the same sha in each clone, things would be very happy, but even if you squash it to different shas, the union merge should result in those  different versions of the same data automatically merging together.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="git annex merge driver?"
 date="2013-07-19T17:04:53Z"
 content="""

I've tried rebasing git-annex branch, and I hit a bunch of conflicts (both in uuid.log and for individual content file logs) of the form:

<<<<<<>>>>>> 52e60e8... update
1369615359.195672s 1 38c359dc-a7d9-498d-a818-2e9beae995b8

As I understand, git-annex has a special timestamp-based merge driver to deal with these. Is there a way to use that with git rebase?

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="git checkout --orphan"
 date="2013-07-19T17:49:37Z"
 content="""
Instead of rebase, --orphan seems to be the right answer for pruning history: create a new git-annex orphan branch and git add and commit the files. So:

git status

# verify there are no uncommitted or untracked files

# master branch
git branch -m old-master
git checkout --orphan master
git add .
git commit -m 'first commit'

# git annex branch
git branch -m git-annex old-git-annex
git checkout git-annex
git checkout --orphan git-annex
git add .
git commit -m 'first commit'
git checkout master

# at this point, you may want to double-check that everything is still OK

# finally, remove branches and clean up the objects:
git branch -D old-master old-git-annex
git reflog expire --expire=now --all
git prune
git gc

The repo remains functional and .git is smaller.

"""]]


code: This is just a comment on git-annex building on haskell platform 2011.2.0.0 on archlinux. It just works.


code:
I am trying to S3 as a file store for git annex. I have set up the remote via the following command:

    git annex initremote xxx-s3 type=S3 encryption=shared embedcreds=yes datacenter=EU bucket=xxx-git-annex fileprefix=test/

The remote gets set up correctly and creates the directory I want, and adds a annex-uuid file.

Now when I try to copy a file to the xxx-s3 remote, I get the following error:

    $ git annex add ssl-success-and-failure-with-tl-logs.log
    add ssl-success-and-failure-with-tl-logs.log ok
    (Recording state in git...)
    $ git annex copy ssl-success-and-failure-with-tl-logs.log --to xxx-s3
    copy ssl-success-and-failure-with-tl-logs.log (gpg) gpg: no valid OpenPGP data found.
    gpg: decrypt_message failed: eof

    git-annex: user error (gpg ["--batch","--no-tty","--use-agent","--quiet","--trust-model","always","--batch","--passphrase-fd","10","--decrypt"] exited 2)
    failed
    git-annex: copy: 1 failed

Any ideas what might be wrong? Is shared cipher broken somehow?


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmAINLSovhWM_4_KrbngOcxduIbBuKv8ZA"
 nickname="Nuutti"
 subject="comment 1"
 date="2014-08-01T09:28:21Z"
 content="""
Sorry, this should probably be in bugs.
"""]]


code:
I tried

    git annex initremote encsharedtest type=directory encryption=shared directory=/home/lee/gitannexplay

and got errors:

    initremote encsharedtest gpg: error reading key: public key not found

Looks like it thinks "shared" should be the name of a key rather than an instruction to use the shared cipher.
Am I doing something wrong?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.23"
 subject="comment 1"
 date="2012-10-23T19:51:03Z"
 content="""
That's the right syntax, and it works when I try it, even if I move my `~/.gnupg` directory away.

You sure you have a new enough git-annex for the shared cipher feature? It appeared in version 3.20120430

You could try passing the `--debug` option to see what it's trying to do with gpg when it fails.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlFUH5H4OUj9vMJIoXQs8bheiptgANQ6fU"
 nickname="lee"
 subject="A little too old"
 date="2012-10-23T20:47:40Z"
 content="""
That's the problem, then. I have 3.20120406. For my purposes a gpg key with no passphrase works, though. Thanks.
"""]]


code:
The web app used to be unresponsive without manual refreshes.  Now it's all snappy and responsive and quickly reflects what the assistant is doing.

It used to be a complete crapshoot whether the assistant actually shut down when you told it to (either on the command line or through the webapp).  I used to have to manually kill leftover assistants and child processes routinely.  Now... you want it gone, and it's gone instantly, every time.

Something really good happened with this last version, thank you joey!


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 1"
 date="2013-01-25T22:35:00Z"
 content="""
We fixed some broken javascript that only worked in all the browsers I tried, but not yours. It's always been this cool, just glad you can also experience it now!
"""]]


code:
I checked in my music collection into git annex (about 25000 files) and i'm really impressed by the performance of git annex (after i've done an git-repack). Now i'm also moving my movies into the same git-annex, but i have the following layout of my disk drives:

* small raid-1 for important stuff (music, documents), which is also backupped (aka: raid)
* big bulk data store (aka: media)

In the git-annex the following layout of files is used:

* documents/ <- on raid
* music/ <- on raid
* videos/ <- on media

Now i didn't simply clone the raid-annex to media, but did an sparse-checkout (possible since version 1.7.0)

* raid: .git-annex/, documents/ and music
* media: .git-annex/, videos/

As you can see i have to checkout the .git-annex directory with the file-logs twice which slows down git operations. Everything else works fine until now. git-annex doesn't have any problem, that only a part of the symlinks are present, which is really great. Is there a possibility to sparse checkout the .git-annex directory also? Perhaps splitting the log files in .git-annex/ into N subfolders, corresponding to the toplevel subfolders, like this?

Before:

     $ ls .git-annex
     00 01 02....

After:

     $ ls .git-annex
     documents/ music/ videos/
     $ ls .git-annex/documents
     00 01 02....

This would make it possible to checkout only the part of the log files which i'm interested in.


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2011-04-07T16:32:04Z"
 content="""
That's awesome, I had not heard of git sparse checkouts before.

It does not make sense to tie the log files to the directory of the corresponding files, as then the logs would have to move when the files are moved, which would be a PITA and likely make merging log file changes very complex. Also, of course, multiple files in different locations can point at the same content, which has the same log file. And, to cap it off, git-annex can need to access the log file for a given key without having the slightest idea what file in the repository might point to it, and it would be very expensive to scan the whole repository to find out what that file is in order to lookup the filename of the log file.

The most likely change in git-annex that will make this better is in [[this_todo_item|todo/branching]] -- but it's unknown how to do it yet.
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 2"
 date="2011-04-07T16:33:30Z"
 content="""
BTW, git-annex unused *will* have a problem that not all the symlinks are present. It will suggest dropping content belonging to the excluded symlinks.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkptNW1PzrVjYlJWP_9e499uH0mjnBV6GQ"
 nickname="Christian"
 subject="comment 3"
 date="2011-04-08T07:31:03Z"
 content="""
So perhaps checking if git-status (or similar) complains about missing files is a possible solution for this?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkptNW1PzrVjYlJWP_9e499uH0mjnBV6GQ"
 nickname="Christian"
 subject="comment 4"
 date="2011-04-08T07:54:37Z"
 content="""
And something else i've done is, that i symlinked the video/ directory from the media annex to the normal raid annex

    ln -s ~/media/annex/video ~/annex

And it's working out great.

    ~annex $ git annex whereis video/series/episode1.avi
    whereis video/series/episode1.avi(1 copy)
        	f210b45a-60d3-11e0-b593-3318d96f2520  -- Trantor - Media
    ok

I really like this, perhaps it is a good idea to store all log files in every repo, but maybe there is a possibilitiy to to pack multiple log files into one single file, where not only the time, the present bit and the annex-repository is stored, but also the file key. I don't know if this format would also be merged correctly by the union merge driver.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="Neat idea"
 date="2012-10-26T18:35:21Z"
 content="""
I imagine that most mail servers have a maximum attachment size; that seems the most likely problem with using this.

The directory special remote has support for splitting up keys into chunks to work around file size limits. It would be good, I suppose, to add that to the hook special remote too.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlFUH5H4OUj9vMJIoXQs8bheiptgANQ6fU"
 nickname="lee"
 subject="chunking"
 date="2012-10-26T22:05:16Z"
 content="""
That's a good point. But the splitting could easily be added to the hooks, and wouldn't need to be built in to git-annex.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmWg4VvDTer9f49Y3z-R0AH16P4d1ygotA"
 nickname="Tobias"
 subject="comment 3"
 date="2013-05-22T09:53:47Z"
 content="""
I had to change the checkpresent hook to:

	hogneygmail-checkpresent-hook = (imaputils.pl --subject $ANNEX_KEY --count | grep -q \\"1 messages\\" -) && echo $ANNEX_KEY || exit 0

Also, i have 5 files more than 30MB in size, that refuse to transfer.

 postdrop: warning: uid=1001: File too large
 sendmail: fatal: tou(1001): message file too big
 Error sending message, child exited 75 (Deferred.).
 Could not send the message.

  store hook exited nonzero!
 failed

This really needs some split operations to support large files.

"""]]


code:
I know versions of this question have been asked before, but I'm looking for a different answer.

I would like a braindead special remote that can be used with devices such as portable music players. No symbolic links, no hashing, no rewriting of the filenames. The remote can be untrusted, file identity can be checked with just the filename and maybe the size. The "directory" special remote with the WORM backend seems to come closest, but does too much.

Should I just try to roll it using hooks?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.149"
 subject="comment 1"
 date="2012-10-05T00:39:43Z"
 content="""
I've been thinking in similar directions on this page: [[design/assistant/desymlink]]
"""]]


code: Starting the assistant on a repository with a huge amount of files (f.e. `annexed files in working tree: 22087`) takes a very long time. Is there a way to speed it up? F.e. just checking the file stamps and comparing them with the last assistant run time? Checking if the files are modified could be done with the regular repository check. Thoughts?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 1"
 date="2013-11-05T16:43:26Z"
 content="""
Checking the time stamps is what it does.

How long is a very long time?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawne_amN4fko4p5cRY_9EYwaYuJKNn7LRio"
 nickname="Tobias"
 subject="measuring how-to"
 date="2013-11-08T16:55:58Z"
 content="""
How can I measure how much time the startup scan takes? I tried to find it out using the debug log, but couldn't find any usable notice when the startup scan was finished. Maybe this two lines?

    [...]
    [2013-11-08 16:00:45 CET] Watcher: Performing startup scan
    [...]
    [2013-11-08 16:15:30 CET] Watcher: watching
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 3"
 date="2013-11-08T17:04:23Z"
 content="""
That's right.

15 minutes is certainly a very long time.

Is this on a slow spinning disk? USB disk?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawniayrgSdVLUc3c6bf93VbO-_HT4hzxmyo"
 nickname="Tobias"
 subject="comment 4"
 date="2013-11-17T17:28:06Z"
 content="""
The mentioned repository is stored on spinning disk.
Maybe it would help to store the .git directory on an SSD, but I'm not sure how to do it.
"""]]


code:
I setup a local git annex repo and tried to pair it with an empty repo on my notebook. All done via the webapp.

But pairing failed somehow, because I get this error while transfer seems to go on forever.

    ssh_exchange_identification: read: Connection reset by peer
    ssh_exchange_identification: read: Connection reset by peer
    fatal: Could not read from remote repository.

    Please make sure you have the correct access rights
    and the repository exists.

The webapp shows the remote repo as "unfinished repository".

I can access the other computer via ssh and an other git-annex repo pairing between the two computers work fine.

UPDATE: In the meantime I setup another paired repo on these two computers. Everything works fine.

I added the repos from scratch again and the same errors occurs again. The repositories which do not pair are the Documents folder on my desktop and on my macbook. Is there something special with macs documents folder?

    [2013-12-31 10:53:23 CET] PushRetrier: Syncing with macbook.local_Documents
    fatal: 'macbook.local_Documents' does not appear to be a git repository
    fatal: Could not read from remote repository.

    Please make sure you have the correct access rights
    and the repository exists.

juh


code:
[[!comment format=mdwn
 username="Uic1ohv6"
 ip="62.253.227.100"
 subject="Are you using encfs?"
 date="2014-01-09T16:30:50Z"
 content="""
I've seen similar error messages when the local annex was on an encfs-mounted decrypted directory. Seemingly git-annex tries to create SSH sockets in .git, and this goes wrong at least in some such situations.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.43"
 subject="comment 2"
 date="2014-01-14T21:16:14Z"
 content="""
I tried using git-annex on encfs to see if I could reproduce this problem, and I could not get exactly this error, but I certianly did see it cause problems with the ssh connection caching socket.

It would be a good idea to `git config annex.sshcaching false` in repositories stored on encfs.

Actually, given other reports of problems caused by encfs, it would be a good idea to not use git-annex on encfs. There are better ways to encrypt your files.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnPgn611P6ym5yyL0BS8rUzO0_ZKRldMt0"
 nickname="Samuel"
 subject="What are those ?"
 date="2014-01-31T07:37:49Z"
 content="""
Hi,

Could you please indicate what other means of file encryption systems you have in mind?

The others I know are at a partition level (cryptsetup, LUKS) and are less pratical to setup than a simple encfs.
"""]]


code:
I'm trying to set up a ssh remote on Android using the latest 5.20140421. For some
reason the assistant fails to set up passwordless login on the remote server. It
adds the required line in `.ssh/authorized_keys` on the server side, but it still
keeps asking for the password for every connection. Nothing suspicious appears in
the assistant's log. Also, if I set up a ssh remote on a different directory on the
same server, a new key is generated and added to `authorized_keys`, but the passwordless
login still doesn't work.

I didn't file a bug since this would make for a very lousy bug report. How could I
look more into what's causing this? Where should the generated keys reside on the
Android filesystem?

I think it would be useful for the assistant to check that the generated ssh keys are
working properly, and inform the user and/or try to set them up again if there is a
problem, instead of silently falling back to asking for the login password on the console.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="216.145.95.162"
 subject="comment 1"
 date="2014-05-19T15:50:34Z"
 content="""
There have been recent improvements in the assistant's handling of ssh keys. This includes no more prompting for ssh keys in the console, ever. Instead, the ssh remote setup would presumably fail if the ssh key didn't work for some reason.

To debug your problem, you need to use the shell. I would first try running \"git annex get\" or \"git annex copy\" or \"git annex drop\" on a file, and verify that it prompts for the ssh password. Then, take a look at /sdcard/git-annex.home/.ssh/config, and see if you can ssh to the special hostname set up there, from android, and if it asks for a password. If so, ssh -v might be interesting, as it should show it presenting the ssh key to the server.
"""]]


code:
Currently I have my netbook running git annex and a raspberry pi as a remote storage set up as a git repository.  I currently have it set up so that from the terminal i can just do 'ssh pi@mypi -p XXX -C' and it has the password saved.  Currently I'm having a problem that any time I add a file to my annex folder on my netbook I have to reenter my password for ssh-askpass for it to upload the file.  Once that file is uploaded, it looks to me like it closes the connection and then reopens it and asks for my password again.

I'm looking in ~/.ssh/ and I'm sure that something there needs to be edited or copied, I'm not really sure what though.  Any suggestions?


code:
[[!comment format=mdwn
 username="http://a-or-b.myopenid.com/"
 ip="203.45.2.230"
 subject="comment 1"
 date="2013-05-30T00:08:52Z"
 content="""
It sounds like you don't have the ssh-agent set up properly in your environment.

Try starting the git annex agent from the command line and see if that helps.  If it works then it is a problem with environment variables.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-05-30T15:49:27Z"
 content="""
If you set this repository up using the git-annex webapp, and choosing \"remote ssh server\", it will generate a new ssh key, stored in `~/.ssh/git-annex/`, and it will configure `~/.ssh/config` to use this key for a special made-up hostname which is used in the url for the git remote. On the server, it will set up `~/.ssh/authorized_keys` to allow this ssh key to connect without a password and run git-annex-shell.

I don't know if you used the webapp to set this up. If not, you can try to replicate this setup on your own. If you did and it's not working, you now know where all the peices it sets up are, to figure out what went wrong.
"""]]


code:
Hi,

I own a computer that I use as a home server. I'm just getting started with git-annex, and I was trying to add an ssh remote. It connected fine, but only gave me the "encrypted rsync" option. I'd like to be able to access these files on the server as well, but the "use a git repository" option didn't show up. How do I enable this?


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmR-xG9O2HWoybxYZPVbYVYxwitfcfDgtE"
 nickname="Charles"
 subject="comment 1"
 date="2013-09-09T00:29:14Z"
 content="""
Oh actually, I just clicked the available option and it gave me a \"connection refused\" due to my using a port other than 22. I edited the .ssh/config, but it decided to use 22 anyway?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmR-xG9O2HWoybxYZPVbYVYxwitfcfDgtE"
 nickname="Charles"
 subject="comment 2"
 date="2013-09-09T14:50:42Z"
 content="""
Never mind, I simply didn't have git-annex installed on the other computer. I'm going to make a thread in bugs about assistant always assuming port 22, you can delete this thread.
"""]]


code:
Hello,

I'm new to git-annex and I would like to know which is the best method to deploy it for my case. I would like to have a central repo in my Synology NAS but with files to be able to browse files (not a bare repo). This repo will not have remotes configured. The mobile clients will use the central repo to exchange files. I thinks that this setup is totally valid x-D. Now my question is more related to how to share the repo in the Synology NAS. I am considering to use cifs/webdav for LAN/WAN access to the repo or SSH for LAN/WAN access. Which is the recommended setup? Which is the most optimum setup in terms of bandwidth or cpu usage (the CPU of my synology NAS is ARM). Thanks in advance and for this great piece of software!

Best regards,

Christian


code:
I wanted to try out this probably great piece of software you wrote, alas I do not use a Debian distro with managed software menus (i.e. I use OpenBox), so I cannot simply open the assistant through my programs-menu. In fact, to do that I'd need the command to start the assistant so I can add it manually myself. Sadly, the man page did not gave me any advices... :(

Any chance to give me the terminal command to start the assistant?

***
edit: just to update the thread: everythings okay now. Thanks for this great program!

the commands (as posted by Paul):

    git annex assistant
    git annex webapp


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmBhQgaA5QrFwT67-Bo0qPIx0HD9roDrso"
 nickname="Paul"
 subject="comment 1"
 date="2013-10-21T19:20:28Z"
 content="""
    git annex assitant

starts the assitant itself, you propably want the gui which you start by

   git annex webapp
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkj7tMEJKcZpNXIFkHAAcNi5qJFSFyjn6o"
 nickname="thissideup"
 subject="comment 2"
 date="2013-10-22T11:05:13Z"
 content="""
ah okay. That was what I was trying to do, but git-annex failed on my. Remembering that I was on Debian and checking the version (which was June 2012) - I supposed that I wasn't able to use the assistant yet.

So now I am trying to compile from source using cabal, however I sadly ran into a roadblock: http://git-annex.branchable.com/forum/cabal_install_fails_on_uuid/
"""]]


code:
so git annex migrate can switch a file from using one backend to the other.

I've done that with a bunch of files.

The old files should become "unused" right?  But they don't seem to be.  "git annex unused" still shows me only 2 unused files, and I've just migrated dozens of files from SHA256 to SHA256E.

Is it possible they're still "used" by other repos?  I have two other repos, one reached by SSH and one on a USB drive.  Neither one of them is "bare."  So maybe those files are still used by the "master" branches there, I thought... I went over and did "git annex sync" on each.  Still my newly migrated files are not showing up as "unused."

I'm worried that my repo is going to bloat with unused files with the SHA256 backend, which mysteriously do not show up as "unused" in git annex unused, if I migrate any more.

any ideas what piece of the puzzle I could be missing here?


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="99.54.57.201"
 subject="comment 1"
 date="2013-01-28T03:20:38Z"
 content="""
Just noticed this -- http://git-annex.branchable.com/bugs/git_annex_migrate_leaves_old_backend_versions_around/

so I don't need to worry about disk space bloat; they're all just hardlinks.  That's cool.  Still would be nice to know why they don't show up as \"unused.\"  But not as important.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 2"
 date="2013-01-28T06:28:28Z"
 content="""
It won't show as unused as long as there are git refs that still refer to the file. That could be branches, or more likely you have not synced everything to the local repo yet.
"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 4"
 date="2013-01-28T14:30:55Z"
 content="""
If by \"synced to the local repo\" you mean run a git annex sync, then yes, I have done that.

Here's an example of what I'm seeing:

    annex$ readlink subdir/subdir/file.txt
    ../../.git/annex/objects/zV/57/SHA256E-s3952--cb67d2cbadfea373ed8aa5ec434f246acccb33471a7e28a444138cc4fe8552c2.txt/SHA256E-s3952--cb67d2cbadfea373ed8aa5ec434f246acccb33471a7e28a444138cc4fe8552c2.txt
    annex$ find .git/annex/objects/ -name SHA256-s3952--cb67d2cbadfea373ed8aa5ec434f246acccb33471a7e28a444138cc4fe8552c2
    .git/annex/objects//1w/Q4/SHA256-s3952--cb67d2cbadfea373ed8aa5ec434f246acccb33471a7e28a444138cc4fe8552c2
    .git/annex/objects//1w/Q4/SHA256-s3952--cb67d2cbadfea373ed8aa5ec434f246acccb33471a7e28a444138cc4fe8552c2/SHA256-s3952--cb67d2cbadfea373ed8aa5ec434f246acccb33471a7e28a444138cc4fe8552c2
    annex$ git annex sync
    commit
    # On branch master
    nothing to commit, working directory clean
    ok
    pull homeworld
    ok
    push homeworld
    Everything up-to-date
    ok
    annex$ git annex unused
    unused . (checking for unused data...) (checking master...) (checking sync/master...) (checking homeworld/master...) (checking toshiba/master...) (checking toshiba/synced/master...) ok
    annex$ find .git/annex/objects/ -name SHA256-s3952--cb67d2cbadfea373ed8aa5ec434f246acccb33471a7e28a444138cc4fe8552c2
    .git/annex/objects//1w/Q4/SHA256-s3952--cb67d2cbadfea373ed8aa5ec434f246acccb33471a7e28a444138cc4fe8552c2
    .git/annex/objects//1w/Q4/SHA256-s3952--cb67d2cbadfea373ed8aa5ec434f246acccb33471a7e28a444138cc4fe8552c2/SHA256-s3952--cb67d2cbadfea373ed8aa5ec434f246acccb33471a7e28a444138cc4fe8552c2
    annex$

As you can see, there are two remotes, \"toshiba\" and \"homeworld\".  Neither of them is bare, they have checked out \"master\" branches.  Could they be the reason that some of this data isn't \"unused\"?  (I ask because I notice it reports that it is checking homeworld/master and toshiba/master when it does the unused check)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.3.125"
 subject="comment 4"
 date="2013-02-05T19:16:48Z"
 content="""
If you have not yet run `git annex sync` on the remotes, then they are still referring to the old unmigrated keys, which is why `unused` will not remove them.
"""]]


code: Is that really unsafe?  Because I did that and the switch to direct mode seemed to leave a lot of files still as links, and when I panicked and switched back to indirect mode, a whole lot of stuff seemed to have become unannexed and I reannexed it.


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="99.54.57.201"
 subject="comment 1"
 date="2013-01-25T03:39:16Z"
 content="""
This may have been an unrelated issue -- I just realized that I somehow ended up with a bunch of files in the annex under the directory \"misc\" and others under the directory \"Misc\" -- I'm not sure how they got that way, since I'm on OS X and those are supposed to be the same directory -- but they get synced over to a linux box where they're different directories.  This may be the source of some of the weirdness I saw; I'm not sure.  I tried to fix it all up on the linux box so everything was in its own directory.

"""]]


code:
I have a music repository(direct mode) in my Nexus 5 which I want to sync with remote repository(indirect mode).

When I run 'git annex sync --content', it did not sync the content but when I changed remote repository to direct mode, content got synced.

Do I need to set some configuration ?
Is it possible to sync content between direct and indirect mode repositories ?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 1"
 date="2014-03-10T17:57:51Z"
 content="""
There is no difference between direct and indirect mode repositories when it comes to communication between repositories, so they can be used in any combination.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnDXvDvWTXmCqQ90ATTD4dV3Ii4YbnE8E0"
 nickname="sys"
 subject="git annex direct when not all content is in this repo"
 date="2014-03-12T14:41:53Z"
 content="""
what does git annex direct do when all the content isn't available in the current repo (i am assuming it will leave symlinks for the missing content)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.154"
 subject="comment 3"
 date="2014-03-12T17:03:30Z"
 content="""
Direct mode does not require all content be present any more than indirect mode does, and missing content is represented the same in both modes, with broken symlinks.
"""]]


code:
I'm having an issue with 2 repos: one on my laptop, the other on my NAS. Both are in indirect mode, running Arch Linux, and have the latest Git version. Laptop has git-annex 5.20140411-gda795e0, NAS has 5.20140319-g9aa31b7 (from prebuilt tarballs).

The issue is quite simple. When I `git-annex add` new files on my laptop, commit them, and then `git-annex sync` them, they show up as staged for deletion on my NAS.

    laptop $ git annex add some-file
    laptop $ git commit -m "Add some-file"
    laptop $ git annex sync
    commit  ok
    pull ds413j
    ok
    push ds413j
    Counting objects: 133, done.
    Delta compression using up to 8 threads.
    Compressing objects: 100% (78/78), done.
    Writing objects: 100% (80/80), 10.64 KiB | 0 bytes/s, done.
    Total 80 (delta 12), reused 0 (delta 0)
    To ssh://**/**
       1dcd188..8ef4249  git-annex -> synced/git-annex
       c0f45a6..21711d6  master -> synced/master
    ok
    laptop $ ssh $NAS
    nas $ git status
    On branch master
    Changes to be committed:
      (use "git reset HEAD <file>..." to unstage)

            deleted:    some-file

    nas $

If I run `git annex sync` on the NAS, it will create a new commit that deletes that file. So I have to play with `git reset`/`git checkout` by hand to make sure that the new file won't be deleted.

I'm not sure when this started, but I think it was after I did some stupid mistake (`git checkout -B master synced/master`, kill a `git annex sync` with Ctrl+C, or something else that even resulted in my non-bare repo to have "bare=true" in .git/config...). And I haven't yet been able to fix this.

Any idea what can have caused this, how to fix it, and how to prevent it from happening again in the future?

.git/config on NAS:

    [core]
            repositoryformatversion = 0
            filemode = true
            logallrefupdates = true
    [annex]
            uuid = d54ae60a-1f59-403c-923f-32ea3bf2d00f
            version = 5
            diskreserve = 1 megabyte
            autoupgrade = ask
            debug = false

.git/config on laptop:

    [core]
            repositoryformatversion = 0
            filemode = true
            bare = false
            logallrefupdates = true
    [branch "master"]
    [annex]
            uuid = f20cb506-945d-4c78-af1a-0aa884bb899b
            version = 5
            diskreserve = 20 gigabytes
            autoupgrade = ask
            debug = false
            expireunused = 7d
            genmetadata = true
    [push]
            default = matching
    [remote "ds413j"]
            url = ssh://**/**
            fetch = +refs/heads/*:refs/remotes/ds413j/*
            annex-uuid = d54ae60a-1f59-403c-923f-32ea3bf2d00f
            annex-sync = true


code:
[[!comment format=mdwn
 username="http://schnouki.net/"
 nickname="Schnouki"
 subject="comment 1"
 date="2014-04-17T12:16:41Z"
 content="""
Here's the output of `git annex sync --debug` (for a different commit): <http://ix.io/bJZ>
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.191"
 subject="comment 2"
 date="2014-04-17T19:28:59Z"
 content="""
`git annex sync` will push changes to your NAS, but this does not cause the work tree there to be updated. You must be running some command on the NAS that gets it work tree into the state you show. You need to tell us what that command is, since it seems to be where things are going wrong. Maybe you're running the git-annex assistant on the NAS, or maybe you run `git annex sync` on the NAS.
"""]]


code:
[[!comment format=mdwn
 username="http://schnouki.net/"
 nickname="Schnouki"
 subject="comment 3"
 date="2014-04-22T14:59:47Z"
 content="""
Not running the assistant (freshly rebooted NAS, checked with `ps ax | grep annex`), not running any other git command. Still happened, until a few minutes ago: I found the cause for this issue... and it's quite frustrating.

Every now and then, I interrupt a running `git annex sync` with Ctrl+C. And sometimes this causes my NAS repository to end up with `bare = true` in its `.git/config`. When this happens, I just remove the offending line and call it a day. Now I just added a `bare = false` line to that file, and that solved the problem. It seems that somehow, if you don't explicitely tell git that your repo is not bare, it considers it as a bare repository -- and the final push from `git annex sync` actually updates the `master` branch without updating the work tree.

This is probably not a bug in git-annex, but rather a weird behaviour in git -- and for sure something I didn't expect to happend. But anyway it works now :) Leaving this here in case it helps other people.

Joey, thanks again for your time and your help.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.114"
 subject="comment 4"
 date="2014-04-24T18:22:16Z"
 content="""
git sets bare = false in newly created non-bare repositories. Perhaps this is why? It seems likely that git has an inconsistent default if there's no configuration. Sounds worth filing a bug on git for. OTOH, messing with .git/config  in nonstandard ways seems like a good way to cause yourself random pain too.
"""]]


code:
Hi,

I synchronize home directories on a couple of machines using unison.  Is there a recommended way to do the equivalent using git-annex?

Thanks,

John


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-03-05T20:34:22Z"
 content="""
There are a few problems you are likely to run into if you try to store everything in git-annex:

* git does not store certain things, including special file permissions, in a git repository
* you can't check a .git directory into another git repository
* git-annex locks files to prevent them being changed and this involves replacing them with symlinks (much less of a problem now that it has the direct mode)

For these reasons, I currently encourage users to use one or more git-annex repositories to store and sync whatever large data files they need, but to confine this to subdirectories of the home directory. There are other tools like vcsh that are better suited to storing things like dotfiles in git, rather than using git-annex for that. And other tools like mr can make it easy to work with a set of repositories, so you can have your dotfiles stored in one or more repositories and your large files in others and update them all with one command.

See <http://vcs-home.branchable.com/> for more on this..
"""]]


code:
I have a bunch of directory trees with large data files scattered over various computers and disk drives - they contain photos, videos, music, and so on.  In many cases I initially copied one of these trees from one machine to another just as a cheap and dirty backup, and then made small modifications to both trees in ways I no longer remember. For example, I returned from a trip with a bunch of new photos, and then might have rotated some of them 90 degrees on one machine, and edited or renamed them on another.

What I want to do now is use git-annex as a way of initially synchronising the trees, and then fully managing them on an ongoing basis.  Note that the trees are *not* yet git repositories.  In order to be able to detect straight-forward file renames, I believe that [[the SHA1 backend|tips/using_the_SHA1_backend]] probably makes the most sense.

I've been playing around and arrived at the following setup procedure.  For the sake of discussion, I assume that we have two trees `a` and `b` which live in the same directory referred to by `$td`, and that all large files end with the `.avi` suffix.

     # Setup git in 'a'.
     cd $td/a
     git init

     # Setup git-annex in 'a'.
     echo '* annex.backend=SHA1' > .gitattributes
     git add .gitattributes
     git commit -m'use SHA1 backend'
     git annex init

     # Annex all large files.
     find -name \*.avi | xargs git annex add
     git add .
     git commit -m'Initial import'

     # Setup git in 'b'.
     cd $td/b
     git clone -n $td/a new
     mv new/.git .
     rmdir new
     git reset # reset git index to b's wd - hangover from cloning from 'a'

     # Setup git-annex in 'b'.
     # This merges a's (origin's) git-annex branch into the local git-annex branch.
     git annex init

     # Annex all large files - because we're using SHA1 backend, some
     # should hash to the same keys as in 'a'.
     find -name \*.avi | xargs git annex add
     git add .
     git commit -m'Changes in b tree'

     git remote add a $td/a

     # Now pull changes in 'b' back to 'a'.
     cd $td/a
     git remote add b $td/b
     git pull b master

This seems to work, but have I missed anything?


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2011-12-14T17:31:31Z"
 content="""
This is an entirely reasonable way to go about it.

However, doing it this way causes files in B to always \"win\" -- If the same filename is in both repositories, with differing content, the version added in B will superscede the version from A. If A has a file that is not in B, a git commit -a in B will commit a deletion of that file.

I might do it your way and look at the changes in B before (or even after) committing them to see if files from A were deleted or changed.

Or, I might just instead keep B in a separate subdirectory in the repository, set up like so:

<pre>
mv b old_b
git clone a b
cd b
mv ../old_b .
git annex add old_b --not --exclude '*.avi'
</pre>

Or, a third way would be to commit A to a branch like branchA and B to a separate branchB, and not merge the branches at all.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmKPMUX0YHBjE93eBsEnacwZsddSDue3PY"
 nickname="Oliver"
 subject="comment 2"
 date="2011-12-23T22:04:08Z"
 content="""
As joey points out the problem is B overwrites A, so that any files in A that aren't in B will be removed. But the suggestion to keep B in a separate subdirectory in the repository means I'll end up with duplicates of files in both A and B. What I want is to have the merged superset of all files from both A and B with only one copy of identical files.

The problem is that unique symlinks in A/master are deleted when B/master is merged in. To add back the deleted files after the merge you can do this:

    git checkout master~1 deleted_file_name                                                              #checkout a single deleted file called deleted_file_name
    git diff master~1 master --numstat --name-only --diff-filter=D                                       #get the names of all files deleted between master and master~1
    git diff master~1 master --numstat --name-only --diff-filter=D | xargs git checkout master~1         #checkout all deleted files between master and master~1

Once the first merge has been done after set up, you can continue to make changes to A and B and future merges won't require accounting for deleted files in this way.
"""]]


code:
[[!comment format=mdwn
 username="http://adamspiers.myopenid.com/"
 nickname="Adam"
 subject="re-annexing previously annexed files"
 date="2012-03-29T21:41:54Z"
 content="""
Here's another handy command-line which annexes all files in repo B which have already been annexed in repo A:

    git status --porcelain | sed -n '/^ T /{s///;p}' | xargs git annex add

The 'T' outputted by git status for these files indicates a type change: it's a symlink to the annex in repo A, but a normal file in repo B.

"""]]


code:
I try to sync my taskWarrior files .task/*.data with git-annex ... but there is two problem :

- i need to chmod 755 my files because taskWarrior doesn't recognize them, or say "problem with permission"

- taskwarrior seems crazy with symbolic link used by git-annex, undo not work, task appear multiple times , etc.

Is there any solution ?
Any user experienced the same problem?

Thanks
Sr.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 1"
 date="2013-08-06T12:44:11Z"
 content="""
* You could try metastore (not merge friendly), git-cache-meta (pretty minimal) or metamonger (not done yet) to sync your file permissions
* Look into direct mode to avoid symlinks
* Alternatively, check your taskwarrior files into git, not git-annex, to avoid symlinks
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/FHnTlSBo1eCGJRwueeKeB6.RCaPbGMPr5jxx8A--#ce0d8"
 nickname="Hamza"
 subject="comment 2"
 date="2014-04-14T15:27:48Z"
 content="""
Using direct mode would replace symlinks with actual files.
"""]]


code:
Hi

I have read some info about telehash. It looks verry promising. I was wondering though how syncing will work. For example. I have 2 computers. Normal PC and a laptop. Mostly only one is on at a time.

* Sync messages will be sent over telehash protocoll ?
* What if I push some changes (they will be synced to a shared repository) and laptop is not online. How will git-annex know what to sync from a shared repository ?
* Do you plan to send files/commits directly to online clients ? If 2 friends are online at the same time.
* What will happen with data on a shared repository if all clients have synced content ? Will it be deleted since it is not longer needed ?

I was thinking of a model where you sync directly (if possible), and just drop shared content to repo for offline users. Whan everyone have pulled content it may be removed from shared repo.


code:
Tell your git-annex stories here. Feel free to give as little or as much detail
as appropriate about how you're using it. How's it working out for you?

See [[testimonials]] for some other stories.

[[!meta author=Joey]]


code:
[[!comment format=mdwn
 username="http://mildred.fr/"
 subject="just amazing"
 date="2012-04-12T17:12:41Z"
 content="""
git-annex is just amazing. I just started using it and for once, I have hope to be able to organize my files a little better than now.

Currently, I have a huge homedir. From time to time, I move file away in external hard drives, then forget about them. When I want to look at them back, I just can't because I have forgotten where they are. I have also a ton of files on those drives that I can't access because they are not indexed. With git-annex I have hope to put all of these files on a git repository. I will be able to see them everywhere, and find them when I need to.

I might stop loosing files for once.

I might avoid having multiple copies of the same things over and over again, without knowing so. and regain some more disk space.

For the moment, I'm archiving my photographs. But there is one thing that might not go very well: directory hierarchies where everything is important (file owner, specific permissions, symlinks). I won't just be able to blindly annex all of these files. But for the moment I'll stick at archiving ocuments and it should be amazing.

[Mildred](http://mildred.fr)
"""]]


code:
[[!comment format=mdwn
 username="http://cgray.myopenid.com/"
 nickname="cgray"
 subject="comment 2"
 date="2012-04-14T01:18:53Z"
 content="""
Git-annex has really helped me with my media files.  I have a big NAS drive where I keep all my music, tv, and movies files, each in their own git annex.  I tend to keep the media that I want to watch or listen to on my laptop and then drop it when it is done.  This way I don't have too much on my laptop at any one time, but I have a nice selection for when I'm traveling and don't have access to my NAS.

Additionally, I have a mp3 player that will format itself randomly every few months or so.  I keep my podcasts on it in a git annex and in a git annex on my laptop.  When I am done with a podcast, I can delete it from the mp3 player and then sync that information with my laptop.  With this method, I have a backup of what should be on my mp3 player, so I don't need to worry about losing it all when the mp3 player decides it's had enough.
"""]]


code:
[[!comment format=mdwn
 username="http://bergey.dreamwidth.org/"
 ip="66.80.90.109"
 subject="git-annex, media, emacs"
 date="2012-08-31T01:47:36Z"
 content="""
I've been using git-annex for a month or two to manage my music collection.  It's great to know I can drop files if I need space on the laptop, knowing exactly where I have other copies.  Now I'm writing an emacs mode to help me keep track: https://gitorious.org/emacs-contrib/annex-mode

Locally available files are colored differently, and pressing g runs `git annex get` on the file at point.
"""]]