code: My current setup involves 3 computers, one desktop one laptop and a vps. In my current setup I have created annex repos on the server and cloned from it, both machines sync to the server. All three use non base repos. Now I have another annex folder on the desktop that I would like to sync between the three. Both machines are behind NAT so server can not communicate with the machines. In order to init the repo on the vps, I was thinking of setting up a temporary VPN/port forward between the desktop and the VPS then clone from the desktop finally remove the remote section in .git/config on the server so VPS becomes the master again. First of all is there an easier way to do this? if not is it safe to do this? or is it going to cause problems down the line.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.63"
 subject="comment 1"
 date="2013-08-26T18:46:06Z"
 content="""
You could certainly do that. I don't think it's the easiest way.

Note that this is essentially a git question. It really has nothing to do with git-annex, unless you want to use the git-annex assistant, which can sync a repository over XMPP without needing a central git repository at all.

If I had this problem with git in general, I would make a new empty repository on the server, and push the local repository I have on the one machine to it. Then on the other machine, I would clone from the server. Problem solved, I think?
"""]]

repo_name: kakafarm/git

license: unknown


code:
I am just returning to git-annex after some time. The last time I was using it, the [[/todo/direct_mode_guard]] was not yet in place, and I could supply my own commit messages even in a direct mode repository using `git commit`. Now that the guard is in place, is there a built-in way to provide a commit message that will be used instead of "git-annex automatic sync" by `git annex sync`?

I have tried doing a manual commit with `git --git-dir=/path/to/local/.git --work-tree=/path/to/local -c core.bare=false commit -m 'Manual Commit Message'`, and it seems to work. Does this do the same thing as the part of `git annex sync` that commits any outstanding changes before syncing with remotes?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.176"
 subject="comment 1"
 date="2014-05-27T16:10:29Z"
 content="""
Yes, you can use the `-c core.bare=false` trick to bypass the direct mode guard if you want to. Just bear in mind that it's very easy to shoot yourself in the foot by passing -a or filenames to git commit, which then checks the whole large file into git..
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnxvt4mc32O0ctvPGhhII_ZKsHUiaD2o7I"
 nickname="William"
 subject="comment 2"
 date="2014-05-28T04:40:27Z"
 content="""
Joey, thanks for clarifying that. I learned about the perils of using regular git commands in direct mode before the guard was implemented, so I will use with care. Specifying a commit message is really the only case where I am tempted to bypass the the `git annex` subcommands in direct mode.

"""]]


code:
Hi,

I've recently set-up a server which uses Southpaw's Tactic DAM system and I've initialised a git-annex directory using the assistant which will manage any files which Tactic puts into the git-annex. I plan to make some remote repositories to Amazon S3, friends and some local machines on my home network. The server is running Ubuntu 64-bit and so I've written an upstart job which runs 'git-annex assistant --autostart' as the user 'git-annex' as this user doesn't log-in and run the xdg autostart .desktop at all.

I saw that you can set the purpose of each repository which it will sync to and noticed 'Manual Mode.' From the description, it seems it will only work if I do explicit git-annex commands to it which would be perfect for me as I'd like to write tools which run git-annex add/get/drop/etc. manually on some remotes like ones to friends as I don't want them to sync to everything or any files that they produce, only files which they request with a special tool that I'll write and have Tactic marshal the file changes/names/etc.

I've set those remotes to manual-mode via the assistant and tried copying a file to the remote's directory, but it auto-synced the file anyway. Maybe I'm getting confused at how manual mode works but I'd like to only explicitly set which files to add to the repo by a command and not just 'any' file which gets placed into that directory. If this is more of a wishlist request I guess just move this post into there and I'll reword it as a wish request.

I don't need to use the assistant if that makes more sense, but I would like to be able to still monitor things as the webapp makes for a great GUI to check for that stuff. I can't wait for the https version  of webapp too, I currently run git-annex webapp --listen=<ip>:<port> and then run the link it outputs on my desktop machine to manage it from the server.

Thanks, and this is such a great bit of software, especially as my Internet connection is really bad for 2013 standards, and having the option for friends/remote servers to sync up via an encrypted S3 or box.com account is great!


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.63"
 subject="comment 1"
 date="2013-08-24T16:20:29Z"
 content="""
In the assistant, putting a repository in manual mode prevents the assistant from automatically downloading every file that is added to other clones of the repository. As you've noticed, it does not prevent the assistant from automatically adding new files that are put into the local repository, or from uploading those files to other clients that want them (ie, clients not themselves set to use manual mode).

You can prevent the assistant from noticing when you add new files to a repository by clicking on \"syncing enabled\" in the first repository in the list. (The repository labeled as \"here\".) You can then `git annex add` the files you want to add by hand, and manually `copy annex copy` them to other repositories, and manually `git annex sync`.
"""]]


code:
Hello,

I have a Samsung ARM Chromebook running Ubuntu with crouton (chroot in chromeos). I want to use git-annex, and add the chromebook as a second "device" to my already set up git-annex repo on my main pc. That repo is one folder set up with the assistant, and it has one encrypted ssh(rsync) remote.

I've read that the webapp is not available on ARM, and I've also found a topic with a simple script to replicate git-annex-assistant steps: http://git-annex.branchable.com/forum/Running_assistant_steps_manually/. So, I've used the `cabal` steps to install git-annex manually, which is working.

I want to use git-annex in a dropbox-like fashion, folder and file sync between these two devices (and maybe later on another encrypted ssh remote). However I'm in need of some guidance regarding the command line usage and such.

My questions are:

- How do I "add" the chromebook repo to the existing git-annex setup?
- The machines are not on at the same time. One is my main workstation and the chromebook is my "mobile friend". The encrypted ssh remote is on a VPS, that is always on.
  - How do I configure the syncing between the main machine, the chromebook and the encrypted ssh remote on the vps? If possible, bi-directional.

If it is possible, both explanation and command line examples would help a lot.


code:
When copying to a megaannex remote, it hangs after a while, and I get this:

    copy Boomarks/Android (gpg) Traceback (most recent call last):
      File "/usr/local/bin//git-annex-remote-mega", line 511, in <module>
        common.startRemote()
      File "/home/zack/megaannex/lib/CommonFunctions.py", line 557, in startRemote
        sys.modules["__main__"].checkpresent(line)
      File "/usr/local/bin//git-annex-remote-mega", line 483, in checkpresent
        folder = setFolder(conf["folder"], common.ask("DIRHASH " + line[1]))
      File "/usr/local/bin//git-annex-remote-mega", line 401, in setFolder
        folder = createFolder(conf["folder"], 2)
      File "/usr/local/bin//git-annex-remote-mega", line 378, in createFolder
        res = m.create_folder(subject, folder)
      File "/usr/lib/python2.7/site-packages/mega/mega.py", line 617, in create_folder
        'i': self.request_id})
      File "/usr/lib/python2.7/site-packages/mega/mega.py", line 110, in _api_request
        timeout=self.timeout)
      File "/usr/lib/python2.7/site-packages/requests/api.py", line 88, in post
        return request('post', url, data=data, **kwargs)
      File "/usr/lib/python2.7/site-packages/requests/api.py", line 44, in request
        return session.request(method=method, url=url, **kwargs)
      File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 354, in request
        resp = self.send(prep, **send_kwargs)
      File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 460, in send
        r = adapter.send(request, **kwargs)
      File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 250, in send
        raise SSLError(e)
    requests.exceptions.SSLError: The read operation timed out
    (external special remote protocol error, unexpectedly received "" (unable to parse command)) failed

Any help would be appreciated, thanks!


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 1"
 date="2014-07-11T19:42:24Z"
 content="""
This looks to me like the connection to Mega fails, so either they have a server down, or a network problem, or perhaps the API that the megaannex remote was relying on might have been removed..
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmqaNwDQ367zpW6cIRviLz6zJZZFODgoEI"
 nickname="Zack"
 subject="comment 2"
 date="2014-07-19T12:48:08Z"
 content="""
I believe it may have been removed, unfortunately. :(
"""]]


code:
Hi,

I've said up two clients to sync locally as well as over ssh-rsync.

However, locally one client is complaining about missing `runshell` at a location where `git annex` is not installed.  The log is below.  On the assistent overview it just says "unfinished repository".  Note that I am able to ssh into the other machine and from the other machine I'm also able to ssh into this machine.

I'm using the latest prebuild binary package (git-annex-bin at Archlinux AUR).  So my `git annex` setup contains

    $> pacman -Ql git-annex-bin
    git-annex-bin /usr/
    git-annex-bin /usr/bin/
    git-annex-bin /usr/bin/git-annex
    git-annex-bin /usr/bin/git-annex-shell

    $> pacman -Qi git-annex-bin # with chops
    Name           : git-annex-bin
    Version        : 4.20130909-1
    Description    : Precompiled version of git-annex, webapp and assistant
    included.
    Architecture   : x86_64

Here is a log of one event where it fails to transfer.

    [2013-09-28 18:06:38 CEST] TransferScanner: queued Upload NoUUID config/emacs.d/ac-l-dict/amsmath-c-a-* Nothing : expensive scan found missing object
    [2013-09-28 18:06:38 CEST] Transferrer: Transferring: Upload NoUUID config/dotfiles/stardict/dic/stardict-oxford-2.4.2/oxford.ifo Nothing
    [2013-09-28 18:06:38 CEST] call: git-annex ["transferkeys","--readfd","96","--writefd","94"]
    [2013-09-28 18:06:38 CEST] read: git ["--git-dir=/home/rasmus/annex/.git","--work-tree=/home/rasmus/annex","show-ref","git-annex"]
    [2013-09-28 18:06:38 CEST] read: git ["--git-dir=/home/rasmus/annex/.git","--work-tree=/home/rasmus/annex","show-ref","--hash","refs/heads/git-annex"]
    [2013-09-28 18:06:38 CEST] read: git ["--git-dir=/home/rasmus/annex/.git","--work-tree=/home/rasmus/annex[","log2","0r1e3f-s/0h9e-ad2s8/ g1i8t-annex..90df78a0910a6f2998655e6:06:38 CEST] 127.0.0.1 GET /transfers/NotificationId%201 Mozilla/5.0 (X11; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0 Aurora/26.0a2
    ea47d478ea0589b54","--oneline","-n1"]
    [2013-09-28 18:06:38 CEST] read: git ["--git-dir=/home/rasmus/annex/.git","--work-tree=/home/rasmus/annex","log","refs/heads/git-annex..5f6870ed24e5ded1764765bbfef2b85aff046569","--oneline","-n1"]
    [2013-09-28 18:06:38 CEST] chat: git ["--git-dir=/home/rasmus/annex/.git","--work-tree=/home/rasmus/annex","cat-file","--batch"]
    [2013-09-28 18:06:38 CEST] read: ssh ["-S","/home/rasmus/annex/.git/annex/ssh/00b8ef4cb08290718ba625d9bd86ca0b","-o","ControlMaster=auto","-o","ControlPersist=yes","-T","rasmus@git-annex-192.168.1.107-rasmus_annex","git-annex-shell 'configlist' '/~/annex/'"]
    /home/rasmus/.ssh/git-annex-shell: line 4: /opt/git-annex.linux/runshell: No such file or directory
    [2013-09-28 18:06:38 CEST] call: git ["--git-dir=/home/rasmus/annex/.git","--work-tree=/home/rasmus/annex","fetch","--quiet","192.168.1.107_annex"]
    [2013-09-28 18:06:38 CEST] 127.0.0.1 GET /transfers/NotificationId%201 Mozilla/5.0 (X11; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0 Aurora/26.0a2
    [2013-09-28 18:06:38 CEST] 127.0.0.1 GET /log Mozilla/5.0 (X11; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0 Aurora/26.0a2
    /home/rasmus/.ssh/git-annex-shell: line 4: /opt/git-annex.linux/runshell: No such file or directory
    fatal: Could not read from remote repository.

    Please make sure you have the correct access rights
    and the repository exists.
    git-annex: Unknown UUID


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.8.80"
 subject="comment 1"
 date="2013-09-30T16:08:40Z"
 content="""
You apparently at one point installed git-annex from the prebuilt tarball, in `/opt/git-annex.linux/`. It set up a ~/.ssh/git-annex-shell that points to this location. If you no longer have it installed that way, you can remove that file.
"""]]


code:
I've noticed how, when files are synchronized/transferred from one repository (annex?) to another by the assistant, the files in the «remote» repository do no retain the modification time of the original files.
From what I know git does not save file-specific metadata (ownership/timestamps/etc.).
Since I've just started using git-annex, and I'd very much like for backup copies to retain the mtime of the originals, I would like to ask whether I am experiencing a malfunction or whether this is expected behaviour.

Regards


code:
Is there a way to move a large file without "git annex unlock"ing it (Which takes very long for copying and then rehashing the file)?
Using a simple "mv" or "git mv" results in broken symlinks, if the target directory is on a different hierarchy level:

Example:

Initial State:

    $ ls -Hl Pictures:
    Pictures/showImages.jpg -> ../.git/annex/objects/90/32/SHA1-s8737--a8bfb285d0ae394cb75c86f1eb9f703fb678a51e/SHA1-s8737--a8bfb285d0ae394cb

Move:

    $ mv Pictures/showImages.jpg .

Result:

    $ ls -H showImages.jpg
    ls: cannot access showImages.jpg: No such file or directory


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn7gQ1zZDdWhXy9H51W2krZYShNmKL3qfM"
 nickname="Karsten"
 subject="comment 1"
 date="2012-11-28T13:28:21Z"
 content="""
Just commit, git-annex will fix the symlinks. Or do it manually with 'git annex fix'.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlgyVag95OnpvSzQofjyX0WjW__MOMKsl0"
 nickname="Sehr"
 subject="Does not work"
 date="2012-11-28T14:50:16Z"
 content="""
    $ git annex fix

Does not change anything. and

    $ git annex fix showImages.jpg
    git-annex: showImages.jpg not found
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn7gQ1zZDdWhXy9H51W2krZYShNmKL3qfM"
 nickname="Karsten"
 subject="comment 3"
 date="2012-11-28T15:03:42Z"
 content="""
In my tests 'git annex fix' works only on symlinks in directories alredy known to git. So if you created a new directory inside your annex, be sure to 'git add' the directory before running 'git annex fix'.

HTH,
Karsten
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlgyVag95OnpvSzQofjyX0WjW__MOMKsl0"
 nickname="Sehr"
 subject="comment 4"
 date="2012-11-28T15:14:49Z"
 content="""
Thanks. It seems that the file itself has to be added to git annex to be able to bet git annex fixed!
"""]]


code:
Hi,
Is there any way I can move or copy file content of older version without doing checkout to that version, by passing commit hash as parameter in move command itself?

Thank you


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnpdM9F8VbtQ_H5PaPMpGSxPe_d5L1eJ6w"
 nickname="Rafael"
 subject="comment 1"
 date="2012-05-15T07:59:57Z"
 content="""
I had a similiar question in forum/new_microfeatures/. I would like to fetch/copy all the annexed content from a repo, be it on the current branch, another branch, or corresponds to an old version of a file. A command like \"git annex copy --all --from=source [path]\" would then ensure I have access to all the content I need even if I have later no longer access to source. Sure I could use rsync.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.2.51"
 subject="comment 2"
 date="2012-05-15T17:00:10Z"
 content="""
Yes, I think that [[todo/add_-all_option]] is the right approach for this. Seems unlikely you'd have some files' hashes handy without having them checked out, but operating on all content makes sense.

That page discusses some problems implementing it for some commands, but should not pose a problem for `move`. It would also be possible to support `get` and `copy`, except `--auto` couldn't be used with `--all`. Even `fsck` could support it.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.193"
 subject="comment 3"
 date="2013-07-03T18:10:29Z"
 content="""
Now the --all option is built. You can, for example, run `git annex copy --all --to backup`
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-04-23T21:41:31Z"
 content="""
This is a lot of stuff. It would be better to file individual detailed bug reports if you want any of this looked at in depth. Otherwise, I just read it to get an impression of how things are working, or not, for you, and try to hit some of the main points:

* The OSX app icon behavior is probably because OSX does not realize that this is supposed to be a daemon. I don't know how to fix that. Clicking on the app icon is supposed to launch the web app, and has been reported to do so by others. I have no access to an OSX desktop.

* Transfer repositories only keep files that have not yet reached all your client repositories. However, it's possible that files
  may be uploaded to a transfer repository unnecessarily (if there is some other path for the file to get to the other client
  repositories, ie if you've locally paired them). In this case, the file will be uploaded and then later removed from the transfer
  repository, without being used.

* I have never seen encrypted filenames appear in the webapp. I don't see how that could happen TBH.

* git-annex logs information to `.git/annex/debug.log` within your repository. You can go to Configuration -> Preferences to enable
  additional debug logging. I absolutely need these logs to help debug almost any problems with the assistant.

* \"Clicking a couple of hundred play buttons\" could result in it trying to make a couple of hundred uploads to box.com
  at the same time. This is probably not a good idea.

* The fragment of log you posted shows me some problem that caused gpg to fail to encrypt a file before sending it to somewhere,
  but it's lacking all context (and starts when the file transfer is already 87% complete!) so I can't say more. I need to see the whole log to make any informed guesses about what might be going on.
"""]]


code:
[[!comment format=mdwn
 username="EmanueleAina"
 ip="93.38.211.231"
 subject="comment 2"
 date="2013-09-11T17:55:14Z"
 content="""
I seem to hit the same issue (`ResponseTimeout`) on box.com, but I don't have any log file under .git/annex.

The timeout is quite unpredictable, sometimes I'm able to transfer less than 20% of the file while sometimes it fails after reaching 51%. The file is quite big, 6.2G.

Unfortunately, after the first upload fails, `git-annex` seems to think that the file has been uploaded successfully and will refuse to copy it again. Even `whereis` will list the box.com location.

I think there are really two bugs: the one triggering the timeouts, and the fact that `git-annex` thinks that the failing upload succeeded and is unable to recover the interrupted upload.

Here's the log I got on the shell. Let me know how I can provide any additional information needed. Thanks!

	$ git annex copy --verbose --debug home.tar.gz --to box.com
	[2013-09-11 17:42:52 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"show-ref\",\"git-annex\"]
	[2013-09-11 17:42:52 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
	[2013-09-11 17:42:52 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"log\",\"refs/heads/git-annex..1e029f92c5c82b094cbe70b55927996c31579e2e\",\"--oneline\",\"-n1\"]
	[2013-09-11 17:42:52 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"log\",\"refs/heads/git-annex..1090c21b121fd8b6e2ab49a772c8eab5235a3930\",\"--oneline\",\"-n1\"]
	[2013-09-11 17:42:52 CEST] chat: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"cat-file\",\"--batch\"]
	[2013-09-11 17:42:52 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"ls-files\",\"--cached\",\"-z\",\"--\",\"home.tar.gz\"]
	copy home.tar.gz (gpg) [2013-09-11 17:42:52 CEST] chat: gpg [\"--quiet\",\"--trust-model\",\"always\",\"--decrypt\"]
	(checking box.com...) (to box.com...)
	[2013-09-11 17:42:57 CEST] chat: gpg [\"--quiet\",\"--trust-model\",\"always\",\"--batch\",\"--passphrase-fd\",\"11\",\"--symmetric\",\"--force-mdc\",\"--no-textmode\"]
	35%      660.8KB/s 1h44mResponseTimeout
	gpg: [stdout]: write error: Broken pipe
	gpg: DBG: deflate: iobuf_write failed
	gpg: build_packet failed: file write error
	gpg: [stdout]: write error: Broken pipe
	gpg: iobuf_flush failed on close: file write error
	gpg: [stdout]: write error: Broken pipe
	gpg: iobuf_flush failed on close: file write error
	gpg: symmetric encryption of `[stdin]' failed: file write error
	git-annex: fd:13: hPutBuf: resource vanished (Broken pipe)
	failed
	git-annex: copy: 1 failed
	$ git annex copy --verbose --debug home.tar.gz --to box.com
	[2013-09-11 19:38:26 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"show-ref\",\"git-annex\"]
	[2013-09-11 19:38:26 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
	[2013-09-11 19:38:26 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"log\",\"refs/heads/git-annex..1e029f92c5c82b094cbe70b55927996c31579e2e\",\"--oneline\",\"-n1\"]
	[2013-09-11 19:38:26 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"log\",\"refs/heads/git-annex..1090c21b121fd8b6e2ab49a772c8eab5235a3930\",\"--oneline\",\"-n1\"]
	[2013-09-11 19:38:26 CEST] chat: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"cat-file\",\"--batch\"]
	[2013-09-11 19:38:26 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"ls-files\",\"--cached\",\"-z\",\"--\",\"home.tar.gz\"]
	copy home.tar.gz (gpg) [2013-09-11 19:38:26 CEST] chat: gpg [\"--quiet\",\"--trust-model\",\"always\",\"--decrypt\"]
	(checking box.com...) ok
	$ git annex whereis --verbose --debug home.tar.gz
	[2013-09-11 19:38:57 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"show-ref\",\"git-annex\"]
	[2013-09-11 19:38:57 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
	[2013-09-11 19:38:57 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"log\",\"refs/heads/git-annex..1e029f92c5c82b094cbe70b55927996c31579e2e\",\"--oneline\",\"-n1\"]
	[2013-09-11 19:38:57 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"log\",\"refs/heads/git-annex..1090c21b121fd8b6e2ab49a772c8eab5235a3930\",\"--oneline\",\"-n1\"]
	[2013-09-11 19:38:57 CEST] chat: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"cat-file\",\"--batch\"]
	[2013-09-11 19:38:57 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"ls-files\",\"--cached\",\"-z\",\"--\",\"home.tar.gz\"]
	whereis home.tar.gz (2 copies)
		d7db543e-5463-11e2-b7dd-9f423f798cc4 -- here (em@ocracy:/data/backup)
		e71fa45e-5463-11e2-a14e-93ca09c272da -- box.com
	ok
	$ git annex fsck --verbose --debug home.tar.gz
	[2013-09-11 19:39:05 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"ls-files\",\"--cached\",\"-z\",\"--\",\"home.tar.gz\"]
	[2013-09-11 19:39:05 CEST] chat: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"check-attr\",\"-z\",\"--stdin\",\"annex.backend\",\"annex.numcopies\",\"--\"]
	fsck home.tar.gz [2013-09-11 19:39:05 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"show-ref\",\"git-annex\"]
	[2013-09-11 19:39:05 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
	[2013-09-11 19:39:05 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"log\",\"refs/heads/git-annex..1e029f92c5c82b094cbe70b55927996c31579e2e\",\"--oneline\",\"-n1\"]
	[2013-09-11 19:39:05 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"log\",\"refs/heads/git-annex..1090c21b121fd8b6e2ab49a772c8eab5235a3930\",\"--oneline\",\"-n1\"]
	[2013-09-11 19:39:05 CEST] chat: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"cat-file\",\"--batch\"]
	(checksum...) [2013-09-11 19:39:05 CEST] read: sha256sum [\"/data/backup/.git/annex/objects/mG/kp/SHA256E-s6640317400--dcf0a535728f3f3f787db6339b740a4a6f6529e5ce1d238f28574499a8172670.tar.gz/SHA256E-s6640317400--dcf0a535728f3f3f787db6339b740a4a6f6529e5ce1d238f28574499a8172670.tar.gz\"]
	ok
	$ git annex fsck --verbose --debug home.tar.gz --from box.com
	[2013-09-11 19:42:15 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"show-ref\",\"git-annex\"]
	[2013-09-11 19:42:15 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
	[2013-09-11 19:42:15 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"log\",\"refs/heads/git-annex..1e029f92c5c82b094cbe70b55927996c31579e2e\",\"--oneline\",\"-n1\"]
	[2013-09-11 19:42:15 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"log\",\"refs/heads/git-annex..1090c21b121fd8b6e2ab49a772c8eab5235a3930\",\"--oneline\",\"-n1\"]
	[2013-09-11 19:42:15 CEST] chat: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"cat-file\",\"--batch\"]
	[2013-09-11 19:42:15 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"ls-files\",\"--cached\",\"-z\",\"--\",\"home.tar.gz\"]
	[2013-09-11 19:42:15 CEST] chat: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"check-attr\",\"-z\",\"--stdin\",\"annex.backend\",\"annex.numcopies\",\"--\"]
	fsck home.tar.gz (gpg) [2013-09-11 19:42:15 CEST] chat: gpg [\"--quiet\",\"--trust-model\",\"always\",\"--decrypt\"]
	(checking box.com...)
	[2013-09-11 19:42:28 CEST] chat: gpg [\"--quiet\",\"--trust-model\",\"always\",\"--batch\",\"--passphrase-fd\",\"10\",\"--decrypt\"]
	gpg: decrypt_message failed: eof
	ok

"""]]


code:
The instructions for building git-annex on [[install/Debian]] stable don't seem to be valid anymore.

1. `dpkg-checkbuilddeps` is looking for the wrong packages, e.g. libghc-missingh-dev instead of libghc6-missingh-dev.

2. Not all dependencies are available in the Squeeze repositories anymore (at least not Crypto and hS3), if I am not mistaken.


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2011-04-26T15:27:49Z"
 content="""
I have updated the instructions.
"""]]


code:
[[!comment format=mdwn
 username="gernot"
 ip="213.196.216.21"
 subject="comment 2"
 date="2011-04-26T18:56:44Z"
 content="""
Thanks for the update, Joey. I think you forgot to change libghc-missingh-dev to libghc6-missingh-dev for the copy & paste instructions though.

Also, after having checked that I have everything installed I'm still getting this error:

	...
	[15 of 77] Compiling Annex            ( Annex.hs, Annex.o )

	Annex.hs:19:35:
		Module `Control.Monad.State' does not export `state'
	make[1]: *** [git-annex] Error 1
	make[1]: Leaving directory `/home/gernot/dev/git-annex'
	dh_auto_build: make -j1 returned exit code 2
	make: *** [binary] Error 2

"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 3"
 date="2011-04-26T23:40:33Z"
 content="""
Both problems fixed.
"""]]


code:
Hi,
first sorry for my poor english it's not my native language.

I have one repository on my laptop and two repository on usb disk. Made with following  walkthrough (creating a repository and adding a remote).

Yesterday I have a backup of my repository on usb disk before add some file with

cd /media/usb/annex;git fetch laptop; git merge laptop/master&&git annex get .&&git annex sync

Now the repository on my usb disk is a mess.
Every file before the commit are lost.
For example :
 After the sync : file Z.7z
Z.7z: broken symbolic link to `../../../../.git/annex/objects/2K/49/SHA256-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'

The file ../../../../.git/annex/objects/2K/49/SHA256-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 doesn't exist.

On the repository before the sync (inside the Backup) :
file Z.7z
Z.7z: symbolic link to `../../../../.git/annex/objects/J1/f4/SHA256-s696365035--d2dcc67bf2f05fcfc7f42723b2d415d4f057a2eeadc282b40f5bc3724534f2f4/SHA256-s696365035--d2dcc67bf2f05fcfc7f42723b2d415d4f057a2eeadc282b40f5bc3724534f2f4'

The file ../../../../.git/annex/objects/J1/f4/SHA256-s696365035--d2dcc67bf2f05fcfc7f42723b2d415d4f057a2eeadc282b40f5bc3724534f2f4/SHA256-s696365035--d2dcc67bf2f05fcfc7f42723b2d415d4f057a2eeadc282b40f5bc3724534f2f4  still exist in the repository after the sync.

3 questions, if somebody could help me :

 - what I do wrong ?
 - why the- symlink for every file had change after "cd /media/usb/annex;git fetch laptop; git merge laptop/master&&git annex get .&&git annex sync"
 - how could I fix my repository ? recover file from the backup ? how ? Copy every file to start my repository from a new clean state ?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-04-22T20:00:12Z"
 content="""
1. Use `git log --stat` to find the commit that changed your file `Z.7z`
2. Use `git revert` to revert that commit

That will put things back the way they were.

You should then investigate what caused the bad commit to be made. It seems to have involved the file being reset to be 0 bytes in size.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkyQL8VPwrSf8LA0k8PKDWFCEKFH3O5Jds"
 nickname="marco"
 subject="Brvo"
 date="2013-04-22T23:33:10Z"
 content="""
Bravo. Mon Hero.

You save my data ! I can't find what happen but thanks a lot.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="glad I could help"
 date="2013-04-23T21:16:47Z"
 content="""
I'm curious if your repository is using direct mode, and is perhaps on a FAT filesystem, or other filesystem that does not support symlinks?

One way this could have happened is due to a bug in `git annex sync` when used on such a filesystem. I've just fixed this bug in commit 07580dc3dfb2f4d9af9a20bc172cbc32953c49d6
"""]]


code:
I've two huge annex repos (about 100G) with all my photos.
One is on my laptop and it uses indirect mode; the other one is on my NAS and I mount it into a CIFS folder, then it uses direct mode because of the crippled fs.
Each repo is remote of the other one. The NAS repo belongs to backup group with the standard preferred content.
I changed about 1000 photos in my PC repo (mainly renamed them) and I wanted to sync changes to the NAS repo, so I went through this sequence:

* "git annex add" and "git annex sync" on my PC => OK;

* "git annex sync" on the NAS repo => it started copying all the content BUT I stopped it because I had not enough time to wait at that very moment;

* again "git anne sync" on the NAS repo => I had more time to wait BUT it didn't resume the copy from the PC repo, it just said something like "all done, nothing to do"

* I checked that several files were missing on the NAS that I had modified/renamed on the PC repo so I did "git annex add" again on the PC => nothing new to add, nothing to do;

* I did again "git annex sync" on the PC repo => it deleted also from my PC repo all the photos that were also missing on the NAS repo;

* I've also checked for unused files on the PC repo and it now gives about 1000 files.

So here are my questions:

1. where was I wrong?

2. can I restore my photos on my PC repo (which uses indirect mode) ?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 1"
 date="2014-06-06T18:18:36Z"
 content="""
It's not clear from your description when the files got deleted or why, but you can use `git log --stat` to find out. Then you can `git revert` the problem commit.
"""]]


code:
[[!comment format=mdwn
 username="feulif"
 ip="151.27.145.89"
 subject="Thank you"
 date="2014-06-08T08:36:40Z"
 content="""
Thank you so much!
That solved my problem and restored my precious photos!

Now I see there are similar issues in the forum already, but I was searching for the bad keywords (like \"restore unused\") and I could not find them.

By the way, your software is outstanding.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 3"
 date="2014-06-09T19:44:49Z"
 content="""
What did the bad commit that you ended up reverting look like?

It seems to me that something must have went wrong due to your interrupting the first sync on the NAS. Perhaps this left it with newly added files not yet moved into place in the tree. So the next time you synced, this would cause it to see that the files seemed to have been deleted and make a commit reflecting that.

This, then is a bug in how direct mode handles merges. [[bugs/direct_mode_merge_interrupt]]
"""]]


code:
[[!comment format=mdwn
 username="feulif"
 ip="151.27.145.89"
 subject="comment 4"
 date="2014-06-10T18:43:34Z"
 content="""
Here is the log of the \"bad\" commit:

    commit 38e0e1ef009b820f008b46f53c24d9835ca0f9a6
    Author: me <me@myself.com>
    Date:   Wed Jun 4 00:04:37 2014 +0200

    git-annex automatic sync

     folder/subfolder1/subfolder2/photo.jpg          | 1 +
     [...]
     [...]
     [...]
     folder/subfolder1/subfolderN/video.MOV          | 1 +
     2266 files changed, 2266 insertions(+)

I stopped (ctrl-c) when it was adding the video.
"""]]


code:
I've started a another project to provide Emacs integration for git-annex: [[https://github.com/jwiegley/git-annex-el]]

My problem with the existing mode is that it didn't feel at all Emacsy, while my mode just piggy backs on existing paradigms, like using `C-x C-q` in a locked file to make it editable, and allowing you to browse and lock/unlock annexed files from Dired.


code:
New user? Can't figure out the basics? Add it here - what you wanted, what you tried.

#### I wanted to keep track of some files I had all organized in a directory outside of my ~/annex:

    $ cd ~/annex
    $ git annex add /path/to/some/photos
    fatal: '/path/to/some/photos' is outside repository

 But git-annex doesn't work that way. I had to do this instead

    $ rsync -a /path/to/some/photos
    $ git annex add photos
    (Recording state in git...)
    $ git annex status
    ... lots of helpful info...

#### I just have the OS/X app, can I do commandline stuff?

yes

    $ /Applications/git-annex.app/Contents/MacOS/git-annex add photos/
    (Recording state in git...)

but perhaps there is a better way.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.8.117"
 subject="comment 1"
 date="2012-12-17T16:21:36Z"
 content="""
For importing your photos, you can use `git annex import /path/to/some/photos` .. this will move them from the given location into the current directory, and add them to the annex. It preserves subdirectory structure too.

For using the OSX app at the command line, the only thing that is guaranteed to work properly is to run `/Applications/git-annex.app/Contents/MacOS/runshell`, which will modify your PATH and other environment appropriately.
"""]]


code:
I'm brand new to git-annex and am having trouble getting going.  I've got an annex and I get messages like "Ran daily sanity check" and "Synced with web", but I can't get connected to a Jabber account.  I've been using my Gmail account, which works fine for logging into this forum but gives me the message "Unable to connect to the Jabber server. Maybe you entered the wrong password?" when I try to configure the Jabber account.

I'm using version 3.20121112ubuntu2.  I think a later version might exist but I don't know how to reach it; I've done the standard install for Ubuntu-style OSs (I'm running Kubuntu right now), and that's the git-annex version that arrives.

Any suggestions?  Sorry for the simplistic startup question, but it looks like this will be a great tool once I get it running.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.246.110"
 subject="comment 1"
 date="2013-07-27T22:13:49Z"
 content="""
Since 2012 git-annex has had massive improvements to its Jabber support. See [[/install/Ubuntu]] to get a current version.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlutNrg7ajiNAfi5gqJfD8crv0aimZa27k"
 nickname="Chuck"
 subject="Now I'm really confused 8-/"
 date="2013-07-28T05:26:02Z"
 content="""
Yes, I followed the directions on that page and executed \"sudo apt-get install git-annex\" on my 13.04 (Raring) system.  I did it again just now to verify, and I got the same result from \"git-annex version\", namely \"3.20121112ubuntu2\".  Do I need to add a repo for Raring as for Precise?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmKKg3Vmzk7KwRGRKjHVdtyoj1JfxLX6NM"
 nickname="Tom"
 subject="comment 3"
 date="2013-10-01T18:33:05Z"
 content="""
I've got the same issue on Xubuntu 13.04. I installed using this script: https://github.com/zerodogg/scriptbucket/blob/master/gitannex-install

`git-annex version` makes no mention of DNS or ADNS

`host` command is installed on my machine. any suggestions on how best to fix for this setup?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmKKg3Vmzk7KwRGRKjHVdtyoj1JfxLX6NM"
 nickname="Tom"
 subject="comment 4"
 date="2013-10-02T21:27:36Z"
 content="""
Update to my comment above - it seems my problem is related to ejabberd as discussed here: <http://git-annex.branchable.com/forum/XMPP_authentication_failure/>

Think I might change my server to [Prosody](https://prosody.im/) and see what happens then.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkr_EUIHZ8CTtOGm-j-guMBzaYuOzrXJwg"
 nickname="Christoph"
 subject="be aware of the 2-step verification"
 date="2013-10-15T20:03:12Z"
 content="""
if you enabled the \"2-step verification\" in our google account (Account -> Security -> 2-step verification: Status: ON) you have to make a \"Application-specific passwords\" for your Jabber login.

NoXorius
"""]]


code:
As far I can tell, the encryption mentioned on the wiki for S3 [1] refers to file-level encryption. That is, it encrypts files with GPG before storing them (or after retrieving them) from S3. However, even if I have GPG encryption off, I still want S3 requests to use SSL encryption. As far as I can tell, HTTPS isn't used with S3 remotes. Is there any way to enable it (if so, it should be the default)?

I am using git-annex version: 3.20120406

Cheers,
--acrefoot

[1] http://git-annex.branchable.com/special_remotes/S3/


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.238"
 subject="comment 1"
 date="2013-01-15T20:25:42Z"
 content="""
<http://hackage.haskell.org/package/hS3> is a Haskell library for S3, which git-annex uses. It does not support HTTPS. I'm sure its author would appreciate help, or maybe even just gentle motivation.

FWIW, I think that S3's authorization is designed to be pretty secure even over an un-encrypted transport.
It uses HMAC to sign the request with your AWS credentials securely, and includes a date that is hopefully used to avoid replay attacks.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnRIwPkmDOZSpul-Lx61_jUGHh9f_F3od8"
 nickname="Michael"
 subject="comment 2"
 date="2013-01-15T20:47:50Z"
 content="""
The authorization is fine, but it's scary to see your data floating along the wire in plain-text :)
"""]]


code:
I'm running version 5.20131224-g692aa01 standalone tarball on linux x86_64 (Linux Mint 15). When I run `git-annex status` in the terminal I get no output.

I've tried:

* system linked (`ln -s`) the `git-annex`, `git-annex-shell`, and `git-annex-webapp` to `~/bin/` which is on my PATH
* run the `runshell` program from the terminal

Other things to consider:

* I've just upgraded the repo from version 3 to version 5
* Repo was created using git-annex version 3
* I don't use assistant or the webapp
* Only used from my terminal
* Repo is in indirect mode

What do I need to do to see the results of `git-annex status`? in my terminal?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 1"
 date="2013-12-26T23:06:34Z"
 content="""
Run `git annex info` instead. That is what older versions of git-annex called `git annex status`

Meanwhile, `git annex status` is now like `git status` -- it shows an overview of uncommitted changes in your work tree.
"""]]


code:
[[!comment format=mdwn
 username="https://openid.stackexchange.com/user/e65e6d0e-58ba-41de-84cc-1f2ba54cf574"
 nickname="Mica Semrick"
 subject="Thanks!"
 date="2013-12-27T01:37:36Z"
 content="""
Not sure how I missed that :D
"""]]


code:
My situation goes something like this:

I have a machine with an annex and a number of special remotes (s3, box, and an rsync'd nas).  I also have a git remote that's doesn't have git annex on it, so it's just got the git branches.  That machine has some problems so I start getting the annex set up on a different machine.  These are the steps that I went through:

* clone from the git remote
* git annex doesn't think this is an actual annex at this point, so `git annex init` it
* Now I can start the webapp `git annex webapp`
* My rsync'd remote works and the assistant starts downloading the files from there (which is great, it's the local network one) but the box and s3 remotes are disabled (sure, not a huge deal).
* Click enable on the box remote, I have to specify that it's a full backup remote
* Things start copying from box as well, so I disable it until everything from the local network is done
* Click enable on the s3 remote and it wants my AWS creds
* Download those and add it
* Set the remote up the same way, as a full backup.

At this point, all my files have copied from the rsync remote, so I enable the other remotes.

Now I want to make sure that all the remotes are set up and working properly.

I turn off the webapp, turn off direct mode (I think it was indirect mode by default, but I'd been playing with things before then), and `git annex drop <file>` a file that I don't particularly care about.  Everything drops successfully.

I'm able to `git annex get -f <rsync remote> <file>` and from `<box remote>` successfully, but when I try to get from the s3 remote, it doesn't give me any output and doesn't download the file.

Having used regular git annex without the assistant before, I try re-initing the remote `git annex initremote <s3>`.  It complains that there's no type, so I `git annex initremote <s3> type=S3`, then it complains about encryption.  `git annex initremote <s3> type=S3 encryption=shared`.  It says it worked, but I `git annex get -f <s3> <file>` still doesn't do anything.

After more looking around, it turns out that I may have created a second remote with the same name as the original s3 remote... (figure that out).  I use the webapp to rename the remotes so they're different, but neither of them will `get -f` successfully.

At this point, I turn to you and ask what the heck I did wrong.  I tried editing the remote.log and uuid.log files to remove the new s3 remote (and I figured out which one was which) from the git-annex branch.  I also marked the new s3 remote as dead, but I still can't get access to s3.

`git annex fsck -f <s3>` doesn't actually seem to hit s3 (I seem to recall that it used to, it calculated checksums), it just checks some git local information.

I don't mind deleting my current checkout and starting from the clone step again if you think I've made too much of a mess.  At least I know I can get my files off my rsync remote and box :)


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 10"
 date="2013-04-26T20:33:03Z"
 content="""
Oh.. I got confused by you talking about the box remote. Lines you pasted look ok anyway.

Ok, looking at the S3 remote then...

> I did notice that initially the s3 remote was named \"annex\". That was probably the web interface's doing, way back when I added it.

So, you can never change the names used to refer to remotes in remote.log. These names can be different from the names used to refer to the same remotes in .git/config. (Which can vary from repository to repository anyway..) So, if you originally added a s3 remote and called it \"annex\", you still need to use that name when running initremote elsewhere to add that remote to your repository.

The remote with name \"s3\" added in the 11:26 is a separate s3 remote, and I think one you don't want. (And have marked dead?)

I think all you need to do is \"git annex initremote annex\" to add the s3 remote you want to your new repository.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkZktNHFhxC1kYA9KKdKpYJO4clq9WDsjE"
 nickname="Jason"
 subject="comment 11"
 date="2013-04-26T20:37:51Z"
 content="""
Ah, I see.  It looks like that did solve my problem.

Yes, I did mark the old s3 remote as dead.

At least now I know how to fix it if it ever happens again.  I wonder if I'll ever be able to recreate it...

Thanks!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 12"
 date="2013-04-26T20:52:36Z"
 content="""
It's easy to recreate. As I understand it, the entire process went something like this:

git annex initremote annex type=S3 encryption=blahblah # possibly this was done in the webapp?

git remote rename annex s3 # also possibly done in the webapp

# clone to different computer, and on the new clone:

git annex initremote s3

git-annex: Specify the type of remote with type=

git annex initremote s3 type=S3 encryption=blahblah

The last line creates a *new* remote.

I'm inclined to think the main confusing thing here is that initremote is used to both create a new special remote, and to configure the repository to use an already existing special remote that was created elsewhere. If you had to use `enableremote` for the latter,
things could be less confusing:

# clone to different computer, and on the new clone:

git annex enableremote s3

git-annex: No existing special remote named s3. Choose from one of these existing special remotes: annex
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkZktNHFhxC1kYA9KKdKpYJO4clq9WDsjE"
 nickname="Jason"
 subject="comment 13"
 date="2013-04-26T21:16:30Z"
 content="""
I tend to agree with you.  At first I liked the idea that initremote could be used to re-initialize a remote, but then I got confused about what the name of that remote was.  I suppose git annex status could have told me.  I kept wanting to have something like \"git annex remote\" (which would list them) and then \"git annex remote init\" to initialize them.  That way the remote actions would follow the same sort of interface as \"git remote\", where you could list, init, create, edit, rename, enable, disable, kill (dead?), etc.  The main drawback I see with that is having too many levels to type.

I really like the idea of having the ability to \"git annex remote show s3\" and it will tell me what the type, uuid, options, etc are for that remote.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 14"
 date="2013-04-26T22:25:04Z"
 content="""
Have now split out an enableremote command.

<pre>
joey@gnu:~/tmp/annex>git annex initremote foo
git-annex: There is already a special remote named \"foo\". (Use enableremote to enable an existing special remote.)
joey@gnu:~/tmp/annex>git annex enableremote
git-annex: Specify the name of the special remote to enable. Known special remotes: foo
</pre>

Also, I wrote something wrong before. It *is* possible to change the name used by initremote (now enableremote).

With the current release of git-annex:

`git annex initremote annex name=mys3`

With the next release:

`git annex enableremote annex name=mys3`
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-04-24T23:56:54Z"
 content="""
What you describe, `git annex get $file --from remote` silently not doing anything, is the expected behavior if the remote doesn't have the file. This allows you to eg, run `git annex get . --from remote` and get all files that the remote does have while skipping the rest.

Did you ever try looking at `git annex whereis $file` ?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkZktNHFhxC1kYA9KKdKpYJO4clq9WDsjE"
 nickname="Jason"
 subject="comment 2"
 date="2013-04-25T00:20:59Z"
 content="""
When I run `git annex whereis $file` it tells me it's available at box, s3, my rsync remote, the original repo (from the machine I created the annex on), and my temporary usb stick repo (nothing wrong with redundancy...).  So it seems that git annex thinks the file is available in the s3 remote, even though it's refusing to download it.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 3"
 date="2013-04-25T00:50:01Z"
 content="""
Does the uuid that whereis prints next to the name of the s3 remote match the annex.<remote>.uuid setting for that remote in .git/config?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkZktNHFhxC1kYA9KKdKpYJO4clq9WDsjE"
 nickname="Jason"
 subject="comment 4"
 date="2013-04-25T01:18:00Z"
 content="""
Oh! That might be it!  During the whole \"I have two remotes with the name s3\" situation, it seems that both of them in my .git/config ended up with the same uuid, even though the original one had a different uuid.  If I change it back, I end up getting an access denied when I try to `git annex get ...`.  Progress!

I thought that you were supposed to do a `git annex initremote s3` from a clone to enable a remote with the credentials stored in the repo.  It seems that internally something still thinks that the \"s3\" remote has the new uuid.  When I run that command it changes the uuid back to the new (invalid) one.

Is there a way I can totally remove the bad s3 (which I've partially renamed to s3thefirst) remote from my history/repo (I'm pretty sure it's been synced back up to origin at this point) or properly rename it so it doesn't keep getting confused?  Hopefully that will address my problem.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 5"
 date="2013-04-25T02:14:18Z"
 content="""
Yes, when you run `git annex initremote $remotename` with no other parameters, it enables a remote from the stored configuration.
Which does not include `AWS_SECRET_ACCESS_KEY` and `AWS_ACCESS_KEY_ID`; you need to set those and then
you should not get access denied.

You seem to say your .git/config contains two remotes with the same name, but I don't think that's possible.

I don't know how you could end up with two remotes with the same name in `git show git-annex:remote.log`, unless the two were added in separate repositories which were then synced together. Since this is not a usual situation there's not any UI to deal with it. I've just committed a change that will make `initremote` prefer remotes that have not been marked dead when there's a naming comflict.

However, I'm more curious how this situation came about.  I have not been able to reproduce the problem when enabling a S3 remote using the webapp.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 6"
 date="2013-04-25T02:18:56Z"
 content="""
What you could do to help track down how this occurred is to check out the `git-annex` branch, and use `git blame` to find out when the second remote with the same name was first added to the `remote.log` file.

Then you should be able to tell, either from the email address used for that commit, or at least the date of the commit, whether this occurred recently when you enabled the S3 remote in the webapp, or perhaps at some time in the past.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkZktNHFhxC1kYA9KKdKpYJO4clq9WDsjE"
 nickname="Jason"
 subject="comment 7"
 date="2013-04-26T19:10:15Z"
 content="""
http://pastebin.com/CM2EfQ21

This is what the commit log looks like for the remote.log file.  There is some interesting stuff in here.  I'll try to highlight the changes without giving too much of the important bits away.

The I commit at 2013-04-22 11:57 is when I added the box remote:

0490d177-78e2-421b-a004-47d88ee7a2e3 chunksize=10mb cipher=... davcreds=... embedcreds=yes name=box.com type=webdav url=https://www.box.com/dav/annex timestamp=1366657062.972357s
1d0ab67c-6a43-11e2-9feb-df22c6d1e308 bucket=annex-1d0ab67c-6a43-11e2-9feb-df22c6d1e308 cipher=... datacenter=US host=s3.amazonaws.com name=annex port=80 storageclass=REDUCED_REDUNDANCY type=S3 timestamp=1359484726.520727s

The contents also includes my nas remote, but I will omit that for brevity's sake.  I did notice that initially the s3 remote was named \"annex\".  That was probably the web interface's doing, way back when I added it.

The next commit at 2013-04-24 10:55 seems to have added encryption=shared and highRandomQuality=false to the nas remote (I think this was when I re-enabled the nas remote through the webapp).

The commit at 2013-04-24 11:05 looks like it added similar stuff to the box remote (added highRandomQuality=false).  Probably this was from enabling it then as well.

At 2013-04-24 11:12 the s3 remote had highRandomQuality=false added also.

At 2013-04-24 11:26, a new remote was added:

4d86972d-9b0a-4095-bc50-f9bec8144c30 bucket=s3-4d86972d-9b0a-4095-bc50-f9bec8144c30 cipher=... datacenter=US host=s3.amazonaws.com name=s3 port=80 storageclass=STANDARD type=S3 timestamp=1366828017.8792s

Very possibly this was me doing a `git annex initremote ...` thinking that the s3 remote was actually named s3 (somehow, I feel like I would have checked that, but I'm going to chalk that up to my own stupidity).

Then at 2013-04-24 11:35, the new s3 remote was changed... but it seems like only the timestamp was altered.  I suspect this was from another command line change, but I don't remember exactly what I did at that point.  Probably a reference in a different file was also modified, but I'm not looking at those.

At 2013-04-24 11:37, again the new s3 remote was changed, but again it was just the timestamp.

In the merge at 2013-04-24 15:15, a bunch of things happened.  This may be where stuff went wrong.  I do find it weird because it should have just been a fast forward, given what the history looks like.  I suspect that this was caused by a `git annex sync`, but I'm not 100% sure.

In this commit the following happened:

* The box remote was duplicated (with different davcreds and one having highRandomQuality=false)
* The annex remote was duplicated (with highRandomQuality=false in one)
* The nas remote was duplicated (one with encryption=shared and highRandomQuality=false and the other without)

In addition, within that commit, my uuid.log file also had duplication that seems to be where part of the confusion comes from:

* The 1d0ab67c-6a43-11e2-9feb-df22c6d1e308 remote shows up twice, once named \"annex\" and the other time named \"s3\".
* The 4d86972d-9b0a-4095-bc50-f9bec8144c30 remote is only include once in there, but its name is also \"s3\".
* Other remotes are duplicated, with different timestamps, but no overlapping uuids.

Then at 2013-04-24 18:13, I think things try to fix themselves:

* The older box remote (I guess based on timestamp) is removed.  Now there's only one.
* The older 1d0ab67c-6a43-11e2-9feb-df22c6d1e308 remote (still named annex) is removed.  Now there's only one there too.
* The single 4d86972d-9b0a-4095-bc50-f9bec8144c30 remote is updated with a new timestamp.
* The older nas remote is also removed.

No duplicates exist in this file and no cross-references exist either.

The uuid.log file seems to be the place where the annex remote is renamed to s3.  I have no idea what caused that, but it was probably me.

* In 2013-04-24 11:12, everything is fine in the uuid.log file.  The annex timestamp is updated, but no problems.
* In 2013-04-24 11:13 (which doesn't show up when I look at the remote.log changes, because it didn't change that), a file's location log is updated and the 1d0ab67c-6a43-11e2-9feb-df22c6d1e308 remote is renamed from annex to s3 in uuid.log, but not in remotes.log.
* In 2013-04-24 11:26, 4d86972d-9b0a-4095-bc50-f9bec8144c30 is added to remotes.log with the name s3 and to uuid.log with the name s3 (which is now a duplicate of the renamed 1d0ab67c-6a43-11e2-9feb-df22c6d1e308, but only in uuid.log).

All of this seems horribly confusing and I don't envy your trying to unwind it.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 8"
 date="2013-04-26T20:06:34Z"
 content="""
Most of this is perfectly normal. The duplication of lines are normal; when two git-annex branches are union merged, it's as if it runs `cat branch1:file branch2:file | uniq > file`. When there are conflicting lines for the same uuid, the one with the newest timestamp is used.

The description of the remote in uuid.log is also not relevant to this bug.

This is the key part:

> The box remote was duplicated (with different davcreds and one having highRandomQuality=false)

As you note, 2013-04-24 15:15 was a merge. So there must have been two branches before, which had different box remotes with different davcreds.

It would probably help if you can paste those lines as they looked after that merge (omitting most of the davcreds).

Also, I'd like to see the box line from the 11:05 commit.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkZktNHFhxC1kYA9KKdKpYJO4clq9WDsjE"
 nickname="Jason"
 subject="comment 9"
 date="2013-04-26T20:21:05Z"
 content="""
Two box lines after 15:15 merge:

0490d177-78e2-421b-a004-47d88ee7a2e3 chunksize=10mb cipher=... davcreds=... embedcreds=yes highRandomQuality=false name=box.com type=webdav url=https://www.box.com/dav/annex timestamp=1366826729.945023s
0490d177-78e2-421b-a004-47d88ee7a2e3 chunksize=10mb cipher=... davcreds=... embedcreds=yes name=box.com type=webdav url=https://www.box.com/dav/annex timestamp=1366657062.972357s

After the 11:05 commit, the box line looked like this:

0490d177-78e2-421b-a004-47d88ee7a2e3 chunksize=10mb cipher=... davcreds=... embedcreds=yes highRandomQuality=false name=box.com type=webdav url=https://www.box.com/dav/annex timestamp=1366826729.945023s

I am curious why you want to know about box, when s3 is the one that I'm having trouble with...
"""]]


code:
Was anyone able to run git annex on Mavericks?

Didn't have time for qualified troubleshooting, sorry


code:
[[!comment format=mdwn
 username="http://mike.magin.org/"
 nickname="mmagin"
 subject="a little"
 date="2013-10-24T17:02:18Z"
 content="""
Some syncing and copying, nothing other than normal remotes and one S3 remote.
Haven't even done an \"add\" yet.

Not an assistant user.  Still running 4.20130422-gb9341fd on this computer.
"""]]


code:
[[!comment format=mdwn
 username="Alex"
 ip="217.8.62.35"
 subject="not at all"
 date="2013-10-24T19:50:11Z"
 content="""
I'd love to see an updated OS X build for Mavericks.

Currently I'm getting this error:

    dyld: Symbol not found: _objc_debug_taggedpointer_mask
      Referenced from: /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
      Expected in: /Applications/git-annex.app/Contents/MacOS/bundle/I
     in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
    Trace/BPT trap: 5

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="67.223.19.96"
 subject="comment 3"
 date="2013-10-24T19:53:08Z"
 content="""
Mavericks is being discussed [[here|bugs/git annex doesn't work in Max OS X 10.9]]
"""]]


code:
[[!comment format=mdwn
 username="Alex"
 ip="217.8.62.35"
 subject="comment 4"
 date="2013-10-24T19:55:56Z"
 content="""
Thanks, Joey!
"""]]


code:
This is a tip for users who wish to use remotes which are based on OSX systems and have used macports to install some of the required utilities for git-annex to work.

The default behaviour of OSX's sshd is to have a "highly restricted" restricted environment. The defaults that it allows is

    jtang@x00:~ $ ssh x00 echo \$PATH
    /usr/bin:/bin:/usr/sbin:/sbin

One solution is to enable *PermitUserEnvironment yes* in `/etc/sshd_config` and then in your own `~/.ssh/environment` file you could add something like (the below is an example)

    PATH=/Users/jtang/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/usr/X11/bin:/Users/jtang/.cabal/bin:/opt/local/libexec/gnubin

If the above is not done, cloning from the OSX host will fail if git is not installed in /usr/bin (which it probably won't be).


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.114"
 subject="comment 1"
 date="2014-04-24T18:08:00Z"
 content="""
If you use git-annex from the OSX .dmg, it will set up a ~/.ssh/git-annex-shell, which is the only command that is needed when git-annex is using an OSX server as a remote. Since version 5.20140421, the webapp will also use ~/.ssh/git-annex-wrapper, which the .dmg also sets up, to run some commands like git.

The upshot is that this should not affect git-annex when installed from the .dmg on the OSX server. If you build git-annex from source yourself, you do need to make sure that it and git end up in PATH.
"""]]


code:
This isn't really a bug of git-annex, but a problem with haskell-platform/ghc6.12.x so this post might need to be moved to a better place (maybe tips).

OSX's haskell-platform doesn't have the dynamic libraries available, as far as I know it just isn't supported therefore git-annex will always be statically built on OSX, so wrappers like <http://tsocks.sourceforge.net/> or [[!google dsocks]] for preloading connect() calls won't work.

jtang@x00:~/annex $ tsocks git annex get .
dyld: could not load inserted library: /opt/local/lib/libtsocks.dylib

error: git-annex died of signal 5

The side effect of this means that users who are behind restrictive firewalls that allow only ssh via a socks proxy, they will need to configure ssh to use something like <http://bent.latency.net/bent/git/goto-san-connect-1.85/src/connect.html>.

host remotemyhost
        ProxyCommand connect -S proxy.mydomain:1080 -R local %h %p


code:
I'm trying to use git-annex for keeping my company invoices data. It seems a good idea because it's encrypted and whatnot, I'd never upload them to Dropbox. However, Libre Office refuses to work with files copied to annex-tracked folder. It's likely a flaw in LO/OOo, but maybe someone here knows a way to make the two work together.

When opening a spreadsheet, I'm getting a "Document in use - document locked by: Unknown User - Open read-only or open a copy" dialog. I tried opening a copy and saving under a different filename , but I can only save once - any subsequent saves to the same filename result in an error.

A graphic version of my story: http://f.gdr.name/annex-ooo-1.png http://f.gdr.name/annex-ooo-2.png


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 1"
 date="2013-01-02T19:10:28Z"
 content="""
Files in git-annex are locked normally. You can use `git annex unlock` to unlock a file for editing. Or you can use the new [[direct_mode]].
"""]]


code:
[[!comment format=mdwn
 username="https://launchpad.net/~gdr-go2"
 nickname="gdr-go2"
 subject="Thanks"
 date="2013-01-02T19:58:44Z"
 content="""
Direct mode has solved the problem. Great new addition.
"""]]


code:
Please file bug reports in [[bugs]], not the forum, so they can be closed
when fixed.

Moved to [[bug report|bugs/Out_of_memory_error_in_fsck_whereis_find_and_status_cmds]].
--[[Joey]]


code:
My collaborators and I use git annex to track various large data files (among some smaller metadata files managed by ordinary git).  Some of these data files need to change completely -- the old ones were just wrong.  So I do a git checkout, but don't `git annex get` because it would just be a waste of time and bandwidth.  This means that my "data files" are just broken symlinks.  Now, I find that by making the necessary directories under `.git/annex/objects/`, I can write to these files in the usual way.  The symlinks are preserved, and the files they link to now exist and are full of my corrected data.  This seems like it's a problem because the hash has presumably changed.  (I'm still a little fuzzy on how exactly git-annex works.)  Also, git/git-annex doesn't seem to realize that anything has changed.  Is this recoverable?

Would it have been better to just `git rm` (or something) the original version of the file, commit that, and then add the new data?  And if so, how should I go about this now that I've created these many very large files?  If not, what would be the preferred way to do this?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-06-12T16:57:52Z"
 content="""
I think you're making this more complicated than it needs to be. You don't need to mess around with .git/annex/objects at all. You can replace git-annex symlinks with new files and git annex add the new content.

For example:

[[!format sh \"\"\"
joey@gnu:~/tmp/repo>git annex drop --force foo
drop foo ok
(Recording state in git...)
joey@gnu:~/tmp/repo>ls
foo@
joey@gnu:~/tmp/repo>rm foo
joey@gnu:~/tmp/repo>echo \"new good contents\" > foo
joey@gnu:~/tmp/repo>git annex add foo
add foo (checksum...) ok
(Recording state in git...)
joey@gnu:~/tmp/repo>git commit -m add
[master ec3ed14] add
 1 file changed, 1 insertion(+), 1 deletion(-)
\"\"\"]]
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkxmke7K8gEXleVRuQvCK5LHPLIzQA6s0E"
 nickname="Michael"
 subject="comment 2"
 date="2013-06-12T17:14:11Z"
 content="""
Yes, it seems I was making this more complicated than it needed to be.  Just a plain rm seems to work.  But just to be clear, I never had the data so I don't need to drop it, right?  (By which I mean, is there some other function of drop that I don't understand?)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 3"
 date="2013-06-12T17:15:26Z"
 content="""
Right. I just put the drop in there to get my repository to the state you described.
"""]]


code:
Posting this in case anyone might find it interesting.

I had noticed impractical and abysmally slow performance when tracking a huge number of files (150k) in a git-annex. In direct mode, the repo was outright unusable, but even in indirect mode, many operations where painfully slow; even operations beyond the well-known offender, «git annex findunused», e.g., the seemingly harmless «git annex info».

I also noticed that the performance was hugely improved on my (otherwise comparable) machine running btrfs, and I wondered how this might be. From previous benchmarks, I had gained the impression that ext4 and btrfs are on par, performance-wise, and you choose btrfs for the features rather than performance. Now, after trying to update my back-ups via rsync, I have had an idea how the contrast between the two machines might might be accounted for. Specifically, I noted that, after converting my 150k folder into a git-annex repo, ascertaining that the back-up is current via rsync would take ~15 minutes, where it used to take mere seconds before. This could then only be due to the demands on directory traversal introduced by the annex-layout.

Accordingly, I wanted to see whether the traversal would be something that explains the difference in performance between btrfs and ext4, so I ran a tiny benchmark, traversing the .git folder on my home drive (ext4, SATA) and the backup drive (btrfs, USB3), and I was astounded by the difference:

zardoz [ /mnt/bak/m-annex ]$ time tree .git >/dev/null

tree .git >/dev/null  14,23s user 2,78s system 24% cpu 1:09,69 total

zardoz [ ~/m-annex ]$ time tree .git >/dev/null

tree .git >/dev/null  26,40s user 0,96s system 1% cpu 23:37,94 total

While running, I peeked into the io using iotop, and observed around 500K/s during traversal on ext4 vs. 5M/s on btrfs.

While I was aquainted with the dogma that file-systems hate to have a single folder with a bazillion files, sources on the net seem to indicate that having lots and lots of sparse folders is even worse, and given the one-file-per-folder structure of the annex objects store, this would then potentially explain the heavy thrashing on ext4.

Something I am wondering now: Which operations in git-annex (or plain git) incite that sort of directory traversal? One candidate which occurred to me is «git annex unused», and the differences in performance between ext4 and btrfs are on the same order as in the above benchmark. Originally, Joey related somewhere that the search in «unused» is expensive; but if traversal is involved, it could actually be that this has even more impact than searching git history.

In any case, if anyone wants to track a very large number of files via git-annex, ext4 seems to be not the ideal file-system for this.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 1"
 date="2014-07-14T18:33:25Z"
 content="""
I'm afraid that you're comparing apples (SATA drives) and oranges (USB drives).
"""]]


code:
[[!comment format=mdwn
 username="zardoz"
 ip="78.49.247.112"
 subject="comment 2"
 date="2014-07-14T19:34:31Z"
 content="""
I suppose you misread something. Note that the 60s traversal was on the (slower) /external/ drive, and the 24minute traversal on the (faster) internal one, so if anything, the results become more pronounced because of the comparison.

In the meantime I converted the internal drive to btrfs, and the traversal time on SATA is now 30s, as opposed to the >20min on ext.
"""]]


code:
[[!comment format=mdwn
 username="zardoz"
 ip="134.147.14.84"
 subject="comment 3"
 date="2014-07-15T07:29:37Z"
 content="""
The ext-community seems to
[corroborate](http://www.redhat.com/archives/ext3-users/2013-March/msg00007.html)
the observation that having many sparse directories scales extremely
poorly, though this still leaves me curious as to how btrfs deals
with.

One thing I read is that btrfs stores a secondary index on directories
and uses this index in the readdir() syscall; this secondary index
works in such a way that inodes are likely to be traversed in
sequential on-disk order, while for ext, the readdir() results will be
ordered by inode number, yielding a random access pattern.

I must confess I’ve always liked to think of the file-system as a
cheap data-base, but apparently that is not such a good idea (i. e.,
it’s not cheap at all, in the long run). On the other hand (supposing
that operations like «git annex unused» do indeed work by traversing
the object tree), it probably wouldn’t be easy coming up with a scheme
that scales better. For traversal-bound operations, one might maintain
a database, but it would be a hassle to ensure that this in always in
sync with the file-system.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 4"
 date="2014-07-21T17:19:42Z"
 content="""
<https://microca.st/clacke/note/kKMqYM4DQtOPzMe065oHiA> suggests using -I 512 -O inline_data (\"bignode\")
"""]]


code:
Hello,

I've a strange (for me ;-)) problem.

I'm trying to use git-annex with 2 debian wheezy (packages from backports)

I've put my ssh pub key on the server in the autorized_keys for my username account (and for the root account to be sure)
my username account is member of the sudo group (not sure it's required)

I can create an ssh session directly to the server with :

    ssh username@remoteserver.lan -p 62322

and work inside this session.

    Sep 14 03:42:35 file sshd[8849]: Accepted publickey for username from 80.201.25.4 port 32969 ssh2
    Sep 14 03:42:35 file sshd[8849]: pam_unix(sshd:session): session opened for user username by (uid=0)
    ...
    Sep 14 05:13:14 file sshd[8851]: Received disconnect from 80.201.25.4: 11: disconnected by user
    Sep 14 05:13:14 file sshd[8849]: pam_unix(sshd:session): session closed for user username

But when I use the git-annex assistant on the same workstation with the same remote server and port,

the 1st step is OK,

but when I've to choose between git or rsync encrypted, I'm choosing git and receive :

    "Permission denied (publickey)."

On the server I can only see that the ssh session is immediately closed after opened and before the choice git or rsync is made.

    Sep 14 05:13:42 file sshd[8882]: Accepted publickey for username from 80.201.25.4 port 32984 ssh2
    Sep 14 05:13:42 file sshd[8882]: pam_unix(sshd:session): session opened for user username by (uid=0)
    Sep 14 05:13:42 file sshd[8886]: Received disconnect from 80.201.25.4: 11: disconnected by user
    Sep 14 05:13:42 file sshd[8882]: pam_unix(sshd:session): session closed for user username

Do you think it's the problem ? or whatelse ...

and could you guide me to a possible solution ?

Best regards


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 1"
 date="2013-11-08T18:24:27Z"
 content="""
The webapp sets up a dedicated ssh public key when you add a ssh repository in there.

You might see some useful debugging info your your problem in ~/annex/.git/annex/daemon.log
"""]]


code:
[[!comment format=mdwn
 username="Auroch"
 ip="87.65.174.80"
 subject="OK,thanx for this"
 date="2013-11-09T09:10:42Z"
 content="""
The problem is not solve, but with your informations, I think i've found somehting.

Is it possible that the problem is linked to my situation of port-forwarding ?
The remoteserver is a host (proxmox) with iptable port forwarding 62322 to the real port 22 of the VM hosting the git where I try to connect.

So could the problem seems to be that some actions are taken on the host and not the VM ... correct ?

    [2013-11-09 09:51:33 CET] read: ssh-keygen [\"-F\",\"bla.remote.tld\"]
    [2013-11-09 09:51:33 CET] read: ssh [\"-oNumberOfPasswordPrompts=0\",\"-n\",\"-p\",\"62322\",\"user@bla.remote.tld\",\"sh -c 'echo git-annex-probe loggedin;if which git-annex-shell; then echo git-annex-probe git-annex-shell; fi;if which rsync; then echo git-annex-probe rsync; fi;if which ~/.ssh/git-annex-shell; then echo git-annex-probe ~/.ssh/git-annex-shell; fi'\"]
    [2013-11-09 09:51:35 CET] chat: ssh [\"user@bla.remote.tld\",\"sh -c 'mkdir -p '\\"'\\"'annex'\\"'\\"'&&cd '\\"'\\"'annex'\\"'\\"'&&if [ ! -d .git ]; then git init --bare --shared; fi&&git annex init'\"]

If it's correct, have you an idea for solving this ?

best regards

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.246"
 subject="comment 3"
 date="2013-11-09T18:07:56Z"
 content="""
Well, it does look like your git-annex did not pass the port to the second command.

What version are you using? I've just tested with a ssh server only listening on 2222 and it used the port throughout.

You might try upgrading to a recent version. I know that version 4.20130627, at least, had a bug with the ssh port.
"""]]


code:
[[!comment format=mdwn
 username="Auroch"
 ip="87.65.174.80"
 subject="comment 4"
 date="2013-11-09T19:24:54Z"
 content="""
I've the wheezy-backports version

    i   4.20131002~bpo70+1                                                        wheezy-backports                                      100

I'll have to wait an update and I'll come back to you.

Thanx this amazing work !

"""]]


code: I'm having trouble installing the latest git-annex. It depends on 'base >= 4.5 && base < 5', but ghc 7.0 only ships base 3.0. I've tried upgrading ghc to 7.4 but that breaks a whole bunch of other things; for example, the Crypto module fails to compile, preventing me from installing ghc. Please fix compatibility with ghc 7.0.


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2012-03-11T15:50:11Z"
 content="""
There is a `ghc7.0` branch in [[git|download]] that is being maintained to work with that version.
"""]]


code:
Hi,
i am following debian testing. The latest git-annex publication has manged to replace the former version on the last possible moment. Now another 10 days waiting.
Would be nice if you can coordinate with testing.

Thank you for git-annex.
Jürgen


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 1"
 date="2013-11-03T19:57:24Z"
 content="""
I do coordinate with testing. That version had no possibility of ever reaching testing, since it failed to build on several architectures.
"""]]


code:
I've been trying to use git-annex with the following strategy.

* Download podcasts into the annex `gpodder-downloads`
* Check the podcasts into the annex using `git annex add`.
* Copy the podcasts over to my mp3 player in the annex `usb-ariaz`.
  This is a FAT-formatted mp3 player, so I have been using a bare
  repository.
* Move the podcasts to a different annex called `gpodder-on-usbdisk`
  to indicate that they have been successfully put on the mp3 player.
* `chmod` the files on the mp3 player to `0600` so that I can delete
  them from the player when I am done listening to them.

Then I go for a run or something and listen to a bunch of podcasts,
deleting them after I have listened to them.  When I get back, I would
like to find the files that I have listened to and remove them from
the annexes that are not on the mp3 player.  What I have been hoping
is that something like

    ~/gpodder-on-usbdisk $ git annex find --not --in usb-ariaz --print0 | xargs -0 git rm
    ~/gpodder-on-usbdisk $ git annex unused
    ~/gpodder-on-usbdisk $ git annex dropunused `seq X`

would work.  However, it appears that `git-annex find` does not
actually check to see that the file contents are present, but only
looks at the `git-annex` branch of the `usb-ariaz` repository.  Since
I have not changed that with my sneaky deletions, it has no way of
knowing that the files have been deleted.

Is there any way to do this properly?  (And by properly, I don't mean
"don't delete the files".  That is really the only way I have of
marking that I have listened to podcasts on this particular mp3 player.)

I tried setting the `usb-ariaz` repository to be untrusted, but that
did not change the behavior of `git annex find`.


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2011-11-27T17:56:31Z"
 content="""
Right, --in goes by git-annex's [[location_tracking]] information; actually checking if a remote still has the files would make --in too expensive in many cases.

So you need to give `gpodder-on-usbdisk` current information. You can do that by going to `usb-ariaz` and doing a `git annex fsck`. That will find the deleted files and update the location information. Then, back on `gpodder-on-usbdisk`, `git pull usb-ariaz`, and then you can proceed with the commands you showed.
"""]]


code:
[[!comment format=mdwn
 username="http://cgray.myopenid.com/"
 nickname="cgray"
 subject="comment 2"
 date="2011-11-27T22:10:44Z"
 content="""
Thanks, that works perfectly!
"""]]


code:
I have an e-mail server configured to save my mail in ~/Maildir on an account that is available over ssh. I'd like to keep emailserver:~/Maildir in a two-way sync with laptop:~/Mail/private essentially creating a poor man's IMAP — without setting up and maintaining an actual IMAP server. **Is it an appropriate use of git-annex or would another tool be more fitting? And how do I go about doing it?** I'd like to sync the files, the content, not just information about the files or other meta-data.

I tried setting it up with the webUI to the assistant but it only offers encrypted storage[1] on the remote server. I looked into setting it up manually but "git-annex does not notice when files are added to remote rsync repositories."[2]

[1] http://git-annex.branchable.com/bugs/Remote_repositories_have_to_be_setup_encrypted/
[2] from comments on http://git-annex.branchable.com/special_remotes/rsync/


code:
[[!comment format=mdwn
 username="http://a-or-b.myopenid.com/"
 ip="220.244.41.108"
 subject="comment 1"
 date="2013-08-14T06:45:56Z"
 content="""
I'd say that git-annex is the wrong tool.

OfflineIMAP ([[http://offlineimap.org]]) is simple to set up, is easy to secure over ssh and does exactly what you want.

"""]]


code:
[[!comment format=mdwn
 username="http://a-or-b.myopenid.com/"
 ip="220.244.41.108"
 subject="comment 2"
 date="2013-08-14T06:53:04Z"
 content="""
From my massive amount of googling, i.e. 2min, apparently mbsync ([[http://isync.sourceforge.net/mbsync.html]]) is good too.

I've never used it, so YMMV.
"""]]


code:
[[!comment format=mdwn
 username="http://spindritf.myopenid.com/"
 nickname="spindritf"
 subject="comment 3"
 date="2013-08-14T07:30:30Z"
 content="""
Offlineimap connects to an IMAP server and I was hoping to avoid running one. Thanks though.
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/FHnTlSBo1eCGJRwueeKeB6.RCaPbGMPr5jxx8A--#ce0d8"
 nickname="Hamza"
 subject="comment 4"
 date="2013-08-14T11:11:59Z"
 content="""
IMHO It is not the wrong tool for the job, after all the whole point of assistant is to sync folder across machines. I have a similar setup. I have two folders on two machines plus an encrypted repo on an ssh server. On both folders (local/server) i keep webapp running they sync using the encrypted repo/xmpp. The only downside to this approach is that VPS has two copies of every file one in the folder on in the encrypted git repo.
"""]]


code:
[[!comment format=mdwn
 username="http://spindritf.myopenid.com/"
 nickname="spindritf"
 subject="comment 5"
 date="2013-08-14T18:08:22Z"
 content="""
> IMHO It is not the wrong tool for the job, after all the whole point of assistant is to sync folder across machines.

That's what I thought. OK, so how do I do it? Should I set up this rsync special remote and run sync every once in a while. When starting the e-mail client for example?
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/FHnTlSBo1eCGJRwueeKeB6.RCaPbGMPr5jxx8A--#ce0d8"
 nickname="Hamza"
 subject="comment 6"
 date="2013-08-16T09:14:52Z"
 content="""
You can either keep running 2 webapps paired using xmpp running all the time that gives you push like notifications or if you are ok with syncing every once in a while you can have a check mail script that adds files on the server commits them then calls git annex sync locally,

    ssh $1 \"cd /path/to/annex/;git add .;git commit 'Update'\";git annex sync

should do the trick. The latter method you just need to annex repos no encrypted third repo. Just init your git annex repo on the server and clone on the laptop thats it.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.63"
 subject="comment 7"
 date="2013-08-23T18:30:04Z"
 content="""
I don't feel that git-annex is the best thing to use for this. Maildir has some specific semantics for the filenames used in it that let imap clients resolve inconsistencies, such as a message that was read on machine A, and deleted on machine B. git-annex is unlikely to work as well.

However, I have to say that the very beginning of this thread has a wrong statement in it.

> I tried setting it up with the webUI to the assistant but it only offers encrypted storage[1] on the remote server.

If you install git and git-annex on your remote server, the git-annex assistant will detect this, and offer the choice between a regular git repository, and encrypted storage. If you don't have them installed, it tells you you don't, offers to let you retry once you do install them, and offers encrypted storage as the only option that works given what's installed on the server.

(Also, the bug you linked to in [1] has nothing at all to do with what you were talking about.)
"""]]


code:
[[!comment format=mdwn
 username="konubinix"
 ip="82.243.233.186"
 subject="And with a lots of mails?"
 date="2013-08-27T06:44:56Z"
 content="""
Hi,

I have a question quite similar, but for a different purpose. I use OfflineImap for imap synchronisation, but in my current situation, I travel a lot between two places: one connected to the Internet and the other not connected.

When I am connected to the Internet, I may synchronize mails, then I rsync my ~/Mail directory to a usb key so that I have access to them in the place without connection. The mails filenames may be changed as Joeyh mentioned while I read or delete them. I rsync them back to the usb key before going back to the place with Internet connection, where the OfflineImap synchronization may occur.

This solution, with rsynced key, works well. But I would love a history of what was done with file names and may be able to retrieve an old mail.

I figured out that that the git annex was a really good solution for that.

The main drawback from my point of view is that I have around 100 000 mails (some would say that's \"[Not much mail](http://notmuchmail.org/)\"), and I am afraid that git will be quite slow with that amount of files.

Did anyone experience an annexed repository with so many items it in?

Best
"""]]


code:
Problem: I don't have admin access on all the computers I want to plug my USB with git-annex on.
Solution: Make git & git-annex portable.

The only catch is I'm not familiar enough with git to be able to actually tell if the more specialized git-annex working properly. The commands execute without crashing so I suspect it's fine but I was hoping someone else more competent could try this out and let me know if I it's working so I can add it to the tips section. Thanks in advance!

Ingredients:

* Portable version of msysgit: <http://code.google.com/p/msysgit/downloads/list>

* Windows git-annex installer: <http://git-annex.branchable.com/install/>

* 7-zip (portable version optional): <http://portableapps.com/apps/utilities/7-zip_portable>

Steps:

1. Use 7-zip to extract the contents of the git-annex installer into their own folder.

2. Extract the portable version of git and copy all files extracted from git-annex installer into the "cmd" folder.

3. Run it!

Edit: the markdown is eating my attempts to create a numbered list and making them a paragraph instead. Apologies!


code:
[[!comment format=mdwn
 username="rtwolf"
 ip="69.165.155.234"
 subject="comment 1"
 date="2014-01-07T17:38:19Z"
 content="""
Seems windows version of git-annex only runs in direct mode so I got the dreaded \"fatal: this operation must be run in a work tree\" error. Ran \"git config core.bare false\" which fixed it.

More info on direct mode: <http://git-annex.branchable.com/direct_mode/>
"""]]


code:
[[!comment format=mdwn
 username="rtwolf"
 ip="69.165.155.234"
 subject="comment 2"
 date="2014-01-07T17:41:25Z"
 content="""
Correction to above. Don't run that command, just use git annex sync instead.

Also, git annex drop gives just a blank line. Maybe it doesn't work in direct mode either?
"""]]


code:
We're nearing the end of the year of development funded by Kickstarter. I'm curious to know what the future of the project looks like.

I assume that development will decrease from its current levels as you focus more of your energy on activities that put bread on the table, but is git-annex still something that you foresee actively working on or will it be in more of a feature-freeze-bug-fix stage?

As a backer my only regret is that during the campaign I wasn't able to donate as much as I would have liked. The project has exceeded my expectations since then. I don't know that another Kickstarter campaign is worth the effort, but if you were to ask for donations to fund another month or two or three I would gladly donate. Or perhaps implement a feature-bounty program where users could donate money toward development of a particular feature they would like to see added. Or perhaps you're sick of working for the Internet!


code:
Hello,

I have the problem that, while git-annex preserves the file access rights (user, group, others) for the actual file, it does not make sure that others can access this file through the directory tree above said file:

    /tmp $ mkdir test
    /tmp $ chown claudius:media test
    /tmp $ chmod 750 test
    /tmp $ ls -dl test
    drwxr-x--- 2 claudius media 40 2012-01-23 19:27 test/
    /tmp $ cd test
    /tmp/test $ git init --shared=all
    Initialized empty shared Git repository in /tmp/test/.git/
    /tmp/test $ git annex init "test"
    init test ok
    /tmp/test $ echo 123 > abc
    /tmp/test $ chmod 640 abc
    /tmp/test $ chown claudius:media abc
    /tmp/test $ ls -l
    total 4
    -rw-r----- 1 claudius media 4 2012-01-23 19:27 abc
    /tmp/test $ git annex add .
    add abc (checksum...) ok
    (Recording state in git...)
    /tmp/test $ ls -l
    total 4
    lrwxrwxrwx 1 claudius claudius 176 2012-01-23 19:27 abc -> .git/annex/objects/8F/pj/SHA256-s4--181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b/SHA256-s4--181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b
    /tmp/test $ ls -l .git/annex/objects/8F/pj/SHA256-s4--181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b/SHA256-s4--181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b
    -r--r----- 1 claudius media 4 2012-01-23 19:27 .git/annex/objects/8F/pj/SHA256-s4--181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b/SHA256-s4--181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b
    /tmp/test $ ls -lR .git/annex/objects/
    .git/annex/objects/:
    total 0
    drwx--S--- 3 claudius claudius 60 2012-01-23 19:28 8F/

    .git/annex/objects/8F:
    total 0
    drwx--S--- 3 claudius claudius 60 2012-01-23 19:28 pj/

    .git/annex/objects/8F/pj:
    total 0
    dr-x--S--- 2 claudius claudius 60 2012-01-23 19:28 SHA256-s4--181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b/

    .git/annex/objects/8F/pj/SHA256-s4--181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b:
    total 4
    -r--r----- 1 claudius media 4 2012-01-23 19:27 SHA256-s4--181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b
    /tmp/test $ stat .git/annex/objects/
      File: `.git/annex/objects/'
      Size: 60        	Blocks: 0          IO Block: 4096   directory
    Device: 11h/17d	Inode: 2365970     Links: 3
    Access: (2700/drwx--S---)  Uid: ( 1000/claudius)   Gid: ( 1000/claudius)
    Access: 2012-01-23 19:28:10.614948386 +0100
    Modify: 2012-01-23 19:28:10.614948386 +0100
    Change: 2012-01-23 19:28:10.614948386 +0100
     Birth: -

The use case is that I have a rather large collection of music I would like to manage with git-annex in various locations (all of it on my external hard drive, some on my notebook etc. This music is played by MPD, which can access the collection because it is in the "media" group. After changing to git-annex, however, this fails.

I tried to avoid this specific problem by declaring the git repository to be shared, which does appear to have some effect on the other files in .git:

    /tmp/test $ ls -l .git
    total 16
    drwx--S---  5 claudius claudius 160 2012-01-23 19:28 annex/
    drwxrwsr-x  2 claudius claudius  40 2012-01-23 19:27 branches/
    -rw-rw-r--  1 claudius claudius 218 2012-01-23 19:27 config
    -rw-rw-r--  1 claudius claudius  73 2012-01-23 19:27 description
    -rw-rw-r--  1 claudius claudius  23 2012-01-23 19:27 HEAD
    drwxrwsr-x  2 claudius claudius 220 2012-01-23 19:27 hooks/
    -rw-rw-r--  1 claudius claudius 104 2012-01-23 19:28 index
    drwxrwsr-x  2 claudius claudius  60 2012-01-23 19:27 info/
    drwxrwsr-x  3 claudius claudius  60 2012-01-23 19:27 logs/
    drwxrwsr-x 15 claudius claudius 300 2012-01-23 19:28 objects/
    drwxrwsr-x  4 claudius claudius  80 2012-01-23 19:27 refs/

I could obviously try to change the rights of annex/, annex/objects etc., but I would like to avoid having to adapt them each time a new folder is added somewhere below annex/objects/.

My knowledge of git and especially git-annex is not too good, so it might well be that I missed something obvious. Any hints? :)

(And thank you, of course, for taking the time to read all this)


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2012-01-23T19:00:40Z"
 content="""
You say you started the repo with \"git init --shared\" .. but what that's really meant for is bare repositories, which can have several users pushing into it, not a non-bare repository.

The strange mode on the directories \"dr-x--S---\" and files \"-r--r-----\" must be due to your umask setting though. My umask is 022 and the directories and files under `.git/annex/objects` are \"drwxr-xr-x\" and \"-r--r--r--\", which allows anyone to read them unless an upper directory blocks it -- and with this umask, none do unless I explicitly remove permissions from one to lock down a repository.

About mpd, the obvious fix is to run mpd not as a system user but as yourself. I put \"@reboot mpd\" in my crontab to do this.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlB7-aXsqwzOi2BIR_Q4sUF8sjj24H6F3c"
 nickname="Claudius"
 subject="comment 2"
 date="2012-01-23T19:39:17Z"
 content="""
Thank you for your comment! Indeed, setting the umask to, for example, 022 has the desired effect that annex/objects etc. are executable (and in this special case also writable), my previous umask setting was 077; the \"strange\" permissions on the git directories was probably due to --shared=all, and the mode of \"440\" on the files within the git-annex tree is correct (the original file was 640 and stripped of its write permission).

Using this umask setting and newgrp to switch the default group, I was successfully able to set up the repositories.

However, I would like to suggest adding the execute bit to the directories below .git/annex/objects/ per default, even if the umask of the current shell differs. As the correct rights are already preserved in the actual files (minus their write permission) together with correct owner and group, the files are still protected the same way as previously, and because +x does not allow directory listings, no additional information can leak out either. Not having to set the umask to something \"sensible\" before operating git-annex would be a huge plus, too :)

The reason why I am not running MPD as my user is that I am a bit wary of running an application even exposed to the local network as my main user, and I see nothing wrong with running it as its own user.

Thank you again for your help and the time you put into this project!
"""]]


code:
I am in Windows and therefore forced to use direct mode. After reading for a long while I still have not found a way of "git annex get" previous versions. Is this supported? is it only the last version in the repository?

any chance to use indirect mode in Windows? In theory Windows supports NTFS symlinks, why is this not supported?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 1"
 date="2013-12-24T20:56:00Z"
 content="""
git-annex for windows is targeting being used with msysgit, and msysgit does not seem to support symlinks. That's one good reason. Since cywin's git does support symlinks, it might be possible to get git-annex to use it and support indirect mode.

Doing anything about this is probably \"patches accepted\" territory for me. I'm mostly interested in porting git-annex to Windows so it can be used on computers owned by non-technical users, to interoperate with git-annex on more capable operating systems.
"""]]


code:
While trying to compile the current master I run in the following error:

    Utility/Yesod.hs:21:14:
        Couldn't match expected type `String'
                    with actual type `WidgetFileSettings'
        Expected type: String -> Q Exp
          Actual type: WidgetFileSettings -> FilePath -> Q Exp
        In the expression: widgetFileNoReload
        In an equation for `widgetFile': widgetFile = widgetFileNoReload
    make: *** [git-annex] Fehler 1

I installed all dependencies from the INSTALL document. What is wrong?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.246.16"
 subject="comment 1"
 date="2012-08-29T15:51:15Z"
 content="""
I think you're building with the Makefile, not with cabal, and have the recent yesod 1.1.0 release installed. The Makefile contains a -DWITH_OLD_YESOD that you need to remove in that situation.
"""]]