code:
Integrating git-annex and your file manager provides an easy way to select
annexed files to get or drop.

[[!toc]]

## GNOME (nautilus)

Recent git-annex comes with built-in integration for Nautilus.

[[!img assistant/nautilusmenu.png]]

[[!img assistant/downloadnotification.png]]

This is set up by git-annex creating simple scripts in
`~/.local/share/nautilus/scripts`, with names like "git-annex get"

## KDE (Dolphin/Konqueror)

Even more recent git-annex comes with built-in integration with Konqueror.

[[!img assistant/konquerormenu.png]]

This is set up by git-annex creating a
`~/.kde/share/kde4/services/ServiceMenus/git-annex.desktop file.

## XFCE (Thunar)

XFCE uses the Thunar file manager, which can also be easily configured to allow for custom actions. Just go to the "Configure custom actions..." item in the "Edit" menu, and create a custom action for get and drop with the following commands:

    git-annex drop --notify-start --notify-finish -- %F

for drop, and for get:

    git-annex drop --notify-start --notify-finish -- %F

This gives me the resulting config on disk, in `.config/Thunar/uca.xml`:

    <action>
        <icon>git-annex</icon>
        <name>git-annex get</name>
        <unique-id>1396278104182858-3</unique-id>
        <command>git-annex get --notify-start --notify-finish -- %F</command>
        <description>get the files from a remote git annex repository</description>
        <patterns>*</patterns>
        <directories/>
        <audio-files/>
        <image-files/>
        <other-files/>
        <text-files/>
        <video-files/>
    </action>
    <action>
        <icon>git-annex</icon>
        <name>git-annex drop</name>
        <unique-id>1396278093174843-2</unique-id>
        <command>git-annex drop --notify-start --notify-finish -- %F</command>
        <description>drop the files from the local repository</description>
        <patterns>*</patterns>
        <directories/>
        <audio-files/>
        <image-files/>
        <other-files/>
        <text-files/>
        <video-files/>
    </action>

The complete instructions on how to setup actions is [in the XFCE documentation](http://docs.xfce.org/xfce/thunar/custom-actions).

## OS X (Finder)

For OS X, it is possible to get context menus in Finder. Due to how OS X deals with sym links, one needs to operate on folders if using indirect mode. Direct mode operation has not been tested.

1. Open Automator and create a new Service.
2. Using the Drop down menus in the top create the sentence "Service receives selected folders in Finder.app" to have it work on folders. For direct mode operation it is probably reasonable to select "files or folders".
3. Add a "Run shell script" element and fill in line with the following script:

        #!/usr/bin/bash
        source ~/.bash_profile
        for f in "$@"
        do
            cd "$(dirname "$f")" && git-annex get "$f"
        done

The purpose of the first line is there to get git-annex on to the path. The reason for the for loop is in case multiple files or folders are marked when running the context menu command.

Finally save the the workflow under the name for which it should be listed in the context menu.

## your file manager here

Edit this page and add instructions!

## general

If your file manager can run a command on a file, it should be easy to
integrate git-annex with it. A simple script will suffice:

	#!/bin/sh
	git-annex get --notify-start --notify-finish -- "$@"

The --notify-start and --notify-stop options make git-annex display a
desktop notification. This is useful to give the user an indication that
their action took effect. Desktop notifications are currently only
implemented for Linux.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnmF_9CAtfqdZkC4e-_dCX-rK5bqh4RWkw"
 nickname="Carl"
 subject="Finder added"
 date="2014-04-23T13:19:02Z"
 content="""
For some reason I do not manage to get the shell script to pretty print. I can also supply a screenshot of the Automator, but do not manage to upload it to the wiki.
"""]]

repo_name: kakafarm/git

license: unknown


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 2"
 date="2014-04-23T17:49:09Z"
 content="""
@Carl, I've fixed the formatting. To add a picture, you'd need to send me a pull request adding it to the git repo.
"""]]


code:
Maybe you had a lot of files scattered around on different drives, and you
added them all into a single git-annex repository. Some of the files are
surely duplicates of others.

While git-annex stores the file contents efficiently, it would still
help in cleaning up this mess if you could find, and perhaps remove
the duplicate files.

Here's a command line that will show duplicate sets of files grouped together:

	git annex find --include '*' --format='${file} ${escaped_key}\n' | \
		sort -k2 | uniq --all-repeated=separate -f1 | \
		sed 's/ [^ ]*$//'

Here's a command line that will remove one of each duplicate set of files:

	git annex find --include '*' --format='${file} ${escaped_key}\n' | \
		sort -k2 | uniq --repeated -f1 | sed 's/ [^ ]*$//' | \
		xargs -d '\n' git rm

--[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmTNrhkVQ26GBLaLD5-zNuEiR8syTj4mI8"
 nickname="Juan"
 subject="comment 10"
 date="2013-08-31T18:20:58Z"
 content="""
I'm already spreading the word. Handling scientific papers, data, simulations and code has been quite a challenge during my academic career. While code was solved long ago, the three first items remained a huge problem.
I'm sure many of my colleagues will be happy to use it.
Is there any hashtag or twitter account? I've seen that you collected some of my tweets, but I don't know how you did it. Did you search for git-annex?
Best,
    Juan
"""]]


code:
[[!comment format=mdwn
 username="sameerds"
 ip="106.51.197.116"
 subject="a shell script that handles spaces in file names"
 date="2013-12-31T10:24:06Z"
 content="""
I used the following shell pipeline to remove duplicate files in one go:

    (1) git annex find --format='${key}:${file}\n' \
    (2)    | cut -d '-' -f 4- \
    (3)    | sort \
    (4)    | uniq --all-repeated=separate -w 40 \
    (5)    | awk -vRS= -vFS='\n' '{for (i = 2; i <= NF; i++) print $i}' \
    (6)    | cut -d ':' -f 2- \
    (7)    | xargs -d '\n' git rm

1. Generate a list of keys and file names separated by a colon (':').
2. Cut out the initial part of the key so that the hash is at the beginning of the line. The `-f 4-` ensures that dashes in the filename do not result in truncation.
3. Sort the entire list.
4. Uniquify and print duplicates in groups separated by blank lines. Use the first 40 characters, which matches the length of a SHA1 hash. Other hashes will require a different length.
5. Use awk to print all but the first line in each group. The empty `-vRS` sets blank line as the record separator, and the `-vFS` sets newline as the field separator. The for-loop prints each field except the first.
6. Cut out the key and keep only the file name by relying on the colon introduced in the first step.
7. Use xargs to separate file names by newline, which takes care of spaces in the file names. Send this list of arguments to `git rm`.
"""]]


code:
[[!comment format=mdwn
 username="http://adamspiers.myopenid.com/"
 nickname="Adam"
 subject="Cool"
 date="2011-12-23T19:16:50Z"
 content="""
Very nice :)  Just for reference, here's [my Perl implementation](https://github.com/aspiers/git-config/blob/master/bin/git-annex-finddups).  As per [this discussion](http://git-annex.branchable.com/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/#comment-fb15d5829a52cd05bcbd5dc53edaffb2) it would be interesting to benchmark these two approaches and see if one is substantially more efficient than the other w.r.t. CPU and memory usage.
"""]]


code:
[[!comment format=mdwn
 username="bremner"
 ip="156.34.89.108"
 subject="problems with spaces in filenames"
 date="2012-09-05T02:12:18Z"
 content="""
note that the sort -k2 doesn't work right for filenames with spaces in them. On the other hand, git-rm doesn't seem to like the escaped names from escaped_file.
"""]]


code:
[[!comment format=mdwn
 username="mhameed"
 ip="82.32.202.53"
 subject="problems with spaces in filenames"
 date="Wed Sep  5 09:38:56 BST 2012"
 content="""

Spaces, and other special chars can make filename handeling ugly.
If you don't have a restriction on keeping the exact filenames, then
it might be easiest just to get rid of the problematic chars.

    #!/bin/bash

    function process() {
        dir="$1"
        echo "processing $dir"
        pushd $dir >/dev/null 2>&1

        for fileOrDir in *; do
            nfileOrDir=`echo "$fileOrDir" | sed -e 's/\[//g' -e 's/\]//g' -e 's/ /_/g' -e "s/'//g" `
            if [ "$fileOrDir" != "$nfileOrDir" ]; then
                echo renaming $fileOrDir to $nfileOrDir
                git mv "$fileOrDir" "$nfileOrDir"
            else
                echo "skipping $fileOrDir, no need to rename."
            fi
        done

        find ./ -mindepth 1 -maxdepth 1 -type d | while read d; do
        process "$d"
        done
        popd >/dev/null 2>&1
    }

    process .

Maybe you can run something like this before checking for duplicates.

"""]]


code:
[[!comment format=mdwn
 username="bremner"
 ip="156.34.89.108"
 subject="more about spaces..."
 date="2012-09-09T19:33:01Z"
 content="""
Ironically, previous renaming to remove spaces, plus some synching is how I ended up with these duplicates. For what it is worth, aspiers perl script worked out for me with a small modification. I just only printed out the duplicates with spaces in them (quoted).
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkaBh9VNJ-RZ26wJZ4BEhMN1IlPT-DK6JA"
 nickname="Alex"
 subject="printing keys first is the easiest workaround"
 date="2013-04-01T23:32:23Z"
 content="""
Since the keys are sure to have nos paces in them, putting them first makes working with the output with tools like sort, uniq, and awk simpler.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnkBYpLu_NOj7Uq0-acvLgWhxF8AUEIJbo"
 nickname="Chris"
 subject="Find files by key"
 date="2013-05-03T04:14:55Z"
 content="""
Is there any simple way to search for files with a given key?

At the moment, the best I've come up with is this:

````
git annex find --include '*' --format='${key} ${file}' | grep <KEY>
````

where `<KEY>` is the key. This seems like an awfully longwinded approach, but I don't see anything in the docs indicating a simpler way to do it. Am I missing something?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 7"
 date="2013-05-13T18:42:14Z"
 content="""
@Chris I guess there's no really easy way because searching for a given key is not something many people need to do.

However, git does provide a way. Try `git log --stat -S $KEY`
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmTNrhkVQ26GBLaLD5-zNuEiR8syTj4mI8"
 nickname="Juan"
 subject="This is an awesome feature"
 date="2013-08-28T13:40:23Z"
 content="""
Thanks. I have quite a lot of papers in PDF formats. Now I'm saving space, have them controlled, synchronized with many devices and found more than 200 duplicates.
Is there a way to donate to the project? You really deserve it.
Thanks.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.8.7"
 subject="comment 9"
 date="2013-08-28T20:25:20Z"
 content="""
@Juan the best thing to do is tell people about git-annex, help them use it, and file bug reports. Just generally be part of the git-annex community.

(If you really want to donate to me, <http://campaign.joeyh.name/> is still open.)
"""]]


code:
flickrannex
=========

Hook program for gitannex to use flickr as backend

# Requirements:

    python2

Credit for the flickr api interface goes to: http://stuvel.eu/flickrapi
Credit for the png library goes to: https://github.com/drj11/pypng
Credit for the png tEXt patch goes to: https://code.google.com/p/pypng/issues/detail?id=65

# Install
Clone the git repository in your home folder.

    git clone git://github.com/TobiasTheViking/flickrannex.git

This should make a ~/flickrannex folder

# Setup
Make the file executable, and link it into PATH

    cd ~/flickrannex; chmod +x git-annex-remote-flickr; sudo ln -sf `pwd`/git-annex-remote-flickr /usr/local/bin/git-annex-remote-flickr

# Commands for gitannex:

    USERNAME="username@provider.com" git annex initremote flickr type=external externaltype=flickr encryption=shared folder=gitannex

An oauth authentication link should now be launched in the default browser. The hook will wait for 30s for you to login and authenticate.

    git annex describe dropbox "the flickr library"

# Notes

## Unencrypted mode
The photo name on flickr is currently the GPGHMACSHA1 version.

## Encrypted mode
The current version base64 encodes all the data, which results in ~35% larger filesize.

## Including directories as tags
This feature is currently disabled, if it gets implemented again it will most likely not require user action to enable it.

In this case the image:
   /home/me/annex-photos/holidays/2013/Greenland/img001.jpg
would get the following tags:  "holidays" "2013" "Greenland"
(assuming "/home/me/annex-photos" is the top level in the annex...)

Caveat Emptor - Tags will *always* be NULL for indirect repos - we don't (easily) know the human-readable file name.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8"
 nickname="develop"
 subject="comment 10"
 date="2013-06-07T09:39:59Z"
 content="""
I'm not even sure if chunksize is exposed to the hooks at all.

As it is, the hook will check the filesize, and if the filesize is more than 30mbyte it will exit 1.

Chunking may be implemented down the road. I do believe joeyh might have some plans that will touch this issue, so I'd rather wait. Than re-invent the wheel yet again.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnaH44G3QbxBAYyDwy0PbvL0ls60XoaR3Y"
 nickname="Nigel"
 subject="git annex get failed"
 date="2013-08-02T14:29:30Z"
 content="""
Hi, I am coming back to this and testing Flickr as a repository for moving files about and have run into what may be my very basic misunderstanding with vanilla annex.

I copied one file to Flickr and dropped it elsewhere (--force).  I assumed that the file was on Flickr ok but that the numcopies setting required the force because of the semi-trust level of the Flickr remote.

Then I find I can't get the file back, even though there is a record of it from whereis.

Can you help enlighten me as to what am I missing?  I assumed whereis would only report files that exist and can be copied back.  If not my error, I can raise bug or search for logs.  Thanks in advance for any help.

[[!format  perl \"\"\"

nrb@nrb-ThinkPad-T61:~/tmp$ git annex whereis
whereis libpeerconnection.log (3 copies)
  	31124688-0792-4214-9e00-7ed115aa6b8e -- flickr (the flickr library)
   	3e3d40d7-de8f-4591-a4ab-747d74a3b278 -- origin (my laptop)
   	ec2d64fc-30d6-48b4-99bf-7b1bc22d420d -- portable USB drive
ok
whereis test.cgi (1 copy)
  	31124688-0792-4214-9e00-7ed115aa6b8e -- flickr (the flickr library)
ok
whereis walkthrough.sh (3 copies)
  	31124688-0792-4214-9e00-7ed115aa6b8e -- flickr (the flickr library)
   	3e3d40d7-de8f-4591-a4ab-747d74a3b278 -- origin (my laptop)
   	ec2d64fc-30d6-48b4-99bf-7b1bc22d420d -- portable USB drive
ok
whereis walkthrough.sh~ (3 copies)
  	31124688-0792-4214-9e00-7ed115aa6b8e -- flickr (the flickr library)
   	3e3d40d7-de8f-4591-a4ab-747d74a3b278 -- origin (my laptop)
   	ec2d64fc-30d6-48b4-99bf-7b1bc22d420d -- portable USB drive
ok
nrb@nrb-ThinkPad-T61:~/tmp$ git annex get test.cgi
get test.cgi (from flickr...)

git-annex: /home/nrb/tmp/.git/annex/tmp/SHA256E-s48--a01eedbee949120aeda41e566f9ae8faef1c2bacaa6d7bb8e45050fb8df6d09d.cgi: rename: does not exist (No such file or directory)
failed
git-annex: get: 1 failed
nrb@nrb-ThinkPad-T61:~/tmp$

\"\"\"]]
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnaH44G3QbxBAYyDwy0PbvL0ls60XoaR3Y"
 nickname="Nigel"
 subject="re: git annex get failed"
 date="2013-08-02T15:02:14Z"
 content="""
Another try - this time a slightly simpler setup using my version of the walkthrough commands

[[!format  bash \"\"\"

nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex drop walkthrough.sh --from usbdrive
drop usbdrive walkthrough.sh ok
(Recording state in git...)
nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex move walkthrough.sh --to flickr
move walkthrough.sh (gpg) (checking flickr...) (to flickr...)
/home/nrb/repos/gits/flickrannex/flickrannex.py:92: FutureWarning: The behavior of this method will change in future versions.  Use specific 'len(elem)' or 'elem is not None' test instead.
  if res:
/home/nrb/repos/gits/flickrannex/flickrannex.py:100: FutureWarning: The behavior of this method will change in future versions.  Use specific 'len(elem)' or 'elem is not None' test instead.
  if res:
ok
(Recording state in git...)
nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex whereis
whereis walkthrough.sh (1 copy)
  	161b7af0-2075-4314-9767-308a49b86018 -- flickr (the flickr library)
ok
whereis walkthrough.sh~ (3 copies)
  	161b7af0-2075-4314-9767-308a49b86018 -- flickr (the flickr library)
   	7803d853-d231-4bb4-b696-f12a950fb96b -- here (my laptop)
   	d60d75f9-d878-4214-af20-fa055134ae77 -- usbdrive (portable USB drive)
ok
nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex get walkthrough.sh
get walkthrough.sh (from flickr...) (gpg)
git-annex: /home/nrb/repos/annex/laptop-annex/.git/annex/tmp/GPGHMACSHA1--02f600d7e8b071d2945270fd5e7fc26dd066ff31: openBinaryFile: does not exist (No such file or directory)
gpg: decrypt_message failed: eof

  Unable to access these remotes: flickr

  Try making some of these repositories available:
  	161b7af0-2075-4314-9767-308a49b86018 -- flickr (the flickr library)
failed
git-annex: get: 1 failed
nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex fsck --from flickr
fsck walkthrough.sh (gpg) (checking flickr...) (fixing location log)
  ** Based on the location log, walkthrough.sh
  ** was expected to be present, but its content is missing.

  ** No known copies exist of walkthrough.sh
failed
fsck walkthrough.sh~ (checking flickr...) (fixing location log)
  ** Based on the location log, walkthrough.sh~
  ** was expected to be present, but its content is missing.
failed
(Recording state in git...)
git-annex: fsck: 2 failed
nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$

\"\"\" ]]
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8"
 nickname="develop"
 subject="Version 0.1.10 pushed"
 date="2013-09-11T20:31:25Z"
 content="""
Since the initial release of this hook a lot of issues have been fixed, and a few features added.

I would highly suggest that everyone who is using this hook update to the latest version as i would consider one of the bugs to be fairly major.

"""]]


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="72.0.72.144"
 subject="great job on that plugin!"
 date="2014-04-15T04:47:17Z"
 content="""
it's pretty awesome to have 1TB of free storage like that out there... but for storing photos, it could be improved - I filed a few bugs on the github repo here:

https://github.com/TobiasTheViking/flickrannex/issues/created_by/anarcat?state=open

thanks!
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8"
 nickname="develop"
 subject="comment 2"
 date="2013-06-05T21:33:42Z"
 content="""
Get the statically linked version from here http://git-annex.branchable.com/install/Linux_standalone/

I believe the new hook format was introduced in version 4.20130521
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnaH44G3QbxBAYyDwy0PbvL0ls60XoaR3Y"
 nickname="Nigel"
 subject="missing configuration for flickr-checkpresent-hook"
 date="2013-06-05T20:44:25Z"
 content="""
<https://github.com/TobiasTheViking/flickrannex/issues/3>

9 days ago: [the annex] \"hook format a few versions ago, and this is using the new hook format\".

Looks very handy. I am just starting with this, but can't seem to get it working as a remote after following the simple walkthrough. All goes well until:

    $ git annex copy . --to flickr
    copy walkthrough.sh (checking flickr...)
      missing configuration for flickr-checkpresent-hook
    git-annex: checkpresent hook misconfigured

my Ubuntu 12.04:

    $ git annex version
    git-annex version: 4.20130516.1
    build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP
    local repository version: 3
    default repository version: 3
    supported repository versions: 3 4
    upgrade supported from repository versions: 0 1 2

I guess my \"git-annex version is still too old\"? Any idea what version is needed? Even better if I can figure out which Linux distribution/release has the most up to date version of annex.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8"
 nickname="develop"
 subject="comment 4"
 date="2013-06-05T22:02:29Z"
 content="""
The path for the binary \"/usr/bin/python2\" is wrong.

It could be any of /usr/bin/python /usr/bin/python2.6 /usr/bin/python2.7

Or maybe in /usr/local/bin

you can try running \"which python\" or \"which python2\" to get the real path.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnaH44G3QbxBAYyDwy0PbvL0ls60XoaR3Y"
 nickname="Nigel"
 subject="missing configuration for flickr-checkpresent-hook"
 date="2013-06-05T22:00:48Z"
 content="""
Many thanks.

I used gitannex-install and was left with a slight anomaly:

    Installing...........done
    git-annex version 4.20130601 has been installed
    $ git-annex version
    git-annex version: 4.20130531-g5df09b5

But I guess this includes the new hook format.  I get a bit further:

    $ git annex copy . --to flickr
    copy walkthrough.sh (checking flickr...) (user error (sh [\"-c\",\"/usr/bin/python2 /home/nrb/repos/gits/flickrannex/flickrannex.py\"] exited 1)) failed
    copy walkthrough.sh~ (checking flickr...) (user error (sh [\"-c\",\"/usr/bin/python2 /home/nrb/repos/gits/flickrannex/flickrannex.py\"] exited 1)) failed
    git-annex: copy: 2 failed

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnaH44G3QbxBAYyDwy0PbvL0ls60XoaR3Y"
 nickname="Nigel"
 subject="comment 5"
 date="2013-06-05T22:11:14Z"
 content="""
Thanks, but on my machine I get:

    $ which python2
    /usr/bin/python2

I have scripted all my walkthrough commands, blowing away the test repositories and flickr settings first each time.  This re-runs the flickr scripts and git config annex.flickr-hook etc.

I can't spot anything here.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmWg4VvDTer9f49Y3z-R0AH16P4d1ygotA"
 nickname="Tobias"
 subject="comment 6"
 date="2013-06-06T09:44:11Z"
 content="""
That's weird...

You could try adding \"--dbglevel 1 --stderr\" arguments to the hook command and give me the output. But the way i read the log it seems like it doesn't even launch the python intrepreter. I might be wrong though.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnaH44G3QbxBAYyDwy0PbvL0ls60XoaR3Y"
 nickname="Nigel"
 subject="Unencrypted flickr can only accept picture and video files"
 date="2013-06-06T10:24:58Z"
 content="""
Thanks and sorry to trouble you, it is my error, I picked unencrypted option (thinking it would be less of an issue) and am using a text file for test, gave an error line:

    10:53:07 [flickrannex-0.1.5] main : 'Unencrypted flickr can only accept picture and video files'

I've not looked through your code yet, but could that message be printed when not in debug mode?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmWg4VvDTer9f49Y3z-R0AH16P4d1ygotA"
 nickname="Tobias"
 subject="comment 8"
 date="2013-06-06T10:51:39Z"
 content="""
I'll make it so, in the next version i push.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawleVyKk2kQsB_HgEdS7w1s0BmgRGy1aay0"
 nickname="Milan"
 subject="chunksize"
 date="2013-06-07T09:09:56Z"
 content="""
Hi! Does this backend support chunksize option? If yes, is it possible to set it after the remote has been added to the repository?
Thanks, Milan.
"""]]


code:
[git-remote-gcrypt](https://github.com/joeyh/git-remote-gcrypt/)
adds support for encrypted remotes to git. The git-annex
[[gcrypt special remote|special_remotes/gcrypt]] allows git-annex to
also store its files in such repositories. Naturally, git-annex encrypts
the files it stores too, so everything stored on the remote is encrypted.

Here are some ways you can use this awesome stuff..

[[!toc ]]

This page will show how to set it up at the command line, but the git-annex
[[assistant]] can also be used to help you set up encrypted git
repositories.

## prerequisites

* Install
[git-remote-gcrypt](https://github.com/joeyh/git-remote-gcrypt/)
* Install git-annex version 4.20130909 or newer.

## encrypted backup drive

Let's make a USB drive into an encrypted backup repository. It will contain
both the full contents of your git repository, and all the files you
instruct git-annex to store on it, and everything will be encrypted so that
only you can see it.

First, you need to set up a gpg key. You might consider generating a
special purpose key just for this use case, since you may end up wanting to
put the key on multiple machines that you would not trust with your
main gpg key.

You need to tell git-annex the keyid of the key when setting up the
encrypted repository:

	git init --bare /mnt/encryptedbackup
	git annex initremote encryptedbackup type=gcrypt gitrepo=/mnt/encryptedbackup keyid=$mykey
	git annex sync encryptedbackup

Now you can copy (or even move) files to the repository. After
sending files to it, you'll probably want to do a sync, which pushes
the git repository changes to it as well.

	git annex copy --to encryptedbackup ...
	git annex sync encryptedbackup

Note that if you lose your gpg key, it will be *impossible* to get the
data out of your encrypted backup. You need to find a secure way to store a
backup of your gpg key. Printing it out and storing it in a safe deposit box,
for example.

You can actually specifiy keyid= as many times as you like to allow any one
of a set of gpg keys to access this repository. So you could add a friend's
key, or another gpg key you have.

To restore from the backup, just plug the drive into any machine that has
the gpg key used to encrypt it, and then:

	git clone gcrypt::/mnt/encryptedbackup restored
	cd restored
	git annex enableremote encryptedbackup gitrepo=/mnt/encryptedbackup
	git annex get --from encryptedbackup

## encrypted git-annex repository on a ssh server

If you have a ssh server that has rsync installed, you can set up an
encrypted repository there. Works just like the encrypted drive except
without the cable.

First, on the server, run:

	git init --bare encryptedrepo

(Also, install git-annex on the server if it's possible & easy to do so.
While this will work without git-annex being installed on the server, it
is recommended to have it installed.)

Now, in your existing git-annex repository, set up the encrypted remote:

	git annex initremote encryptedrepo type=gcrypt gitrepo=ssh://my.server/home/me/encryptedrepo keyid=$mykey
	git annex sync encryptedrepo

If you're going to be sharing this repository with others, be sure to also
include their keyids, by specifying keyid= repeatedly.

Now you can copy (or even move) files to the repository. After
sending files to it, you'll probably want to do a sync, which pushes
the git repository changes to it as well.

	git annex copy --to encryptedrepo ...
	git annex sync encryptedbackup

Anyone who has access to the repo it and has one of the keys
used to encrypt it can check it out:

	git clone gcrypt::ssh://my.server/home/me/encryptedrepo myrepo
	cd myrepo
	git annex enableremote encryptedrepo gitrepo=ssh://my.server/home/me/encryptedrepo
	git annex get --from encryptedrepo

## private encrypted git remote on hosting site

You can use gcrypt to store your git repository in encrypted form on any
hosting site that supports git. Only you can decrypt its contents.
Using it this way, git-annex does not store large files on the hosting site; it's
only used to store your git repository itself.

	git remote add encrypted gcrypt::ssh://hostingsite/myrepo.git
	git push encrypted master git-annex

Now you can carry on using git-annex with your new repository. For example,
`git annex sync` will sync with it.

To check out the repository from the hosting site, use the same gcrypt::
url you used when setting it up:

	git clone gcrypt::ssh://hostingsite/myrepo.git

## multiuser encrypted git remote on hosting site

Suppose two users want to share an encrypted git remote. Both of you
need to set up the remote, and configure gcrypt to encrypt it so that both
of you can see it.

	git remote add sharedencrypted gcrypt::ssh://hostingsite/myrepo.git
	git config remote.sharedencrypted.gcryt-participants "$mykey $friendkey"
	git push sharedencrypted master git-annex


code:
[[!comment format=mdwn
 username="tanen"
 ip="83.128.159.25"
 subject="comment 10"
 date="2013-11-04T17:58:36Z"
 content="""
> \"We could symetrically encrypt the repository with a keyfile that's stored in the repository itself\"
> Then you would need to decrypt the repository in order get the key you need to decrypt the repository. The impossibility of this design is why I didn't do that!

Sorry, I ment that the file containing the symmetric encryption key should obviously not be used to encrypt itself, it would be stored in the repository \"unencrypted\" (but protected with a passphrase)

> store a non-encrypted gpg key alongside the repository encrypted with it, but then you have to rely on a passphrase for all your security.

Exactly. I think such a mode be a great addition. It might not be as secure as encryption based on a private key - depending on the passphrase strength -, but it would certainly be a lot more convenient and portable (and still much more secure than the shared encryption method).
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlY4LE-xEyztjJeXSot7JIGGFWgBfabgwc"
 nickname="Peter"
 subject="GPG Key specification"
 date="2013-12-09T13:32:51Z"
 content="""
Hi there,

I try to follow the instructions provided here but I don't manage it to get my repo encrypted. Here are the steps:

1) git init --bare Encrypted
2) git-annex init
3) git annex initremote encryptedbackup type=gcrypt gitrepo=~/tmp/Encrypted encryption=pubkey keyid=DXXXX

The last step takes a lof of time to run. Basically my key doesn't get used at all:

    # ps -ax | grep gpg
    4743 pts/4    SL+    0:00 gpg --quiet --trust-model always --gen-random --armor 2 256
    4753 pts/0    S+     0:00 grep gpg

... instead a new pgp key is generated. How comes??!

Any help would be appreciated.

Thx and cheers,

cyneox

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 12"
 date="2013-12-12T19:27:23Z"
 content="""
@Peter, in your example, it *is* going to use your gpg key to encrypt files. gpg is being used to generate a 256 bit random value (not a key), which will be used as a random seed for HMAC scrambling of the keys stored in the encrypted special remote.

If that's taking too long to generate for your liking, you can pass --fast, which will make gpg use /dev/urandom to generate it rather than /dev/random.
"""]]


code:
[[!comment format=mdwn
 username="http://jspk.clavid.com/"
 nickname="flabbergast"
 subject="gcrypted repository structure"
 date="2013-12-30T12:07:44Z"
 content="""
I'm a bit confused about how do the gcrypted repositories actually work with git-annex. As far as I can tell, using git-remote-gcrypt with pure git produces a directory containing a couple of files with names looking like hashes. These files contain the whole repository - I checked by cloning the encrypted repo.

With git-annex: The instructions on this page suggest first creating a bare repo (creating a normal git repo layout). When I do this and then do the next step (`git annex initremote ...`) git-remote-gcrypt complains that the repository doesn't exist (which is correct from its point of view, as there is no encrypted repo yet) and creates a new one (so now there are both structures of a bare git repo _and_ an encrypted repo alongside in the same directory). The setup sort of works, but the bare git repo is never touched after that (or at least it shouldn't be, as it has nothing to do with the encrypted repo).

I've tried also following the instructions, only skipping the first step entirely (ie. no bare repo created). As far as I can tell, git-remote-gcrypt will run \"fine\" (will create a new encrypted repo), but git-annex itself complains that \"could not lock config file \<dir\>/.git/config\" and quits. Interestingly enough, the following gets around this \"problem\" and also results in a working setup (`~/annex` is a git-annex repo).

         cd ~/annex
         mkdir -p /tmp/test/.git
         touch /tmp/test/.git/config
         git annex initremote encr type=gcrypt gitrepo=/tmp/test keyid=\"XXX\"
         git annex sync encr

Then `git clone gcrypt::/tmp/test restored` will successfully recover the whole git-annex repo into `restored`.

So finally the question: is creating a bare git repo really necessary; and if not, is writing into `.git/config` necessary?

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 14"
 date="2013-12-31T19:26:55Z"
 content="""
@flabbergast, you seem to be confused about how git-remote-gcrypt stores its data. The data is stored as git commits inside a bare repository. That is why the instructions say to create a bare repository first. (I think it's also possible to use git-remote-gcrypt in a rsync mode where it just uploads encrypted files to an empty directory and does not use a bare git repository, but git-annex does not use it like that.)

Your mkdir and touch commands effecetively create a bare git repository too.

If you're having a problem, I suggest filing a bug report (not a comment on this page) with the full details. The examples show on this page have been tested, and work.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkbpbjP5j8MqWt_K4NASwv0WvB8T4rQ-pM"
 nickname="Fabrice"
 subject="Is there a way to specify a preferred pgp key?"
 date="2013-11-01T18:57:38Z"
 content="""
Hi,

I think the current behavior of the special remote is a bit annoying when one has several pgp keys.

Indeed, I've followed the encrypted backup drive example specifying the id of a dedicated key in the initremote step, so far so good. Doing that, I was prompted for my key phrase by the gnome keyring daemon, as expected.

The annoying part starts right at the git annex sync step. Indeed, when git-remote-gcrypt tries to decrypt the manifest from the encrypted remote, rather than trying only the key specified during the initremote step, it tries all my (secret) keys. This means that I get prompted for the key phrase of all those keys (minus the correct one which is already unlocked...).

In the future, this might possible to avoid by allowing gcrypt to fetch a preferred key from git config and to use with the --try-secret-key option available gnupg 2.1.x. But for 1.x or 2.0.x, the simpler option --default-key does not seem to alter the order in which keys are tried to decrypt the manifest. Also, it does not seem to be a problem of the gnome keyring daemon, but rather a gpg problem as when the daemon is replaced by the standard gpg-agent, the same problem occurs.

Meanwhile, is there any way to avoid this problem?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkbpbjP5j8MqWt_K4NASwv0WvB8T4rQ-pM"
 nickname="Fabrice"
 subject="A possible solution"
 date="2013-11-02T14:22:13Z"
 content="""
I'm answering to myself :-). A possible solution to the annoying pass phrase asking with current gnupg is to use a specialized secret keyring. One first exports the secret key used for this repository in a specific keyring as follows:

`gpg --export-secret-keys keyid | gpg --import --no-default-keyring --secret-keyring mygitannexsecret.gpg`

This will create a keyring in $HOME/.gnupg with only the specific key.

Then, in the git-remote-gcrypt shell script, gpg should be called as follows

`gpg --no-default-keyring --secret-keyring mygitannexsecret.gpg -q -d ...`

when decrypting the manifest in order to try only the specific key. This behavior can be easily triggered via some git configuration variable.

Any comment?

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 3"
 date="2013-11-02T17:32:28Z"
 content="""
Fabrice, I've filed a bug report about this: <https://github.com/blake2-ppc/git-remote-gcrypt/issues/9>
"""]]


code:
[[!comment format=mdwn
 username="tanen"
 ip="83.128.159.25"
 subject="comment 4"
 date="2013-11-03T22:35:07Z"
 content="""
The way I would want to setup git-annex (assistant) is \"Wuala/Spideroak style\": two computers with a full checkout of the repository, changes automatically being synced between them, even if the two computers are never online simultaneously, and encryption should be done locally: the (special) remote should not be able to view file listings or content.

Do I understand it correctly that the gcrypt remote is the only way to make this happen? I tried to create such a setup via the webapp but failed. Adding the repository and remote (via \"Encrypt with GnuPG key\") on the first computer went OK*, but trying to enable that remote on the other computer fails: clicking enable asks me for the SSH password, but after that I just get redirected to a blank screen, with nothing to see in the logfile after the succesful call to ssh-keygen. No entry for the second computer is being added to authorized_keys on the remote.

Perhaps this is because at this point the assistant is unable to actually parse the content of the encrypted repository? I tried importing the private key that was used while creating the repository on the other computer, but that made no difference.

Thinking about this for a while, I believe gpg keys aren't actually particularly suited for this usecase. Even without the bug above, one would either have to awkwardly copy a private key to all hosts that are syncing to the repository; or, every time a new (or reinstalled) host wants to sync the repository, you would manually have to add the new keyid to the config and do the forced push + GCRYPT_FULL_REPACK, presumably having to reupload your entire history. Apart from this, having to backup a private key (outside of your git-annex based backups!) would be quite inconvenient.

How would you feel about adding a new mode of operation where encryption is simply based on a passphrase? We could symetrically encrypt the repository with a keyfile that's stored in the repository itself, protecting the keyfile with a passphrase which - if stored at all - would be stored on the individual computers, outside of the repository.

*although it erroneously used \"E0D2F776E7F674E3\" as key-id while the actual id is E7F674E3; where did that other half come from?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmNu4V5fvpLlBhaCUfXXOB0MI5NXwh8SkU"
 nickname="Adam"
 subject="comment 5"
 date="2013-11-04T04:40:53Z"
 content="""
> How would you feel about adding a new mode of operation where encryption is simply based on a passphrase? We could symetrically encrypt the repository with a keyfile that's stored in the repository itself, protecting the keyfile with a passphrase which - if stored at all - would be stored on the individual computers, outside of the repository.

Isn't that what the regular shared-encryption remote already does?  Except it doesn't put a passphrase on the key, because anyone who has access to the local repo wouldn't need access to the remote one anyway.
"""]]


code:
[[!comment format=mdwn
 username="tanen"
 ip="83.128.159.25"
 subject="comment 7"
 date="2013-11-04T09:01:13Z"
 content="""
Thanks for the responses. Please correct me if I'm wrong, but the way I understood it, using the shared encryption scheme creates a conflict between \"changes being synced between them, even if the two computers are never online simultaneously\" and \"encryption should be done locally: the (special) remote should not be able to view file listings or content.\"

- If I use shared encryption \"the webapp way\", only the file contents will be rsynced to the remote, not the repository itself. This means that different hosts are unable to sync unless they are online simultaneously, so that commit data can be sent directly between them via XMPP. In practice, this would mean my hosts are never synced (because I don't keep my home computer running when I leave for work, and vice versa)

- If I use shared encryption and additionally put the repository itself on a remote, that remote would have the keys to fully decrypt the repository, that's not acceptable.

Reading through the docs again, the hybrid scheme actually seems to be closer to what I want than the shared scheme, but it still has a major downside: the encryption only applies to the files itself, so in order to get \"offline sync\" there still has to be a 'remote' for the repository itself, which will contain all your metadata unencrypted. And also it would depend on the user being able to manually setup and backup a set of gpg keys instead of just memorizing a secure passphrase.

@Fabrice Looks like the bug you found could very well be the cause of the problem I had; I'll try it again when a new version is available.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkbpbjP5j8MqWt_K4NASwv0WvB8T4rQ-pM"
 nickname="Fabrice"
 subject="comment 8"
 date="2013-11-04T10:31:56Z"
 content="""
I think you are (at least partially) right. Of course, the only way to sync completely computers that are not on together is to use either a usb drive or a third always on computer. (I've to confess I did not understand first when I read git annex docs, shame on me ;-) If you don't want to trust completely this computer (I don't, for instance), you must :

* use an encrypted git repository on this computer;

* and use either hybrid or pubkey encryption.

But contrarily to what you seem to imply (I hope I understand you correctly), if you do that, the third computer can still figure out a few things (usage patterns, such as where connections come from), but that's all. You've got full sync and everything is encrypted, both the git part and the files handled by the annex. This applied only to encrypted git special remotes as other remotes do not store the git part.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 9"
 date="2013-11-04T17:07:55Z"
 content="""
\"We could symetrically encrypt the repository with a keyfile that's stored in the repository itself\"

Then you would need to decrypt the repository in order get the key you need to decrypt the repository. The impossibility of this design is why I didn't do that!

It would certainly be possible to store a non-encrypted gpg key alongside the repository encrypted with it, but then you have to rely on a passphrase for all your security.

You should file a bug report for the bug you saw..
"""]]


code:
googledriveannex 0.2.0
=========

Hook program for gitannex to use Google Drive as backend

# Requirements:

    python2
    python-httplib2

Credit for the googledrive api interface goes to google

# Install
Clone the git repository in your home folder.

    git clone git://github.com/TobiasTheViking/googledriveannex.git

This should make a ~/googledriveannex folder

# Setup
Make the file executable, and link it into PATH

    cd ~/googledriveannex; chmod +x git-annex-remote-googledrive; sudo ln -sf `pwd`/git-annex-remote-googledrive /usr/local/bin/git-annex-remote-googledrive

# Commands for gitannex:

    git annex initremote googledrive type=external externaltype=googledrive encryption=shared folder=gitannex

An oauth authentication link should now be launched in the default browser. Authenticate and you will be proved with a code.

    OAUTH='authentication code' git annex initremote googledrive type=external externaltype=googledrive encryption=shared folder=gitannex
    git annex describe googledrive "the googledrive library"


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmlnEK_po7A0xMC9Xdp0FdcBgYllsCORXM"
 nickname="Johnny"
 subject="Shared google drive on multiple hosts"
 date="2014-03-14T16:40:11Z"
 content="""
I have now configured my Googledrive on Laptop1 according to the guide above, how do I add the same remote on Laptop2? It would be nice if this was described in the guide, include where the encryption keys are stored.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl64jV2rE8GMogJ6XuqESSkz78RVBgVdGw"
 nickname="Mesut"
 subject="I can't add google drive as remote"
 date="2014-04-10T07:55:56Z"
 content="""
Hi,

I am new to git-annex and I want to use google drive as remote but I can't.

I create syslink to `googledriveannex` in `/usr/local/bin`.

When I execute below command, command waiting but not make anything:

`$ git annex initremote googledrive type=external externaltype=googledrive encryption=shared folder=gitannex`

`initremote googledrive (encryption setup)` # Waiting but does not do anything.

What I am doing wrong?

Thanks for helps

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.191"
 subject="comment 3"
 date="2014-04-17T20:58:41Z"
 content="""
@Mesut, I think you're doing everything right. It can take a long time for the highly secure gpg key to be generated. Sit tight and let it finish, or you can pass --fast to generate a key that is a tiny bit less secure.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmp1ThsNNAbSn46ju-gwFELfStlhl8usJo"
 nickname="donkeyicydragon"
 subject="Googledrive annex on second repository"
 date="2014-04-19T20:45:23Z"
 content="""
Hi Johnny,

I wrote a patch for googledriveannex that fixed this problem for me. First you add the google drive special remote in repo1 then you clone repo1 into repo2. In repo2 you do \"git annex enableremote googldrivespecialremotename\" and it should work.
The problem was that the init method, that is called by git annex when a special remote is first created but also when it is enabled somewhere else, did not factor in the possibility that it had already been created.
I will simultaneously submit the patch to the author of the special remote plugin but here it is for you to quickly get going:

    diff --git a/git-annex-remote-googledrive b/git-annex-remote-googledrive
    index 49cd917..c8e70f3 100755
    --- a/git-annex-remote-googledrive
    +++ b/git-annex-remote-googledrive
    @@ -330,13 +330,16 @@ def initremote(line):
         oauth = os.getenv(\"OAUTH\") or \"\"
         encryption = common.getConfig(\"encryption\")
         myfolder = common.getConfig(\"folder\")
    -    stored_creds = sys.modules[\"__main__\"].login({\"oauth\": oauth})
    -    if len(myfolder) and stored_creds:
    -        common.sprint('SETCONFIG myfolder ' + myfolder + '')
    -        common.sprint('SETCONFIG stored_creds ' + json.dumps(stored_creds) + '')
    -        common.sprint('INITREMOTE-SUCCESS')
    +    if not common.getConfig(\"stored_creds\"):
    +	    stored_creds = sys.modules[\"__main__\"].login({\"oauth\": oauth})
    +	    if len(myfolder) and stored_creds:
    +	        common.sprint('SETCONFIG myfolder ' + myfolder + '')
    +	        common.sprint('SETCONFIG stored_creds ' + json.dumps(stored_creds) + '')
    +	        common.sprint('INITREMOTE-SUCCESS')
    +	    else:
    +	        common.sprint('INITREMOTE-FAILURE You need to set OAUTH environment variables and folder and encryption parameters when running initremote.')
         else:
    -        common.sprint('INITREMOTE-FAILURE You need to set OAUTH environment variables and folder and encryption parameters when running initremote.')
    +	        common.sprint('INITREMOTE-SUCCESS')
         common.log(\"Done\")

     def prepare(line):
    --
"""]]


code:
imapannex 0.2.0
=========

Hook program for gitannex to use imap as backend

# Requirements:

    python2

# Install
Clone the git repository in your home folder.

    git clone git://github.com/TobiasTheViking/imapannex.git

This should make a ~/imapannex folder

# Setup
Make the file executable, and link it into PATH

    cd ~/imapannex; chmod +x git-annex-remote-imap; sudo ln -sf `pwd`/git-annex-remote-imap /usr/local/bin/git-annex-remote-imap

# Commands for gitannex:

    USERNAME="username@provider.com" PASSWORD="password" git annex initremote imap type=external externaltype=imap encryption=shared folder=gitannex method="Normal password" ssl="SSL/TLS" host="imap.host.com" port="993"
    git annex describe imap "the imap library"


code:
megaannex 0.2.0
=========

Hook program for gitannex to use mega.co.nz as backend

# Requirements:

    python2
    requests>=0.10
    pycrypto

Credit for the mega api interface goes to: https://github.com/richardasaurus/mega.py

# Install
Clone the git repository in your home folder.

    git clone git://github.com/TobiasTheViking/megaannex.git

This should make a ~/megaannex folder

# Setup
Make the file executable, and link it into PATH

    cd ~/megaannex; chmod +x git-annex-remote-mega; sudo ln -sf `pwd`/git-annex-remote-mega /usr/local/bin/git-annex-remote-mega

# Commands for gitannex:

    USERNAME="user" PASSWORD="pword" git annex initremote mega type=external externaltype=mega encryption=shared folder=gitannex
    git annex describe mega "the mega.co.nz library"


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 1"
 date="2014-07-21T19:56:13Z"
 content="""
Note that there has apparently been an API break, so this special remote may not work anymore. See: <http://git-annex.branchable.com/forum/MegaAnnex_not_working./>
"""]]


code:
git-annex now has support for storing
[[arbitrary metadata|metadata]] about annexed files. For example, this can be
used to tag files, to record the author of a file, etc. The metadata is
synced around between repositories with the other information git-annex
keeps track of.

One nice way to use the metadata is through **views**. You can ask
git-annex to create a view of files in the currently checked out branch
that have certain metadata. Once you're in a view, you can move and copy
files to adjust their metadata further. Rather than the traditional
hierarchical directory structure, views are dynamic; you can easily
refine or reorder a view.

Let's get started by setting some tags on files. No views yet, just some
metadata:

[[!template id=note text="""
To avoid needing to manually tag files with the year (and month),
run `annex.genmetadata true`, and git-annex will do it for you
when adding files.
"""]]

	# git annex metadata --tag todo work/2014/*
	# git annex metadata --untag todo work/2014/done/*
	# git annex metadata --tag urgent work/2014/presentation_for_tomorrow.odt
	# git annex metadata --tag done work/2013/* work/2014/done/*
	# git annex metadata --tag work work
	# git annex metadata --tag video videos
	# git annex metadata --tag work videos/operating_heavy_machinery.mov
	# git annex metadata --tag done videos/old
	# git annex metadata --tag new videos/lotsofcats.ogv
	# git annex metadata --tag sound podcasts
	# git annex metadata --tag done podcasts/*/old
	# git annex metadata --tag new podcasts/*/recent

So, you had a bunch of different kinds of files sorted into a directory
structure. But that didn't really reflect how you approach the files.
Adding some tags lets you categorize the files in different ways.

Ok, metadata is in place, but how to use it? Time to change views!

	# git annex view tag=*
	view  (searching...)

	Switched to branch 'views/_'
	ok

[[!template id=note text="""
Notice that a single file may appear in multiple directories
depending on its tags. For example, `lotsofcats.ogv` is in
both `new/` and `video/`.
"""]]

This searched for all files with any tag, and created a new git branch
that sorts the files according to their tags.

	# tree -d
	work
	todo
	urgent
	done
	new
	video
	sound

Ah, but you're at work now, and don't want to be distracted by cat videos.
Time to filter the view:

	# git annex vfilter tag=work
	vfilter
	Switched to branch 'views/(work)/_'
	ok

Now only the work files are in the view, and they're otherwise categorized
according to their other tags. So you can check the `urgent/` directory
to see what's next, and look in `todo/` for other work related files.

Now that you're in a tag based view, you can move files around between the
directories, and when you commit your changes to git, their tags will be
updated.

	# git mv urgent/presentation_for_tomorrow_{work;2014}.odt ../done
	# git commit -m "a good day's work"
	metadata tag-=urgent
	metadata tag+=done

You can return to a previous view by running `git annex vpop`. If you pop
all the way out of all views, you'll be back on the regular git branch you
originally started from. You can also use `git checkout` to switch between
views and other branches.

## fields

Beyond simple tags and directories, you can add whatever kinds of metadata
you like, and use that metadata in more elaborate views. For example, let's
add a year field.

	# git checkout master
	# git annex metadata --set year=2014 work/2014
	# git annex metadata --set year=2013 work/2013
	# git annex view year=* tag=*

Now you're in a view with two levels of directories, first by year and then
by tag.

	# tree -d
	2014
	  |-- work
	  |-- todo
	  |-- urgent
	  `-- done
	2013
	  |-- work
	  `-- done

Oh, did you want it the other way around? Easy!

	# git annex vcycle
	# tree -d
	work
	  |-- 2014
	  `-- 2013
	todo
	  `-- 2014
	urgent
	  `-- 2014
	done
	  |-- 2014
	  `-- 2013

## location fields

Let's switch to a view containing only new podcasts. And since the
podcasts are organized into one subdirectory per show, let's
include those subdirectories in the view.

	# git checkout master
	# git annex view tag=new podcasts/=*
	# tree -d
	This_Developers_Life
	Escape_Pod
	GitMinutes
	The_Haskell_Cast
	StarShipSofa

That's an example of using part of the directory layout of the original
branch to inform the view. Every file gets fields automatically set up
corresponding to the directory it's in. So a file"foo/bar/baz/file" has
fields "/=foo", "foo/=bar", and "foo/bar/=baz". These location fields
can be used the same as other metadata to construct the view.

This has probably only scratched the surface of what you can do with views.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlFZFMFm-AapYAgV_V5X9HRJxWvhdiX9fY"
 nickname="Reiner"
 subject="comment 1"
 date="2014-03-24T21:11:31Z"
 content="""
I have played around with views and found out that I can create new tags by creating directories in the view and that I can created files in those new directories that are not contained in the original working tree. The behavoiur of git annex in this behaviour is a bit strange.

Assume for example you have a file \"foo\" with tag \"t1\" and switch to the tag view. Then create a directory \"t2\" and a file \"bar\" in it. Add the file, sync, and switch back to the master branch. If you enter the tag view again, the directory \"t2\" will be vanished, i.e. your newly created file is gone, too. This is not surprising, as the file has never been added to the original working tree. However, another \"git annex sync\" will restore the file.

I am unsure what behaviour I would expect, maybe it shouldn't be possible to files to a view in the first place, or newly created files might be collected in a separate branch. On the other hand, it seems reasonable to add a new file with a new tag at the same time. Anyway, I found it confusing that I can seemingly lose a file like this. It took me a bit of time to figure out that another sync recovers the file.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="possible to have a view based on file location info?"
 date="2014-06-08T03:55:24Z"
 content="""
For example, I'd like to have a view that only contains files present in this git-annex repository (no dangling symlinks).

"""]]


code:
Maybe you started out using the WORM backend, and have now configured
git-annex to use SHA1. But files you added to the annex before still
use the WORM backend. There is a simple command that can migrate that
data:

	# git annex migrate my_cool_big_file
	migrate my_cool_big_file (checksum...) ok

You can only migrate files whose content is currently available. Other
files will be skipped.

After migrating a file to a new backend, the old content in the old backend
will still be present. That is necessary because multiple files
can point to the same content. The `git annex unused` subcommand can be
used to clear up that detritus later. Note that hard links are used,
to avoid wasting disk space.


code:
Scenario
--------

You are a new git-annex user. You have already files spread around many computers and wish to migrate those into git-annex, without having to recopy all files all over the place.

Let's say, for example, you have a server, named `marcos` and a workstation named `angela`. You have your audio collection stored in `/srv/mp3` in `marcos` and `~/mp3` on `angela`, but only `marcos` has all the files, and `angela` only has a subset.

We also assume that `marcos` has an SSH server.

How do you add all this stuff to git-annex?

Create the biggest git-annex repository
---------------------------------------

Start with `marcos`, with the complete directory:

    cd /srv/mp3
    git init
    git annex init
    git annex add .
    git commit -m "git annex yay"

This will checksum all files and add them to the `git-annex` branch of the git repository. Wait for this process to complete.

Create the smaller repo and synchronise
---------------------------------------

On `angela`, we want to synchronise the git annex metadata with `marcos`. We need to initialize a git repo with `marcos` as a remote:

    cd ~/mp3
    git init
    git remote add marcos marcos.example.com:/srv/mp3
    git fetch marcos
    git annex info # this should display the two repos
    git annex add .

This will, again, checksum all files and add them to git annex. Once that is done, you can verify that the files are really the same as marcos with `whereis`:

    git annex whereis

This should display something like:

    whereis Orange Seeds/I remember.wav (2 copies)
            b7802161-c984-4c9f-8d05-787a29c41cfe -- marcos (anarcat@marcos:/srv/mp3)
            c2ca4a13-9a5f-461b-a44b-53255ed3e2f9 -- here (anarcat@angela)
    ok

Once you are sure things went on okay, you can synchronise this with `marcos`:

    git annex sync

This will push the metadata information to marcos, so it knows which files are available on `angela`. From there on, you can freely get and move files between the two repos!

Importing files from a third directory
--------------------------------------

Say that some files on `angela` are actually spread out outside of the `~/mp3` directory. You can use the `git annex import` command to add those extra directories:

    cd ~/mp3
    git annex import ~/music/

(!) Be careful that `~/music` is not a git-annex repository, or this will [[destroy it!|bugs/git annex import destroys a fellow git annex repository]].

Deleting deleted files
----------------------

It is quite possible some files were removed (or renamed!) on `marcos` but not on `angela`, since it was synchronised only some time ago. A good way to find out about those files is to use the `--not --in` argument, for example, on `angela`:

    git annex whereis --in here --not --in marcos

This will show files that are on `angela` and not on `marcos`. They could be new files that were only added on `angela`, so be careful! A manual analysis is necessary, but let's say you are certain those files are not relevant anymore, you can delete them from `angela`:

    git annex drop <file>

If the file is a renamed or modified version from the original, you may need to use `--force`, but be careful! If you delete the wrong file, it will be lost forever!

> (!) Maybe this wouldn't happen with [[direct mode]] and an fsck? --[[anarcat]]


code:
After you've used git-annex for a while, you will have data in your repository
that you don't want to keep in the limited disk space of a laptop or a server,
but that you don't want to entirely delete.

This is where git-annex's support for offline archive drives shines.
You can move old files to an archive drive, which can be kept offline if
it's not practical to keep it spinning. Better, you can move old files to
two or more archive drives, in case one of them later fails to spin up.
(One consideration when [[future_proofing]] your archive.)

To set up an archive drive, you can take any removable drive, format
it with a filesystem you'll be able to read some years later, and then follow
the [[walkthrough]] to set up a repository on it that is a git remote of
the repository in your computer you want to archive. In short:

	cd /media/archive
	git clone ~/annex
	cd ~/annex
	git remote add archivedrive /media/archive/annex
	git annex sync archivedrive

Don't forget to  tell git-annex this is an archive drive (or a backup
drive; see [[preferred_content]].). Also, give the drive a description that matches something you write on
its label, so you can find it later:

	git annex group archivedrive archive
	git annex wanted archivedrive standard
	git annex describe archivedrive "my first archive drive (SATA)"

Or you can use the assistant to set up the drive for you.
(Nice video tutorial here: [[videos/git-annex_assistant_archiving]])

(Keeping the archive drive in an offsite location? Consider encrypting
it! See [[fully_encrypted_git_repositories_with_gcrypt]].)

Then, when the archive drive is plugged in, you can easily copy files to
it:

	cd ~/annex
	git-annex copy --auto --to archivedrive

Or, if you're using the assistant, it will automatically notice when the drive
gets plugged in and copy files that need to be archived.

When you want to get rid of the local file, leaving only the copy on the
archive, you can just:

	git annex drop file

The archive drive has to be plugged in for this to work, so git-annex
can verify it still has the file. If you had configured git-annex to
always store 2 [[copies]], it will need 2 archive drives plugged in.
You may find it useful to configure a [[trust]] setting for the drive to
avoid needing to haul it out of storage to drop a file.

Now the really nice thing. When your archive drive gets filled up, you
can simply remove it, store it somewhere safe, and replace it with a new
drive, which can be mounted at the same location for simplicity. Set up
the new drive the same way described above, and use it to archive even more
files.

Finally, when you want to access one of the files you archived, you can
just ask for it:

	git annex get file

If necessary git-annex will tell you which archive drive you need to
pull out of storage to get the file back. This is where the description
you entered earlier comes in handy.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkC0W3ZQERUaTkHoks6k68Tsp1tz510nGo"
 nickname="Georg"
 subject="git annex sync"
 date="2013-10-06T08:59:24Z"
 content="""
Shouldn't it be
git annex sync archivedrive
instead of
git annex sync archive
in the first examples. As the name of the remote is \"archivedrive\", IMO \"sync\" should be called with the name of the remote.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkC0W3ZQERUaTkHoks6k68Tsp1tz510nGo"
 nickname="Georg"
 subject="git annex copy not working"
 date="2013-10-06T10:18:10Z"
 content="""
Is the feature \"git-annex copy --auto --to usb\" working?
I have created a backup repo on my usb drive but when I call \"git-annex copy --auto --to usb\" nothing happens.

I have called \"git annex group usb backup\" and \"git annex sync usb\" to set up the repo.

What is the correct way to get the data out to the backup repo?

Best regards, Georg
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.243"
 subject="thanks for your checking.."
 date="2013-10-06T17:04:24Z"
 content="""
The example was missing a preferred content setting, without which --auto doesn't copy anything unless needed to satisfy numcopies:

git annex wanted archivedrive standard
"""]]


code:
owncloudannex 0.2.0
=========

Hook program for gitannex to use owncloud as backend

# Requirements:

    python2
    python-requests

Credit for the webdav api interface goes to https://launchpad.net/python-webdav-lib

# Install
Clone the git repository in your home folder.

    git clone git://github.com/TobiasTheViking/owncloudannex.git

This should make a ~/owncloudannex folder

# Setup
Make the file executable, and link it into PATH

    cd ~/owncloudannex; chmod +x git-annex-remote-owncloud; sudo ln -sf `pwd`/git-annex-remote-owncloud /usr/local/bin/git-annex-remote-owncloud

# Commands for gitannex:

    USERNAME="login" PASSWORD="password" git annex initremote owncloud type=external externaltype=owncloud encryption=shared folder=gitannex url="https://host/remote.php/webdav/"
    git annex describe dropbox "the owncloud library"


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkQvRLq7nMMEGoEKuYx9oaf67IC0nZfmVI"
 nickname="chung yan"
 subject="owncloud hook exited nonzero"
 date="2013-07-10T05:03:47Z"
 content="""
hi

I got the above message, and cannot sync to my owncloud. Here is the log file i had:

>[2013-07-10 12:30:31 HKT] main: starting assistant version 4.20130627
(scanning...) [2013-07-10 12:30:31 HKT] Watcher: Performing startup scan
(started...) git-annex: Daemon is already running.
git-annex: Daemon is already running.
[2013-07-10 12:44:04 HKT] Committer: Adding sync2Servers.sh

>add syncByRsync/sync2Servers.sh (checksum...) [2013-07-10 12:44:04 HKT] Committer: Committing changes to git
(gpg)

>owncloud hook exited nonzero!
>git-annex: Daemon is already running.
[2013-07-10 12:51:07 HKT] main: Syncing with owncloud

>owncloud hook exited nonzero!
>[2013-07-10 12:53:10 HKT] Committer: Adding 2 files
ok
(Recording state in git...)
(Recording state in git...)
add syncByRsync/DSCN1810.JPG (checksum...) ok
add syncByRsync/DSCN1810.JPG (checksum...) [2013-07-10 12:53:10 HKT] Committer: Committing changes to git

>owncloud hook exited nonzero!
[2013-07-10 12:53:50 HKT] main: Syncing with owncloud
  owncloud hook exited nonzero!
  owncloud hook exited nonzero!

thanks

yan
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmLB39PC89rfGaA8SwrsnB6tbumezj-aC0"
 nickname="Tobias"
 subject="comment 2"
 date="2013-07-10T08:21:39Z"
 content="""
Personally i've only seen that when the server ran out of space. But lets see what is going on.

Please run this command

git config annex.owncloud-hook '/usr/bin/python2 ~/owncloudannex/owncloudannex.py --dbglevel 1 --stderr'

And then replicate the error. It should give me some debug information to work with.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkQvRLq7nMMEGoEKuYx9oaf67IC0nZfmVI"
 nickname="chung yan"
 subject="comment 3"
 date="2013-07-10T08:46:27Z"
 content="""
hi Tobias

Thanks your suggestion for list of my error log, here it is:

[2013-07-10 16:27:58 HKT] main: starting assistant version 4.20130627
(scanning...) [2013-07-10 16:27:58 HKT] Watcher: Performing startup scan
(started...) [2013-07-10 16:28:14 HKT] Committer: Adding 2 files

add syncByRsync/sync2Servers.sh (checksum...) ok
add syncByRsync/sync2Servers.sh (checksum...) [2013-07-10 16:28:14 HKT] Committer: Committing changes to git
[2013-07-10 16:31:12 HKT] Watcher: add direct DSCN1810.JPG
[2013-07-10 16:31:12 HKT] read: lsof [\"-F0can\",\"+d\",\"/home/yan/annex_testing/.git/annex/tmp/\"]
[2013-07-10 16:31:12 HKT] Committer: Adding DSCN1810.JPG
ok
(Recording state in git...)
(Recording state in git...)
add DSCN1810.JPG [2013-07-10 16:31:12 HKT](checksum...)  Watcher: add direct DSCN1810.JPG
[2013-07-10 16:31:12 HKT] read: sha256sum [\"/home/yan/annex_testing/.git/annex/tmp/DSCN181012023.JPG\"]

  DSCN1810.JPG changed while it was being added
[2013-07-10 16:31:12 HKT] Committer: delaying commit of 1 changes
[2013-07-10 16:31:13 HKT] read: lsof [\"-F0can\",\"+d\",\"/home/yan/annex_testing/.git/annex/tmp/\"]
[2013-07-10 16:31:13 HKT] Committer: Adding 2 files
failed
add DSCN1810.JPG (checksum...) [2013-07-10 16:31:13 HKT] read: sha256sum [\"/home/yan/annex_testing/.git/annex/tmp/DSCN181012023.JPG\"]
[2013-07-10 16:31:13 HKT] chat: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"hash-object\",\"-t\",\"blob\",\"-w\",\"--stdin\",\"--no-filters\"]
ok
add DSCN1810.JPG (checksum...) [2013-07-10 16:31:13 HKT] read: sha256sum [\"/home/yan/annex_testing/.git/annex/tmp/DSCN181012024.JPG\"]
[2013-07-10 16:31:13 HKT] chat: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"hash-object\",\"-t\",\"blob\",\"-w\",\"--stdin\",\"--no-filters\"]
[2013-07-10 16:31:13 HKT] Committer: committing 2 changes
[2013-07-10 16:31:13 HKT] Committer: Committing changes to git
[2013-07-10 16:31:13 HKT] feed: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"update-index\",\"-z\",\"--index-info\"]
[2013-07-10 16:31:13 HKT] read: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"commit\",\"--allow-empty-message\",\"--no-edit\",\"-m\",\"\",\"--quiet\",\"--no-verify\"]
[2013-07-10 16:31:13 HKT] Committer: queued Upload UUID \"df02d32a-7e3a-4e12-a417-7f1d1a1cf1a6\" DSCN1810.JPG Nothing : new file created
[2013-07-10 16:31:13 HKT] chat: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"hash-object\",\"-w\",\"--stdin-paths\",\"--no-filters\"]
[2013-07-10 16:31:13 HKT] feed: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"update-index\",\"-z\",\"--index-info\"]
[2013-07-10 16:31:13 HKT] read: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
[2013-07-10 16:31:13 HKT] read: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"write-tree\"]
[2013-07-10 16:31:13 HKT] chat: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"commit-tree\",\"a8337a989b58b29eee5fd2fa7a4c0b8ec45d5e59\",\"-p\",\"refs/heads/git-annex\"]
[2013-07-10 16:31:13 HKT] call: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"update-ref\",\"refs/heads/git-annex\",\"267bf35da4d9abe5ed7fe82ea5df8a8df2ddf940\"]
[2013-07-10 16:31:13 HKT] read: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"symbolic-ref\",\"HEAD\"]
[2013-07-10 16:31:13 HKT] read: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"show-ref\",\"refs/heads/master\"]
[2013-07-10 16:31:13 HKT] Transferrer: Transferring: Upload UUID \"df02d32a-7e3a-4e12-a417-7f1d1a1cf1a6\" DSCN1810.JPG Nothing
[2013-07-10 16:31:13 HKT] read: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"show-ref\",\"git-annex\"]
[2013-07-10 16:31:13 HKT] call: git-annex [\"transferkeys\",\"--readfd\",\"46\",\"--writefd\",\"36\"]
[2013-07-10 16:31:13 HKT] read: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
[2013-07-10 16:31:13 HKT] read: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"log\",\"refs/heads/git-annex..267bf35da4d9abe5ed7fe82ea5df8a8df2ddf940\",\"--oneline\",\"-n1\"]
[2013-07-10 16:31:13 HKT] read: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"show-ref\",\"git-annex\"]
[2013-07-10 16:31:13 HKT] read:[2013-07-10 16:31:13 HKT] read: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"ls-tree\",\"-z\",\"--\",\"refs/heads/git-annex\",\"uuid.log\",\"remote.log\",\"trust.log\",\"group.log\",\"preferred-content.log\"]
 git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
[2013-07-10 16:31:13 HKT] read: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"log\",\"refs/heads/git-annex..267bf35da4d9abe5ed7fe82ea5df8a8df2ddf940\",\"--oneline\",\"-n1\"]
[2013-07-10 16:31:13 HKT] chat: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"cat-file\",\"--batch\"]
(gpg) [2013-07-10 16:31:13 HKT] TransferWatcher: transfer starting: Upload UUID \"df02d32a-7e3a-4e12-a417-7f1d1a1cf1a6\" DSCN1810.JPG Nothing
[2013-07-10 16:31:13 HKT] chat: gpg [\"--batch\",\"--no-tty\",\"--use-agent\",\"--quiet\",\"--trust-model\",\"always\",\"--batch\",\"--passphrase-fd\",\"48\",\"--symmetric\",\"--force-mdc\"]
[2013-07-10 16:31:14 HKT] call: sh [\"-c\",\"/usr/bin/python2 /home/yan/annex/SocialBusiness/Resources/Infrastructure/Computer_Tools/Records/AsusU24/owncloudannex/owncloudannex.py --dbglevel 1 --stderr\"]

 16:31:16 [owncloudannex-0.1.1] <module> : 'START'
 16:31:16 [owncloudannex-0.1.1] main : 'ARGS: 'ANNEX_ACTION=store ANNEX_KEY=GPGHMACSHA1--0179fbbf559d5c25cf69b4e92025ff1f007d4f1f ANNEX_HASH_1=fp ANNEX_HASH_2=23 ANNEX_FILE=/home/yan/annex_testing/.git/annex/tmp/GPGHMACSHA1--0179fbbf559d5c25cf69b4e92025ff1f007d4f1f /home/yan/annex/SocialBusiness/Resources/Infrastructure/Computer_Tools/Records/AsusU24/owncloudannex/owncloudannex.py --dbglevel 1 --stderr''
 16:31:16 [owncloudannex-0.1.1] readFile : ''/home/yan/annex/SocialBusiness/Resources/Infrastructure/Computer_Tools/Records/AsusU24/owncloudannex/owncloudannex.conf' - 'r''
 16:31:16 [owncloudannex-0.1.1] readFile : 'Done'
 16:31:16 [owncloudannex-0.1.1] login : ''
 16:31:16 [owncloudannex-0.1.1] login : 'Using base: wingyan.no-ip.org - {'Authorization': 'Basic Y2h1bmd5YW41QGdtYWlsLmNvbTp5YW5fd2lraQ=='}'
 16:31:16 [owncloudannex-0.1.1] login : 'res: <davlib.DAV instance at 0x12915f0>'
 16:31:16 [owncloudannex-0.1.1] findInFolder : 'u'gitannex'(<type 'unicode'>) - '/'(<type 'str'>)'
 16:31:17 [owncloudannex-0.1.1] findInFolder : 'propfind: /owncloud/remote.php/webdav - '<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<d:error xmlns:d=\"DAV:\" xmlns:s=\"http://sabredav.org/ns\">\n  <s:exception>Sabre_DAV_Exception_NotAuthenticated</s:exception>\n  <s:message>Username or password does not match</s:message>\n  <s:sabredav-version>1.7.6</s:sabredav-version>\n</d:error>\n''
 16:31:17 [owncloudannex-0.1.1] findInFolder : 'Failure'
 16:31:17 [owncloudannex-0.1.1] createFolder : '/gitannex'
 16:31:18 [owncloudannex-0.1.1] createFolder : 'Failure: 401 - '<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<d:error xmlns:d=\"DAV:\" xmlns:s=\"http://sabredav.org/ns\">\n  <s:exception>Sabre_DAV_Exception_NotAuthenticated</s:exception>\n  <s:message>Username or password does not match</s:message>\n  <s:sabredav-version>1.7.6</s:sabredav-version>\n</d:error>\n''

  owncloud hook exited nonzero!
[2013-07-10 16:31:18 HKT] TransferWatcher: transfer finishing: Transfer {transferDirection = Upload, transferUUID = UUID \"df02d32a-7e3a-4e12-a417-7f1d1a1cf1a6\", transferKey = Key {keyName = \"911ba6148fbcbe4afe53772f1216b8204f403ed4ee06cb90c3c3ac25e56d9402.JPG\", keyBackendName = \"SHA256E\", keySize = Just 1893431, keyMtime = Nothing}}

I figured out it is a *Username or password does not match*, and i saw the content of owncloudannex.conf as \"uname\": \"myGmail@gmail.com\", but i quickly saw my WebDAV login to owncloud as my user name, not gmail address, so i changed this \"uname\": \"my_normal_user_name_not_gmail_acc\" inside owncloudannex.conf. Finally, i got it work. So, i think, user name should not be a gmail address, should be owncloud login user name.

Another issue, i had a look into /WebDAV.../gitannex/, it is git repos. file, for my user opinion, it is better that it is a real file content that we can see the file(such as photos) by owncloud web client directly, rather that owncloud is a file server to keep git repos only.

Thanks

yan
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmLB39PC89rfGaA8SwrsnB6tbumezj-aC0"
 nickname="Tobias"
 subject="comment 4"
 date="2013-07-10T08:54:13Z"
 content="""
Hm, an error like that it really should print without the need of debug information. I'll look into it.

And if you want the real file content, not encrypted, just change \"shared\" to \"none\" in the line:

git annex initremote owncloud type=hook hooktype=owncloud encryption=shared

I'm not sure if you can change this after you initialized the remote though. And also, the folder structure and filenames will be as you see them now on the owncloud. I'd need some more data from gitannex to make it show the real directory structure. But that doesn't seem feasible.

Why would you even want the data unencrypted on owncloud anyways? i mean on flickr, or googledocs, i kinda get it. but for owncloud?

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkQvRLq7nMMEGoEKuYx9oaf67IC0nZfmVI"
 nickname="chung yan"
 subject="comment 5"
 date="2013-07-10T09:43:55Z"
 content="""
hi Tobias

thanks your sharing, i am still in have a trial of git-annex, so i do not yet have real data on the server, so i just del. all and re-create both client and server repos in owncloud, i can got what you said, and see my photo.

Actually, i am studying git-annex. 1) It can syncs the data from my client computer to server. 2) i would like that other and anywhere computers through browser can view my server data which do not need to download all data into a new client. So my original goal is i can have a web to display the content of my files, and see owncloud have a nice web.

Regarding to GDoc, flickr, etc. they have space limitation, so i install owncloud as my own computer. I also see the <http://git-annex.branchable.com/design/assistant/partial_content/>, but it is not yet at roadmap.

thanks
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmLB39PC89rfGaA8SwrsnB6tbumezj-aC0"
 nickname="Tobias"
 subject="comment 6"
 date="2013-07-10T09:50:14Z"
 content="""
A 1 terabyte limit(for flickr) should be enough for most people. No?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkQvRLq7nMMEGoEKuYx9oaf67IC0nZfmVI"
 nickname="chung yan"
 subject="comment 7"
 date="2013-07-10T09:56:54Z"
 content="""
i have not got this 1 terabyte limit from flick before. I will have a look. I still consider some my own data, it is better to keep at my own server.
"""]]


code:
git-annex has a powerful syntax for making it act on only certain files.

The simplest thing is to exclude some files, using wild cards:

	git annex get --exclude '*.mp3' --exclude '*.ogg'

But you can also exclude files that git-annex's [[location_tracking]]
information indicates are present in a given repository. For example,
if you want to populate newarchive with files, but not those already
on oldarchive, you could do it like this:

	git annex copy --not --in oldarchive --to newarchive

Without the --not, --in makes it act on files that *are* in the specified
repository. So, to remove files that are on oldarchive:

	git annex drop --in oldarchive

Or maybe you're curious which files have a lot of copies, and then
also want to know which files have only one copy:

	git annex find --copies 7
	git annex find --not --copies 2

The above are the simple examples of specifying what files git-annex
should act on. But you can specify anything you can dream up by combining
the things above, with --and --or -( and -). Those last two strange-looking
options are parentheses, for grouping other options. You will probably
have to escape them from your shell.

Here are the mp3 files that are in either of two repositories, but have
less than 3 copies:

	git annex find --not --exclude '*.mp3' --and \
		-\( --in usbdrive --or --in archive -\) --and \
		--not --copies 3


code:
Suppose something goes wrong, and fsck puts all the files in lost+found.
It's actually very easy to recover from this disaster.

First, check out the git repository again. Then, in the new checkout:

	$ mkdir recovered-content
	$ sudo mv ../lost+found/* recovered-content
	$ sudo chown you:you recovered-content
	$ chmod -R u+w recovered-content
	$ git annex add recovered-content
	$ git reset HEAD recovered-content
	$ rm -rf recovered-content
	$ git annex fsck

The way that works is that when git-annex adds the same content that was in
the repository before, all the old links to that content start working
again. This works particularly well if the SHA* backends are used, but even
with the default backend it will work pretty well, as long as fsck
preserved the modification time of the files.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmWBvsZvSsAL8P2ye3F0OBStjFCVnOImzM"
 nickname="Jarno"
 subject="Why fsck?"
 date="2014-01-02T22:09:03Z"
 content="""
Other steps I understand but what is the purpose of the last, \"git annex fsck\"?
Shouldn't \"git annex add\" recreate any missing annex files and therefore fix dangling symlinks?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.35"
 subject="comment 2"
 date="2014-01-03T21:19:39Z"
 content="""
The `git annex fsck` at the end will verify if you've managed to recover all the files.
"""]]


code:
I have found this the most reliable way to recover from a corrupt git repository. I have had a lot of them lately, there might be a regression in btrfs in Ubuntu's Linux 3.8.0-33 (!).

1. Create a clone of a known good repository.
2. Add the clone as an object alternate to the broken repository.
3. Do a `git-repack -a -d` to lift the external objects into repo-local packs.
4. Remove the clone

[[!format sh """
$ cd /tmp/
$ git clone good-host:/path/to/good-repo
$ cd /home/user/broken-repo
$ echo /tmp/good-repo/.git/objects/ > .git/objects/info/alternates
$ git repack -a -d
$ rm -rf /tmp/good-repo
"""]]

... and push early, push often. ;-)


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.246"
 subject="comment 1"
 date="2013-11-11T16:01:59Z"
 content="""
Better, since version 4.20131024, `git annex repair` can be run to automatically do this, and more. (Including recovering data in corrupt git repositories that you forgot to push!)
"""]]


code:
[[!comment format=mdwn
 username="http://id.clacke.se/"
 nickname="clacke"
 subject="comment 2"
 date="2013-11-12T16:14:50Z"
 content="""
I am using git-annex 20131106 and I tried `git annex repair` first. It seems git is making assumptions that if I have object A then I must have object B that A depends on. Or maybe it freaks out because the object is not missing, just full of zeroes. I haven't done any analysis on exactly what situation causes this. When I have time, I will.
"""]]


code:
[[!comment format=mdwn
 username="http://id.clacke.se/"
 nickname="clacke"
 subject="comment 3"
 date="2013-11-12T16:17:42Z"
 content="""
Luckily, so far the objects getting corrupted have been in past commits (maybe only in packfiles?), not the latest ones, so I have been able to recover even though HEAD has been unique to the local repo.
"""]]


code:
[[!comment format=mdwn
 username="http://id.clacke.se/"
 nickname="clacke"
 subject="comment 4"
 date="2013-11-12T17:10:06Z"
 content="""
Hm. This is bad. My hunch that btrfs was the culprit seems to have been wrong. After having switched things around, along with lots of `git annex sync` in various places, I now have a corrupt repo on ext4. It must be git or git-annex that does something wrong.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.246"
 subject="comment 6"
 date="2013-11-12T18:10:21Z"
 content="""
If you can reliably corrupt a git repository, it's highly likely your hardware (disk or memory) is broken.
"""]]


code:
[[!comment format=mdwn
 username="http://id.clacke.se/"
 nickname="clacke"
 subject="comment 6"
 date="2013-11-12T17:22:01Z"
 content="""
This is my symptom too: [[forum/Git annex 'corrupting' itself]]
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.246"
 subject="comment 7"
 date="2013-11-12T18:12:49Z"
 content="""
> I tried git annex repair first. It seems git is making assumptions that if I have object A then I must have object B that A depends on. Or maybe it freaks out because the object is not missing

`git annex repair` is supposed to deal with these situations. If it fails to fix such a broken repository, please file a detailed bug report, ideally with a link to a copy of the repository.
"""]]


code:
[[!comment format=mdwn
 username="http://id.clacke.se/"
 nickname="clacke"
 subject="comment 8"
 date="2013-11-14T10:00:20Z"
 content="""
Several machines started showing this behavior around 4.20131106 or 4.20131101. I will find a way to reproduce when I can find the time.
"""]]


code:
Here's the scenario: You have a remote server you can ssh into,
and you want to use the git-annex webapp there, displaying back on your local
web browser.

Sure, no problem! It can even be done securely!

Let's start by making the git-annex repository on the remote server.

	git init annex
	cd annex
	git annex init

Now, you need to generate a private key and a certificate for HTTPS.
These files are stored in `.git/annex/privkey.pem` and
`.git/annex/certificate.pem` inside the git repository. Here's
one way to generate those files, using a self-signed certificate:

	(umask 077 ; openssl genrsa -out .git/annex/privkey.pem 4096)
	openssl req -new -x509 -key .git/annex/privkey.pem > .git/annex/certificate.pem

With those files in place, git-annex will automatically only accept HTTPS
connections. That's good, since HTTP connections are not secure over the
big bad internet.

All that remains is to make the webapp listen on the external interface
of the server. Normally, for security, git-annex only listens on localhost.
Tell it what hostname to listen on:

	git config annex.listen host.example.com

(If your hostname doesn't work, its IP address certianly will..)

When you run the webapp configured like that, it'll print out the
URL to use to open it. You can paste that into your web browser.

	git annex webapp
	http://host.example.com:42232/?auth=ea7857ad...

Notice that the URL has a big jumble of letters at the end -- this is a
secret token that the webapp uses to verify you're you. So random attackers
can't find your webapp and do bad things with it.

If you like, you can make the server run `git annex assistant --autostart`
on boot.

To automate opening the remote server's webapp in your local browser,
just run this:

	firefox "$(ssh host.example.com git annex webapp)"


code:
Sparkleshare and dvcs-autosync are tools to automatically commit your
changes to git and keep them in sync with other repositories. Unlike
git-annex, they don't store the file content on the side, but directly in
the git repository. Great for small files, less good for big files.

Here's how to use the [[git-annex assistant|/assistant]] to do the same
thing, but even better!

----

First, get git-annex version 4.20130329 or newer.

----

Let's suppose you're delveloping a video game, written in C. You have
source code, and some large game assets. You want to ensure the source
code is stored in git -- that's what git's for! And you want to store
the game assets in the git annex -- to avod bloating your git repos with
possibly enormous files, but still version control them.

All you need to do is configure git-annex to treat your C files
as small files. And treat any file larger than, say, 100kb as a large
file that is stored in the annex.

	git config annex.largefiles "largerthan=100kb and not (include=*.c or include=*.h)"

----

Now if you run `git annex add`, it will only add the large files to the
annex. You can `git add` the small files directly to git.

Note that in order to use `git add` on the small files, your repository
needs to be in indirect mode, rather than [[direct mode]]. If it's in
direct mode, `git add` will fail. You can fix that:

	git annex indirect

----

A less manual option is to run `git annex assistant`. It will *automatically*
add the large files to the annex, and store the small files in git.
It'll notice every time you modify a file, and immediately commit it,
too. And sync it out to other repositories you configure using `git annex
webapp`.

----

It's also possible to disable the use of the annex entirely, and just
have the assistant *always* put every file into git, no matter its size:

	git config annex.largefiles "exclude=*"


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 10"
 date="2013-12-12T17:35:28Z"
 content="""
The behavior is direct mode is a bug: [[bugs/when syncing a direct repository, git annex delete non annexed new git files]]. Hopefully it will be fixed soon.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 11"
 date="2013-12-12T19:58:53Z"
 content="""
The abovementioned bug is fixed in git, and will be in a release tomorrow.
"""]]


code:
[[!comment format=mdwn
 username="http://hands.com/~phil/"
 nickname="hands"
 subject="software from the future?"
 date="2013-03-31T17:30:34Z"
 content="""
I think you probably meant at least version 4.20130323 ;-)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-03-31T18:50:35Z"
 content="""
I meant 4.20130329
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawniayrgSdVLUc3c6bf93VbO-_HT4hzxmyo"
 nickname="Tobias"
 subject="Trying this feature"
 date="2013-04-14T13:04:55Z"
 content="""
I just gave this feature a try, but it seems it doesn't work as expected or maybe I don't understand it:

    ~/annex/largefilestest % git init
    ~/annex/largefilestest (git)-[master] % git annex init \"test repo\"
    ~/annex/largefilestest (git)-[master] % git config annex.largefiles \"not include=*.txt\"

Now I copy two files to this directory and add both to the annex

    ~/annex/largefilestest (git)-[master] % ll
    total 100
    -rw-rw-r-- 1 tobru tobru 93709 Oct 19 16:14 dpkg-get-selections.txt
    -rw-rw-r-- 1 tobru tobru  7256 Jan  6 15:52 x3400.jpg
    ~/annex/largefilestest (git)-[master] % git annex add .
    add x3400.jpg (checksum...) ok
    (Recording state in git...)
    ~/annex/largefilestest (git)-[master] % git status
    # On branch master
    #
    # Initial commit
    #
    # Changes to be committed:
    #   (use \"git rm --cached <file>...\" to unstage)
    #
    #       new file:   x3400.jpg
    #
    # Untracked files:
    #   (use \"git add <file>...\" to include in what will be committed)
    #
    #       dpkg-get-selections.txt
    ~/annex/largefilestest (git)-[master] % ll
    total 96
    -rw-rw-r-- 1 tobru tobru 93709 Oct 19 16:14 dpkg-get-selections.txt
    lrwxrwxrwx 1 tobru tobru   192 Jan  6 15:52 x3400.jpg -> .git/annex/objects/vf/QX/SHA256E-s7256--60e5b69ade5619e37f7fcaa964626da9c415959d861241aa13e2516fffc2dddf.jpg/SHA256E-s7256--60e5b69ade5619e37f7fcaa964626da9c415959d861241aa13e2516fffc2dddf.jpg

So the picture is added to the annex as expected. But the .txt file is not added to git. Do I have to manually add this to git? And why is the picture seen as new file by git?

The second question could be answered by: \"run git annex sync\". Is this correct? Because after running this command, git does not see this file as a new file anymore:

    ~/annex/largefilestest (git)-[master] % git annex sync
    commit
    [master (root-commit) a0afb14] git-annex automatic sync
     1 file changed, 1 insertion(+)
     create mode 120000 x3400.jpg
    ok
    git-annex: no branch is checked out
    ~/annex/largefilestest (git)-[master] % git status
    # On branch master
    # Untracked files:
    #   (use \"git add <file>...\" to include in what will be committed)
    #
    #       dpkg-get-selections.txt
    nothing added to commit but untracked files present (use \"git add\" to track)

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 4"
 date="2013-04-14T18:37:50Z"
 content="""
Like it says in the tip, `git annex add` will add the large files to git. You can add the small files with `git add`; git-annex won't do that for you.

To automatically add both sorts of files, you can use the `git annex watch` or `git annex assistant` daemons. The latter also keeps files in sync between repositories automatically.

(Why did the picture show up as a new file in git? Because you hadn't committed it. This is the same as when you `git add` a file;
it's only staged in the index; `git status` will show it is new until you `git commit`)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawniayrgSdVLUc3c6bf93VbO-_HT4hzxmyo"
 nickname="Tobias"
 subject="mimetypes"
 date="2013-05-01T20:37:33Z"
 content="""
Does `annex.largefiles` support mimetypes? F.e. `git config annex.largefiles \"not mimetype=text/plain\"`
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnPOttrEmm9CQYxzWrmgGN7LXy98gDkrlM"
 nickname="binet"
 subject="annex.largefiles and direct mode"
 date="2013-09-16T22:50:48Z"
 content="""
I was wondering if the annex.largefiles feature was compatible with direct mode?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.14.105"
 subject="comment 7"
 date="2013-09-19T18:03:29Z"
 content="""
annex.largefiles does not support mime types. I agree it would be a useful addition.

annex.largefiles can be used with direct mode. I would only recommending using it this way using the assistant, which will keep straight which files are which and commit them appropriately.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmLdL4zLty4fsfaIXNtPMS5RXs4y8OAk0o"
 nickname="Marc"
 subject="Disappearing files"
 date="2013-12-08T12:16:37Z"
 content="""
I've tried this with version 5.20131130, but my files disappear if I modify them on the remote end.

My setup:
- A local repository, direct mode, client group, annex.largefiles \"exclude=*.txt\"
- A remote one, also direct mode, backup group, annex.largefiles \"exclude=*.txt\"
Both are running the assistant.

If I create a .txt file locally, it gets committed and pushed to the remote as described. But, if I then modify that file on the remote end, the file gets deleted from both repositories. Also, if I create a file on the remote end, it's pushed to the local one (according to the log) but it never appears in the directory.

Changing the remote from 'backup' to 'client' group doesn't seem to make any difference.

Is there a 'best practice' on using git-annex like SparkleShare? I mean, syncing changes on all repositories but keeping a history of changes in git.

Thanks!
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmLdL4zLty4fsfaIXNtPMS5RXs4y8OAk0o"
 nickname="Marc"
 subject="comment 9"
 date="2013-12-08T20:41:41Z"
 content="""
After further testing, it seems that the setup I wanted works when both repos are set as indirect, instead of direct as comment 7 recommends. With both repos in indirect mode, the changes are propagated correctly and the files not selected by annex.largefiles are kept in git.
"""]]