code:
git-annex keeps track of in which repositories it last saw a file's content.
This location tracking information is stored in the git-annex branch.
Repositories record their UUID and the date when they get or drop
a file's content.

This location tracking information is useful if you have multiple
repositories, and not all are always accessible. For example, perhaps one
is on a home file server, and you are away from home. Then git-annex can
tell you what git remote it needs access to in order to get a file:

	# git annex get myfile
	get myfile (not available)
	  I was unable to access these remotes: home

Another way the location tracking comes in handy is if you put repositories
on removable USB drives, that might be archived away offline in a safe
place. In this sort of case, you probably don't have a git remotes
configured for every USB drive. So git-annex may have to resort to talking
about repository UUIDs. If you have previously used "git annex init"
to attach descriptions to those repositories, it will include their
descriptions to help you with finding them:

	# git annex get myfile
	get myfile (not available)
	  Try making some of these repositories available:
	  	c0a28e06-d7ef-11df-885c-775af44f8882  -- USB archive drive 1
	  	e1938fee-d95b-11df-96cc-002170d25c55

In certain cases you may want to configure git-annex to [[trust]]
that location tracking information is always correct for a repository.


code:
Variants of the git-annex logo.

[[logo_small.png]]

[[logo.svg]]

[[logo-old.png]]

[[logo-old_small.png]]

[[logo-old.svg]]

[[logo-old-bw.svg]]

repo_name: kakafarm/git

license: unknown


code:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- Created with Inkscape (http://www.inkscape.org/) -->

<svg
   xmlns:dc="http://purl.org/dc/elements/1.1/"
   xmlns:cc="http://creativecommons.org/ns#"
   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
   xmlns:svg="http://www.w3.org/2000/svg"
   xmlns="http://www.w3.org/2000/svg"
   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
   id="svg2"
   sodipodi:version="0.32"
   inkscape:version="0.47 r22583"
   width="251.1875"
   height="240.34375"
   version="1.0"
   sodipodi:docname="gitlogo.svg">
  <metadata
     id="metadata7">
    <rdf:RDF>
      <cc:Work
         rdf:about="Git Logo">
        <dc:format>image/svg+xml</dc:format>
        <dc:type
           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
        <dc:title></dc:title>
      </cc:Work>
    </rdf:RDF>
  </metadata>
  <defs
     id="defs5">
    <inkscape:perspective
       sodipodi:type="inkscape:persp3d"
       inkscape:vp_x="0 : 94 : 1"
       inkscape:vp_y="0 : 1000 : 0"
       inkscape:vp_z="97 : 94 : 1"
       inkscape:persp3d-origin="48.5 : 62.666667 : 1"
       id="perspective11" />
    <inkscape:perspective
       id="perspective3682"
       inkscape:persp3d-origin="0.5 : 0.33333333 : 1"
       inkscape:vp_z="1 : 0.5 : 1"
       inkscape:vp_y="0 : 1000 : 0"
       inkscape:vp_x="0 : 0.5 : 1"
       sodipodi:type="inkscape:persp3d" />
  </defs>
  <sodipodi:namedview
     inkscape:window-height="825"
     inkscape:window-width="1440"
     inkscape:pageshadow="2"
     inkscape:pageopacity="0.0"
     guidetolerance="10.0"
     gridtolerance="10.0"
     objecttolerance="10.0"
     borderopacity="1.0"
     bordercolor="#666666"
     pagecolor="#ffffff"
     id="base"
     inkscape:zoom="1.7347111"
     inkscape:cx="79.460121"
     inkscape:cy="96.862066"
     inkscape:window-x="0"
     inkscape:window-y="25"
     inkscape:current-layer="svg2"
     showguides="false"
     showgrid="false"
     inkscape:window-maximized="1" />
  <path
     sodipodi:nodetypes="ccccccccccccccccccc"
     id="path1927"
     d="m 115.03986,236.92929 c -15.769481,-2.03151 -31.998702,-16.94187 -34.26618,-34.43444 -2.174671,-14.62409 0.962744,-31.69413 13.051455,-41.34542 5.302266,-3.61399 12.514875,-7.66322 18.875285,-7.18134 0.1905,6.47429 -0.19049,14.34359 0,20.81788 -7.57578,0.84345 -14.909032,9.45432 -14.756408,17.33851 0.185705,10.26001 4.726478,19.68681 14.597668,22.75753 10.91859,3.67394 23.41726,2.62901 33.52469,-3.47905 8.34971,-6.50563 9.12218,-18.61836 4.75506,-27.83618 -1.96442,-5.07246 -6.19899,-8.79638 -11.94647,-8.78081 l 0,18.05778 -16.20641,0 0,-38.87566 49.07678,0 0,13.97989 -13.44239,0.12434 c 12.54392,7.15066 13.82967,22.58379 13.18688,33.29078 0.45188,18.39283 -17.71936,33.38895 -35.74198,35.39399 -6.52041,0.48886 -12.03738,0.51459 -20.70798,0.1722 z"
     style="fill:#666666;fill-opacity:1" />
  <path
     sodipodi:nodetypes="ccccc"
     id="path1925"
     d="m 79.977992,146.2455 -0.103031,-20.40175 91.999999,0 -0.0106,20.5 -91.886388,-0.0983 z"
     style="fill:#d8382d;fill-opacity:1" />
  <path
     sodipodi:nodetypes="ccccccccccccc"
     id="path1917"
     d="m 115.04163,118.67709 0,-21.83334 -35.666669,0 0,-19 35.666669,0 0,-24.5 21.33333,0 0,24.5 35.5,0 0,19 -35.5,0 -0.35168,21.78777 -20.98165,0.0456 z"
     style="fill:#40bf4c;fill-opacity:1" />
  <path
     style="fill:#40bf4c;fill-opacity:1;fill-rule:evenodd;stroke:none"
     d="M 21.28125,0 0,29.75 l 11.8125,0 c 0.02423,3.288346 0.302477,6.52723 0.78125,9.6875 L 31.375,36.59375 C 31.03908,34.358787 30.836567,32.072528 30.8125,29.75 l 11.75,0 L 21.28125,0 z M 32.84375,43.28125 14.6875,48.75 c 0.923692,3.063035 2.046663,6.043138 3.375,8.90625 L 35.28125,49.6875 c -0.956041,-2.062577 -1.776322,-4.198284 -2.4375,-6.40625 z m 5.6875,12.25 L 22.5625,65.8125 C 29.747774,76.954156 40.108616,85.860221 52.375,91.21875 L 59.96875,73.8125 C 51.136943,69.958114 43.695592,63.555939 38.53125,55.53125 z m 27.75,20.5 -5.125,18.3125 c 2.899536,0.810722 5.86596,1.421431 8.90625,1.84375 l 2.625,-18.8125 c -2.184482,-0.307312 -4.321319,-0.760497 -6.40625,-1.34375 z"
     id="path2836" />
  <path
     style="fill:#40bf4c;fill-opacity:1;fill-rule:evenodd;stroke:none"
     d="m 229.90625,0 -21.28125,29.75 11.8125,0 c -0.024,2.322528 -0.22658,4.608787 -0.5625,6.84375 l 18.78125,2.84375 c 0.47877,-3.16027 0.75698,-6.399154 0.78125,-9.6875 l 11.75,0 L 229.90625,0 z m -11.5,43.28125 c -0.66118,2.207966 -1.48146,4.343673 -2.4375,6.40625 l 17.21875,7.96875 c 1.32834,-2.863112 2.45131,-5.843215 3.375,-8.90625 l -18.15625,-5.46875 z m -5.6875,12.25 c -5.16434,8.024689 -12.60569,14.426864 -21.4375,18.28125 L 198.875,91.21875 C 211.14138,85.860221 221.50223,76.954156 228.6875,65.8125 L 212.71875,55.53125 z m -27.75,20.5 c -2.08493,0.583253 -4.22177,1.036438 -6.40625,1.34375 l 2.625,18.8125 c 3.04029,-0.422319 6.00671,-1.033028 8.90625,-1.84375 l -5.125,-18.3125 z"
     id="path3649" />
</svg>


code:
This wiki contains [[!pagecount pages="*"]] pages.

Broken links:

[[!brokenlinks ]]


code:
git-annex allows you to store arbitrary metadata about the content of files
stored in the git-annex repository. The metadata is stored in the
`git-annex` branch, and so is automatically kept in sync with the rest of
git-annex's state, such as [[location_tracking]] information.

Some of the things you can do with metadata include:

* Using `git annex metadata file` to show all
  the metadata associated with a file.
* [[tips/metadata_driven_views]]
* Limiting the files git-annex commands act on to those with
  or without particular metadata.
  For example `git annex find --metadata tag=foo --or --metadata tag=bar`
* Using it in [[preferred_content]] expressions.
  For example "metadata=tag=important or not metadata=author=me"

Each file (actually the underlying key) can have any number of metadata
fields, which each can have any number of values. For example, to tag
files, the `tag` field is typically used, with values set to each tag that
applies to the file.

The field names are limited to alphanumerics (and `[_-.]`), and are case
insensitive. The metadata values can contain absolutely anything you
like -- but you're recommended to keep it simple and reasonably short.

Here are some metadata fields that git-annex has special support for:

* `tag` - With each tag being a different value.
* `year`, `month` - When this particular version of the file came into
  being.
* `$field-lastchanged` - This is automatically maintained for each
  field that's set, and gives the date and time of the most recent
  change to the field. It cannot be modified directly.
* `lastchanged` - This is automatically maintained, giving the data and time
  of the last change to any of the metadata of a file.

To make git-annex automatically set the year and month when adding files,
run `git config annex.genmetadata true`. Also, see
[[tips/automatically_adding_metadata]].

git-annex's metadata can be updated in a distributed fashion. For example,
two users, each with their own clone of a repository, can set and unset
metadata at the same time, even for the same field of the same file.
When they push their changes, `git annex merge` will combine their
metadata changes in a consistent and (probably) intuitive way.

See [[the metadata design page|design/metadata]] for more details.


code:
[[!comment format=mdwn
 username="bremner"
 ip="198.164.160.48"
 subject="access metadata by key?"
 date="2014-03-17T01:26:44Z"
 content="""
I'm hacking around with using metadata from an external special remote. Those work with keys, not files, so one option would be to add a GETMETADATA to the protocol. It also seems like it would not be too hard to add
an option  to \"git annex metadata\" to take a key rather than a file.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.154"
 subject="comment 2"
 date="2014-03-17T19:32:39Z"
 content="""
I've made `git annex metadata --key` work.

I'll wait and see what you come up with your special remote and add something to the protocol later if it makes sense.
"""]]


code:
[[!if test="news/*" then="""
This is where announcements of new releases, features, and other news is
posted. git-annex users are recommended to subscribe to this page's RSS
feed.

[[!inline pages="./news/* and !./news/*/* and !*/Discussion" rootpage="news" show="30"]]

"""
else="""
(Please see the changelog.)
"""]]


code:
Similar to the yearly git user survey, I am doing a
[2013 git-annex user survey](http://git-annex-survey.branchable.com/polls/2013/).

If you use git-annex,
please take a few minutes to answer my questions!


code:
[Linux Weekly News](http://lwn.net/) has a nice
[article on git-annex](http://lwn.net/Articles/418337/) in it this week.


code:
git-annex will be briefly presented at FOSDEM, on Sunday February 4th at 15:40.
[Details](http://fosdem.org/2012/schedule/event/gitannex).

Thanks to Richard Hartmann for making this presentation.


code:
Thanks to my recent successful git-annex
[crowdfunding campaign](http://campaign.joeyh.name/), $1000 has been set
aside to award others for their contributions to git-annex by
the end of 2013. This is not a lot of money, but I hope that the
reward and recognition will encourage some more people to work on git-annex.

You don't need to know Haskell! You could contribute some interesting and
useful special remote hooks, or write better documentation.

On the other hand, if you have learned Haskell, this might be your first
chance to make some money with it! Some ideas for things to work on that
would be particularly award-worthy:

* Port git-annex to IOS, using [ghc-ios](https://github.com/ghc-ios).
  I don't plan to ever do this myself, but you could make it happen.
* Get the git-annex assistant working in Windows, building on the existing
  port to Windows. Otherwise, this is on my roadmap for February.
* Solve existing [[bugs]] or [[todo]] items. There are more than I can
  keep up with.

The amount of the awards will vary depending on the size of the
contribution. Since this is getting started later than is ideal,
I reserve the right to extend it past the end of the year if I don't get
enough participants.

If you'd like to participate in this program, just email me at
<id@joeyh.name>. --[[Joey]]


code:
[[!meta title="sharebox: a FUSE filesystem for git-annex"]]

Christophe-Marie Duquesne has just announced
[Sharebox](https://github.com/chmduquesne/sharebox-fs), a FUSE filesystem
relying on git-annex:

What are your goals?
Seamless synchronization "à la dropbox".
Ability to use with big binary files such as mp3/movies.
Entirely decentralized.
Don't use unnecessary space
Keep it simple: avoid special VCS commands and keep a filesystem
interface as much as possible.

While still alpha, this is promising. --[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlatTbI0K-qydpeYHl37iseqPNvERcdIMk"
 nickname="Tiago"
 subject="comment 1"
 date="2012-09-27T00:08:19Z"
 content="""
This is what the assistant should aim to be...\"Like Dropbox\", but even better.
I would guess many people who pledged were thinking of this.
"""]]


code:
git-annex 5.20140606 released with [[!toggle text="these changes"]]
[[!toggleable text="""
   * webapp: When adding a new local repository, fix bug that caused its
     group and preferred content to be set in the current repository,
     even when not combining.
   * webapp: Avoid stomping on existing description, group and
     preferred content settings when enabling or combining with
     an already existing remote.
   * assistant: Make sanity checker tmp dir cleanup code more robust.
   * unused: Avoid checking view branches for unused files.
   * webapp: Include ssh port in mangled hostname.
   * Windows: Fix bug introduced in last release that caused files
     in the git-annex branch to have lines teminated with \r.
   * Windows: Fix retrieving of files from local bare git repositories."""]]


code:
git-annex 5.20140613 released with [[!toggle text="these changes"]]
[[!toggleable text="""
   * Ignore setsid failures.
   * Avoid leaving behind .tmp files when failing in some cases, including
     importing files to a disk that is full.
   * Avoid bad commits after interrupted direct mode sync (or merge).
   * Fix build with wai 0.3.0.
   * Deal with FAT's low resolution timestamps, which in combination with
     Linux's caching of higher res timestamps while a FAT is mounted, caused
     direct mode repositories on FAT to seem to have modified files after
     they were unmounted and remounted.
   * Windows: Fix opening webapp when repository is in a directory with
     spaces in the path.
   * Detect when Windows has lost its mind in a timezone change, and
     automatically apply a delta to the timestamps it returns, to get back to
     sane values."""]]


code:
git-annex 5.20140707 released with [[!toggle text="these changes"]]
[[!toggleable text="""
   * assistant: Fix bug, introduced in last release, that caused the assistant
     to make many unncessary empty merge commits.
   * assistant: Fix one-way assistant->assistant sync in direct mode.
   * Fix bug in annex.queuesize calculation that caused much more
     queue flushing than necessary.
   * importfeed: When annex.genmetadata is set, metadata from the feed
     is added to files that are imported from it.
   * Support users who have set commit.gpgsign, by disabling gpg signatures
     for git-annex branch commits and commits made by the assistant.
   * Fix memory leak when committing millions of changes to the git-annex
     branch, eg after git-annex add has run on 2 million files in one go.
   * Support building with bloomfilter 2.0.0.
   * Run standalone install process when the assistant is started
     (was only being run when the webapp was opened).
   * Android: patch git to avoid fchmod, which fails on /sdcard.
   * Windows: Got rid of that pesky DOS box when starting the webapp.
   * Windows: Added Startup menu item so assistant starts automatically
     on login.
   * Windows: Fix opening file browser from webapp when repo is in a
     directory with spaces.
   * Windows: Assistant now logs to daemon.log."""]]


code:
git-annex 5.20140709 released with [[!toggle text="these changes"]]
[[!toggleable text="""
   * Fix race in direct mode merge code that could cause all files in the
     repository to be removed. It should be able to recover repositories
     experiencing this bug without data loss. See:
     http://git-annex.branchable.com/bugs/bad\_merge\_commit\_deleting\_all\_files/
   * Fix git version that supported --no-gpg-sign.
   * Fix bug in automatic merge conflict resolution, when one side is an
     annexed symlink, and the other side is a non-annexed symlink.
   * Really fix bug that caused the assistant to make many unncessary
     empty merge commits."""]]


code:
git-annex 5.20140717 released with [[!toggle text="these changes"]]
[[!toggleable text="""
   * Fix minor FD leak in journal code. Closes: #[754608](http://bugs.debian.org/754608)
   * direct: Fix handling of case where a work tree subdirectory cannot
     be written to due to permissions.
   * migrate: Avoid re-checksumming when migrating from hashE to hash backend.
   * uninit: Avoid failing final removal in some direct mode repositories
     due to file modes.
   * S3: Deal with AWS ACL configurations that do not allow creating or
     checking the location of a bucket, but only reading and writing content to
     it.
   * resolvemerge: New plumbing command that runs the automatic merge conflict
     resolver.
   * Deal with change in git 2.0 that made indirect mode merge conflict
     resolution leave behind old files.
   * sync: Fix git sync with local git remotes even when they don't have an
     annex.uuid set. (The assistant already did so.)
   * Set gcrypt-publish-participants when setting up a gcrypt repository,
     to avoid unncessary passphrase prompts.
     This is a security/usability tradeoff. To avoid exposing the gpg key
     ids who can decrypt the repository, users can unset
     gcrypt-publish-participants.
   * Install nautilus hooks even when ~/.local/share/nautilus/ does not yet
     exist, since it is not automatically created for Gnome 3 users.
   * Windows: Move .vbs files out of git\bin, to avoid that being in the
     PATH, which caused some weird breakage. (Thanks, divB)
   * Windows: Fix locking issue that prevented the webapp starting
     (since 5.20140707)."""]]


code:
[[!meta title="what git-annex is not"]]

* git-annex is not a backup system. It may be a useful component of an
  [[archival|use_case/bob]] system, or a way to deliver files to a backup
  system. For a backup system that uses git and that git-annex supports
  storing data in, see [[special_remotes/bup]].

* git-annex is not a filesystem or DropBox clone. However, the git-annex
  [[assistant]] is addressing some of the same needs in its own unique ways.
  (There is also a FUSE filesystem built on top of git-annex, called
  [ShareBox](https://github.com/chmduquesne/sharebox-fs).)

* git-annex is not unison, but if you're finding unison's checksumming
  too slow, or its strict mirroring of everything to both places too
  limiting, then git-annex could be a useful alternative.

* git-annex is more than just a workaround for git scalability
  limitations that might eventually be fixed by efforts like
  [git-bigfiles](http://caca.zoy.org/wiki/git-bigfiles). In particular,
  git-annex's [[location_tracking]] allows having many repositories
  with a partial set of files, that are copied around as desired.

* git-annex is not some flaky script that was quickly thrown together.
  I wrote it in Haskell because I wanted it to be solid and to compile
  down to a binary. And it has a fairly extensive test suite. (Don't be
  fooled by "make test" only showing a few dozen test cases; each test
  involves checking dozens to hundreds of assertions.)

* git-annex is not [git-media](https://github.com/schacon/git-media),
  although they both approach the same problem from a similar direction.
  I only learned of git-media after writing git-annex, but I probably
  would have still written git-annex instead of using it. Currently,
  git-media has the advantage of using git smudge filters rather than
  git-annex's pile of symlinks, and it may be a tighter fit for certain
  situations. It lacks git-annex's support for widely distributed storage,
  using only a single backend data store. It also does not support
  partial checkouts of file contents, like git-annex does.

* git-annex is similarly not [git-fat](https://github.com/jedbrown/git-fat),
  which also uses git smudge filters, and also lacks git-annex' widely
  distributed storage and partial checkouts.

* git-annex is also not [boar](http://code.google.com/p/boar/),
  although it shares many of its goals and characteristics. Boar implements
  its own version control system, rather than simply embracing and
  extending git. And while boar supports distributed clones of a repository,
  it does not support keeping different files in different clones of the
  same repository, which git-annex does, and is an important feature for
  large-scale archiving.

* git-annex is not the [Mercurial largefiles extension](http://mercurial.selenic.com/wiki/LargefilesExtension).
  Although mercurial and git have some of the same problems around large
  files, and both try to solve them in similar ways (standin files using
  mostly hashes of the real content).


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 10"
 date="2013-12-11T15:40:35Z"
 content="""
@Zellyn, what you describe does not sound unreasonable. But it's hard to say if it's a backup. For example, if you delete a file from the archive folder, and that happened to be the only copy of the file, it's gone.

It's definitely possible to use git-annex in backup-like ways, but what I want to discourage is users thinking that just putting files into git-annex means that they have a backup. Proper backups need to be designed, and tested. It helps to use software that is explicitly designed as a backup solution. git-annex is more about file distribution, and some archiving, than backups.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkQafKy7hNSEolLs6TvbgUnkklTctUY9LI"
 nickname="Zellyn"
 subject="Backup"
 date="2013-12-11T17:38:39Z"
 content="""
@joeyh - ok, good to know. So as long as I realize I need to sync with my hard drives before deleting, it should work fine as a backup solution. Sweet!
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkRtPz8CAz_1sBR0Rf-b8OlQQ49v9JxOIE"
 nickname="John"
 subject="re: Not an backup"
 date="2014-02-05T10:45:45Z"
 content="""
@joeyh.name But if I set numcopies=2 it won't let me drop the file right? I don't think we are mean to directly modify the archive; but if we do would git-annex detect the corruption and discourage us from dropping the other file?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="206.74.132.139"
 subject="comment 13"
 date="2014-02-06T17:00:59Z"
 content="""
Yes, git-annex ensures your configured [[numcopies|copies]] is met before dropping a file.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmhufs6QGCQXnUEc6qrCcQIZTomUDKNeAQ"
 nickname="Jeff"
 subject="Git annex in a strange direct with .git/annex/objects mode"
 date="2014-04-21T21:03:24Z"
 content="""
I'm doing something perhaps unreasonable and weird, and I'm wondering if there's a better way.

I'm running a wget -mbc of a particular web site.  It replicates down to a tree.
Then I'm ingesting the content into git annex via the normal 'git annex add' sequence.

Later, when I'm going to update my replica of the website, I am running a 'git annex unlock' on the whole tree (90 gig in this case), and then running the 'wget -mbc ; git annex add' command sequence again.

Is there any mechanism to convince git-annex to scan the file, and ingest (copy) it into objects if it is new content, while leaving the original files unlocked?   This would give me the ability to avoid the 'git annex unlock' copy operation, which is lengthy.

I'm aware this is inherently space inefficient.

I'm sure there's some other problem with this idea that I'm missing.

Thanks.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 15"
 date="2014-05-16T18:59:47Z"
 content="""
@Jeff, why did you post that comment here? Please use the [[forum]] for questions.

(You may find it useful to use direct mode, or `git annex import --skip-duplicates` or something.)
"""]]


code:
[[!comment format=mdwn
 username="http://bergey.dreamwidth.org/"
 ip="66.80.90.109"
 subject="git-media"
 date="2012-07-14T15:42:05Z"
 content="""
I haven't used git-media, but from the README it looks as though they now support several backends.  Might want to update the (very helpful!) comparison.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkurjhi0CRJvgm7QNaZDWS9hitBtavqIpc"
 nickname="Bret"
 subject="Sparkleshare"
 date="2012-09-06T08:09:17Z"
 content="""
How does [sparkleshare](http://sparkleshare.org/) and git-annex (and git-annex assistant) compare?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.236"
 subject="comment 3"
 date="2012-09-06T14:50:43Z"
 content="""
My understanding of sparkleshare (I've not used it) is that it uses a regular git repository, so has git's problems with large files and will not support partial checkouts. However, you might want to try it out and see if it works for you.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn4bbuawnh-nSo9pAh8irYAcV4MQCcfdHo"
 nickname="Stefan"
 subject="Sparkleshare"
 date="2012-09-15T01:28:05Z"
 content="""
Hi,

I used sparkleshare lately in a project involving 3 computers and 2 people. and for ascii texts and even a few smaller binary things it works ok.

But it does \"to much\" for media. at least at the moment, it just uses git for saving the data. That has a possitive and a negative aspect.

possitive:

1. you have a full history, if you delete a file its not gone for ever, so if you change it, the older version is still recoverable.
2. if you would as example use it from a laptop in a train without internet and you use a git server in the internet for the central server, and would change some files, then you or somebody else would write on the same txt file as example (html or something... latex...) you would be able to merge this files.
3. its not totaly bad for backup, because you can restore old files even if you delete it localy, because it will hold all history

negative:

1. for bigger data its cracy. if you use it for movies as example, you would in git annex delete some stuff you want not to see anytime again, so you would delete it everywhere. and its really away, not beeing still there in the history
2. git as it is has issues with saving/transfairing very big files, and its slow on even mid-sized files lets say 100 5mb big files it would be slow. because at the moment sparkleshare uses git all this disatvantages are there.
3. as many clients you use lets say a projekt with 10 people, each of them have all files and all the history of this projekt/directory on their pc.
4. you need a central data-store git folder you can use a seperate pc for that or save it on a client, if you use a client for that you have to save the data double on this pc.

(so you see for big files even if git would handle them faster you would waste massivly hard disk space) but again for pdfs a few pictures text files even some office files and stuff <100mb its great and easy to do.

I try it in a few words, sparkleshare is like dropbox but with file history ( I think dropbox dont have that???) but because git is not designed (yet) for big files it works somewhat ok for < 100mb stuff if you go very much higher > 1GB it will not be optimal.

git annex dont saves the data itself in git but only the locations and the checksums. so its more like a adress book of your data. its a abstraction layer to your data, you can see on as many devises as you want even without no netzwerk internet connection active and only a very small hd see all your 5 Terrabyte of Data you might have, and move around directories sort around them... delete stuff you dont want if you can deside that by the name... and then  when you come back to the connection you sync your actions and it does it to the files.

And one big feature like joey said is that you cannot partialy load files from the repos to your device if it has as example only enough space for 1/10 of it.

There is another thing, but because it is \"only\" a abstraction layer, it is theoreticaly easy to implement extentions to save your data on anything not only git repositories...

Sparkleshare will switch to something else than git, maybe but then it will switch to this single protocol and stick to that. because it does not abstract stuff so hard.

btw there is a alternative out there it forces you not to use git as vcs but you have to use a vcs (like git) and you dont have to use the client written in mono but only a smaller python script:

http://www.mayrhofer.eu.org/dvcs-autosync

but the idea behind it is the same except this 2 points ;)

but many free software developers dont like mono, so the change that it gets more love from more people is not totaly unlikely.

So way to long post but hope that helps somebody ;)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn4bbuawnh-nSo9pAh8irYAcV4MQCcfdHo"
 nickname="Stefan"
 subject="comment 5"
 date="2012-09-15T01:35:06Z"
 content="""
or to make it more simple ;)

sparkleshare is for proejects and maybe backup your documents folder

annex is for managing big binary files that not get modified most of the time and only added/synced or deleted.

hope thats on the point, try to start using it also now, but am a bit blowen away what it all can do and what not... and how to get a good use case, and mixing media-management with backup of home and thinking on solving that all with annex without having it used ever ;)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlatTbI0K-qydpeYHl37iseqPNvERcdIMk"
 nickname="Tiago"
 subject="comment 6"
 date="2012-09-27T10:17:18Z"
 content="""
Stefan: \"annex is for managing big binary files that not get modified most of the time and only added/synced or deleted.\"

While this is true,  the kickstarter title for assistant was \"Like dropbox\", and dropbox makes it transparent to edit files and they work with the filesystem.
So with assistant, lock/unlock should be automated and transparent to the user. Otherwise it's confusing and not simple at all to use, and at least OSX keeps giving errors because of the way it handles aliases.
So something like sharebox is essential to be included in assistant in my opinion.

"""]]


code:
[[!comment format=mdwn
 username="l3iggs"
 ip="24.130.145.126"
 subject="comment 7"
 date="2013-02-03T03:57:05Z"
 content="""
hi joey,

i'm excited by your project here but also confused by its direction. the kickstarter page has the header: \"git-annex assistant: Like DropBox, but with your own cloud.\" this page says \"git-annex is not a ... DropBox clone.\" these seem to be in direct opposition.

i'm looking for what is described by the header on your kickstarter page. i assume your backers are looking for the the same thing (a self hosted DropBox). for my use, dropbox is perfect, except for the fact that i have to pay a monthly fee to store my data on someone else's server when i would like to buy my own storage medium and run some open source dropbox clone on my own server.

can you explain more clearly what dropbox features your project lacks (/will lack)? and why where is a difference between your fundraising page and this one?

maybe i'm just confused by the difference between git-annex and git-annex assistant. does git-annex assistant truely aim to be a dropbox clone?
"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="99.54.57.201"
 subject="comment 8"
 date="2013-02-04T03:17:06Z"
 content="""
It's pretty much exactly what he said:

> git-annex is not a filesystem or DropBox clone. However, the git-annex assistant is addressing some of the same needs in its own unique ways.

The git-annex assistant is not exactly like DropBox; it's not a drop-in replacement that works exactly the way dropbox works.  But as it stands, right now, it can (like Dropbox) run in the background and make sure that all of your files in a special directory are mirrored to another place (a USB drive, or a server to which you have SSH access, or another computer on your home network, or another computer somewhere else which has access to the same USB drive from time to time, or has accesss to the same SSH server or S3 repository or....

It works as is but is still under heavy development and features are being added rapidly.  For example, up until a month or two ago, the files in your annex were replaced with softlinks whose content resided in a hidden directory. This caused some problems esp. on OS X where native programs don't handle softlinked files very gracefully. So Joey added an entirely new way of operating called \"direct mode\" which uses ordinary files, much like Dropbox does.

So -- what you should expect from git-annex assistant is a program which solves many of the same problems Dropbox does (keeping a set of files magically in sync across computers) but does it in its own way, which won't be *exactly* like Dropbox; it will be more flexible but might require a little learning to figure out exactly how to use it the way you want.  It's possible to get a very Dropbox-like system out of the box, especially now that you don't need to use softlinks, if you've got a place on the network you can use as a central remote repository for your files, or if you only want to synchronize two or more computers on the same local network.

\"git-annex\" itself is the plumbing used by git-annex assistant, or to put it another way, the engine that the assistant has under the hood.  Git-annex itself is extremely simple and stable but should only be used by people already familiar with the command line, perhaps even people already familiar with git.

That's my point of view as an enthusiastic user.  Joey may have his own perspective to share. :)

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkQafKy7hNSEolLs6TvbgUnkklTctUY9LI"
 nickname="Zellyn"
 subject="Not a backup system..."
 date="2013-12-10T20:55:05Z"
 content="""
I'd like to understand the \"not a backup system\" point better. My current plan was to use git annex assistant to save my annex folder onto two hard drives, one at home and one at work. Step two is to move most big stuff into .../archive/... folders, and step three is to add an annex folder on my wife's laptop, so we can use the hard drives to back up everything.

Does that sound unreasonable?
"""]]


code:
* [[2013 git-annex user survey|2013]]
* [[old polls|design/assistant/polls]]


code:
This has moved to a separate site due to load issues.

<http://git-annex-survey.branchable.com/polls/2013/>

Any votes you made here were carried over, so no need to vote twice!


code:
git-annex tries to ensure that the configured number of [[copies]] of your
data always exist, and leaves it up to you to use commands like `git annex
get` and `git annex drop` to move the content to the repositories you want
to contain it. But often, it can be good to have more fine-grained
control over which content is wanted by which repositories. Configuring
this allows the git-annex assistant as well as
`git annex get --auto`, `git annex drop --auto`, `git annex sync --content`,
etc to do smarter things.

Preferred content settings can be edited using `git
annex vicfg`, or viewed and set at the command line with `git annex wanted`.
Each repository can have its own settings, and other repositories will
try to honor those settings when interacting with it.
(So there's no local `.git/config` for preferred content settings.)

[[!template id=note text="""
### [[quickstart|standard_groups]]

Rather than writing your own preferred content expression, you can use
several standard ones included in git-annex that are tuned to cover different
common use cases.

You do this by putting a repository in a group,
and simply setting its preferred content to "standard" to match whatever
is standard for that group. See [[standard_groups]] for a list.
"""]]

The idea is that you write an expression that files are matched against.
If a file matches, the repository wants to store its content.
If it doesn't, the repository wants to drop its content
(if there are enough copies elsewhere to allow removing it).

To check at the command line which files are matched by preferred content
settings, you can use the --want-get and --want-drop options.

For example, `git annex find --want-get --not --in .` will find all the
files that `git annex get --auto` will want to get, and `git annex find
--want-drop --in .` will find all the files that `git annex drop --auto`
will want to drop.

The expressions are very similar to the matching options documented
on the [[git-annex]] man page. At the command line, you can use those
options in commands like this:

	git annex get --include='*.mp3' --and -'(' --not --largerthan=100mb -')'

The equivilant preferred content expression looks like this:

	include=*.mp3 and (not largerthan=100mb)

So, just remove the dashes, basically. However, there are some differences
from the command line options to keep in mind:

### difference: file matching

While --include and --exclude match files relative to the current
directory, preferred content expressions always match files relative to the
top of the git repository. Perhaps you put files into `archive` directories
when you're done with them. Then you could configure your laptop to prefer
to not retain those files, like this:

	exclude=*/archive/*

### difference: no "in="

Preferred content expressions have no direct equivilant to `--in`.

Often, it's best to add repositories to groups, and match against
the groups in a preferred content expression. So rather than
`--in=usbdrive`, put all the USB drives into a "transfer" group,
and use "copies=transfer:1"

### difference: dropping

To decide if content should be dropped, git-annex evaluates the preferred
content expression under the assumption that the content has *already* been
dropped. If the content would not be wanted then, the drop can be done.
So, for example, `copies=2` in a preferred content expression lets
content be dropped only when there are currently 3 copies of it, including
the repo it's being dropped from. This is different than running `git annex
drop --copies=2`, which will drop files that currently have 2 copies.

### difference: "present"

There's a special "present" keyword you can use in a preferred content
expression. This means that content is wanted if it's present,
and not otherwise. This leaves it up to you to use git-annex manually
to move content around. You can use this to avoid preferred content
settings from affecting a subdirectory. For example:

	auto/* or (include=ad-hoc/* and present)

Note that `not present` is a very bad thing to put in a preferred content
expression. It'll make it want to get content that's not present, and
drop content that is present! Don't go there..

### difference: "inpreferreddir"

There's a special "inpreferreddir" keyword you can use in a
preferred content expression of a special remote. This means that the
content is preferred if it's in a directory (located anywhere in the tree)
with a special name.

The name of the directory can be configured using
`git annex enableremote $remote preferreddir=$dirname`

(If no directory name is configured, it uses "public" by default.)

### difference: "standard"

git-annex comes with some built-in preferred content expressions, that
can be used with repositories that are in some [[standard_groups]].

When a repository is in exactly one such group, you can use the "standard"
keyword in its preferred content expression, to match whatever content
the group's expression matches.
(If a repository is put into multiple standard
groups, "standard" will match anything.. so don't do that!)

Most often, the whole preferred content expression is simply "standard".
But, you can do more complicated things, for example:
"`standard or include=otherdir/*`"

### difference: "groupwanted"

The "groupwanted" keyword can be used to refer to a preferred content
expression that is associated with a group. This is like the "standard"
keyword, but you can set up groupwanted preferred content expressions
using `git annex vicfg`.

Note that when writing a groupwanted preferred content expression,
you can use all of the keywords listed above, including "standard".
(But not "groupwanted".)

For example, to make a variant of the standard client preferred content
expression that does not want files in the "out" directory, you
could set `groupwanted client = standard and exclude=out/*`.
Then repositories that are in the client group and have their preferred
content expression set to "groupwanted" will use that, while
other client repositories that have their preferred content expression
set to "standard" will use the standard expression.

Or, you could make a new group, with your own custom preferred content
expression tuned for your needs, and every repository you put in this
group and make its preferred content be "groupwanted" will use it.

### difference: metadata matching

This:

	git annex get --metadata tag=done

becomes

	metadata=tag=done

## upgrades

It's important that all clones of a repository can understand one-another's
preferred content expressions, especially when using the git-annex
assistant. So using newly added keywords can cause a problem if
an older version of git-annex is in use elsewhere.

Before git-annex version 5.20140320, when git-annex saw a keyword it
did not understand, it defaulted to assuming *all* files were
preferred content. From version 5.20140320, git-annex has a nicer fallback
behavior: When it is unable to parse a preferred content expression,
it assumes all files that are currently present are preferred content.

Here are recent changes to preferred content expressions, and the version
they were added in.

* "standard" 5.20140314
  (only when used in a more complicated expression; "standard" by
  itself has been supported for a long time)
* "groupwanted=" 5.20140314
* "metadata=" 5.20140221
* "lackingcopies=", "approxlackingcopies=", "unused=" 5.20140127
* "inpreferreddir=" 4.20130501


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.22"
 subject="comment 10"
 date="2013-10-11T17:12:00Z"
 content="""
Georg, `drop --auto` will only drop files that are not preferred content. I'd need to know what preferred content expression you're using to say more.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlgyVag95OnpvSzQofjyX0WjW__MOMKsl0"
 nickname="Sehr"
 subject="Interplay with numcopies"
 date="2012-12-05T20:41:26Z"
 content="""
How does the preferred content settings interfere with the numcopies setting?

I could not get behind it. E.g. a case I do not unterstand:

I have a preferred setting evaluating to true and still

    git annex get --auto

does nothing, if the number of copies produced would surpass the numcopies setting.

Thx
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.253.113"
 subject="comment 2"
 date="2012-12-06T17:24:29Z"
 content="""
Yeah, that didn't make sense. I've fixed it, so it gets files if needed for either numcopies or preferred content.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.8.117"
 subject="comment 4"
 date="2012-12-10T19:46:01Z"
 content="""
It was a bug in the backup group's preferred content pagespec, introduced by the changes I made to fix the previous problem. Now fixed.
"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 4"
 date="2012-12-07T20:24:18Z"
 content="""
Built a new copy of git-annex yesterday.  I have a \"client\" on my macbook, and two \"backup\"s, one on an external HD, one on an ssh git remote.

git annex get --auto works beautifully!

It doesn't seem to work for copying content *to* a place where it's needed, though.

If I drop a file from my \"backup\" USB drive, and then go back to my macbook and do a \"git annex sync\" and \"git annex copy --to=usbdrive --auto\" it does not send the file out to the USB drive, even though by preferred content settings, the USB drive should \"want\" the file because it's a backup drive and it wants all content.

Similarly, if I add a new file on my macbook and then do a \"git annex copy --to=usbdrive auto\" it does not get copied to the USB drive.

Is this missing functionality, or should the preferred content setting for remotes only affect the assistant?
"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 5"
 date="2012-12-11T16:03:04Z"
 content="""
thanks!

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmRFKwny4rArBaz-36xTcsJYqKIgdDaw5Q"
 nickname="Andrew"
 subject="comment 6"
 date="2013-01-10T03:00:52Z"
 content="""
Is there a way to change these definitions for a given annex?

ie: in this repo make \"client\" mean

    present and exclude=*/archive/* and exclude=archive/*
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.238"
 subject="comment 7"
 date="2013-01-10T03:51:38Z"
 content="""
Sorry, there's not. The expressions used for \"standard\" are built in.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmRFKwny4rArBaz-36xTcsJYqKIgdDaw5Q"
 nickname="Andrew"
 subject="comment 8"
 date="2013-01-10T10:24:28Z"
 content="""
By way of a feature request: Maybe the way to do this is to have an additional keyword like \"config\" or \"repo\" that allows you to use vicfg and/or git config to set alternative rules and even additional group names.

In git config:

    annex.groups.<groupname> = present and exclude=*/archive/* and exclude=archive/*

in vicfg:

    # (for passport)
    #trust A0637025-ED47-4F95-A887-346121F1B4A0 = semitrusted

    # (for passport)
    group A0637025-ED47-4F95-A887-346121F1B4A0 = transfer

    # (for passport)
    preferred-content A0637025-ED47-4F95-A887-346121F1B4A0 = repo

    # (for transfer)
    group-content transfer = present and exclude=*/archive/* and exclude=archive/*

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkC0W3ZQERUaTkHoks6k68Tsp1tz510nGo"
 nickname="Georg"
 subject="drop only files located in archive"
 date="2013-10-08T17:53:08Z"
 content="""
Is there a way to drop only the files that are located in an \"archive\" directory?
I want to drop all files when calling

    git annex drop --auto

if I move them to the archive. But I want to keep the files that are outside of the archive, even if they are already present in other repos. As far as I have seen and tested, as soon as I have the files in an other repo all files get dropped, also the ones outside the archive directory. Or do I have to increase \"numcopies\" in order to circumvent the \"(not copies=semitrusted+:1)\" case?
"""]]


code:
git-annex comes with some built-in [[preferred_content]] settings, that can
be used with repositories that are in special groups. To make a
repository use one of these, just set its preferred content expression
to "standard", and put it in one of these groups. For example, to put the current
repository in the manual group and use this group to control the preferred
content, run

    git annex wanted . standard
    git annex group . manual

In the webapp, just edit the repository and select the group.

(Note that most of these standard expressions also make the repository
want to get any content that is only currently available on untrusted and
dead repositories. So if an untrusted repository gets connected,
any repository that can will back it up.)

The following standard groups are available:

### client

All content is wanted, unless it's for a file in a "archive" directory,
which has reached an archive repository, or is unused.

`(((exclude=*/archive/* and exclude=archive/*) or (not (copies=archive:1 or copies=smallarchive:1))) and not unused) or approxlackingcopies=1`

### transfer

Use for repositories that are used to transfer data between other
repositories, but do not need to retain data themselves. For
example, a repository on a server, or in the cloud, or a small
USB drive used in a sneakernet.

The preferred content expression for these causes them to get and retain
data until all clients have a copy.

`not (inallgroup=client and copies=client:2) and ($client)`

(Where $client is a copy of the preferred content expression used for
clients.)

The "copies=client:2" part of the above handles the case where
there is only one client repository. It makes a transfer repository
speculatively  prefer content in this case, even though it as of yet
has nowhere to transfer it to. Presumably, another client repository
will be added later.

### backup

All content is wanted. Even content of old/deleted files.

`include=* or unused`

### incremental backup

Only wants content that's not already backed up to another backup
or incremental backup repository.

`((include=* or unused) and (not copies=backup:1) and (not copies=incrementalbackup:1)) or approxlackingcopies=1`

### small archive

Only wants content that's located in an "archive" directory, and
only if it's not already been archived somewhere else.

`((include=*/archive/* or include=archive/*) and not (copies=archive:1 or copies=smallarchive:1)) or approxlackingcopies=1`

### full archive

All content is wanted, unless it's already been archived somewhere else.

`(not (copies=archive:1 or copies=smallarchive:1)) or approxlackingcopies=1`

Note that if you want to archive multiple copies (not a bad idea!),
you can set `groupwanted archive` to a version of
the above preferred content expression with a larger number of copies
than 1. Then make the archive repositories have a preferred
content expression of "groupwanted" in order to use your modified
version.

### source

Use for repositories where files are often added, but that do not need to
retain files for local use. For example, a repository on a camera, where
it's desirable to remove photos as soon as they're transferred elsewhere.

The preferred content expression for these causes them to only retain
data until a copy has been sent to some other repository.

`not (copies=1)`

### manual

This gives you nearly full manual control over what content is stored in the
repository. This allows using the [[assistant]] without it trying to keep a
local copy of every file. Instead, you can manually run `git annex get`,
`git annex drop`, etc to manage content. Only content that is already
present is wanted.

The exception to this manual control is that content that a client
repository would not want is not wanted. So, files in archive
directories are not wanted once their content has
reached an archive repository.

`present and ($client)`

(Where $client is a copy of the preferred content expression used for
clients.)

### public

This is used for publishing information to a repository that can be
publically accessed. Only files in a directory with a particular name
will be published. (The directory can be located anywhere in the
repository.)

The name of the directory can be configured using
`git annex enableremote $remote preferreddir=$dirname`

### unwanted

Use for repositories that you don't want to exist. This will result
in any content on them being moved away to other repositories. (Works
best when the unwanted repository is also marked as untrusted or dead.)

`exclude=*`


code:
[[!comment format=mdwn
 username="http://mildred.fr/"
 ip="2a01:e35:2f7b:8350:290:f5ff:feea:5546"
 subject="difference between source and unwanted"
 date="2014-04-02T22:25:26Z"
 content="""
What's the difference between `source` and `unwanted` ?

- `source` (`not copies=1`) will keep files that have less than 1 copies, meaning zero copies, meaning no files.
- `unwanted` will exclude all files.

Both gets to the same results, all files are moved elsewhere. Right?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.244"
 subject="comment 2"
 date="2014-04-02T22:44:14Z"
 content="""
@Mildred, I think both preferred content expressions will do the same thing. There is, however a difference between the groups: The webapp notices when all files have gone from an unwanted repository, and will help the user finish deleting the repository.
"""]]


code:
git-annex users entrust it with data that is often intensively private.
Here's some things to know about how to maintain your privacy while using
git-annex.

## browsing this web site

This website supports https. [Use it.](https://git-annex.branchable.com/privacy/)

## repository contents

In general, anyone who can clone a git repository gets the ability to see
all current and past filenames in the repository, and their contents.
It's best to assume this also holds true for git-annex, as a general rule.

There are some obvious exceptions: If you `git annex dropunused` old
content from all your repositories, then it's *gone*. If you `git annex
move` files to a offline drive then only those with physical access can see
their content. (The names of the files are still visible to anyone with a
clone of the repository.)

git-annex can encrypt data stored in special remotes. This allows you to
store files in the cloud without exposing their file names, or their
contents. See [[design/encryption]] for details.

When using the shared encryption method, the encryption key gets stored
in git, and so anyone who has a clone of your repository can decrypt files
from the encrypted special remote.

When using encryption with a GPG key or keys, only those with access to the
GPG key can decrypt the content of files stored in an encrypted special
remote.

## bug reporting

When you file a [[bug|bugs]] report on git-annex, you may need to provide
debugging output or details about your repository. In general, git-annex
does not sanitize `--debug` output at all, so it may include the names of
files or other repository details. You should review any debug or other
output you post, and feel free to remove identifying information.

Note that the git-annex assistant *does* sanitize XMPP protocol information
logged when debugging is enabled.

If you prefer not to post information publicaly, you can send a GPG
encrypted mail to Joey Hess <id@joeyh.name> (gpg key ID 2512E3C7).
Or you can post a public bug report, and send a followup email with private
details.


code:
git-annex repositories are often kept private, but here's a list of some of
the public repositories that you can clone to try out git-annex.

* [downloads.kitenet.net](http://downloads.kitenet.net/)
  `git clone https://downloads.kitenet.net/.git/`
  Various downloads of things produced by Joey Hess, including git-annex
  builds and videos.
* debconf-share
  `git clone http://annex.debconf.org/debconf-share/.git/`
  [DebConf](http://debconf.org/) Media, photos, videos, etc.
* [conference-proceedings](https://github.com/RichiH/conference_proceedings)
  `git clone https://github.com/RichiH/conference_proceedings.git`
  A growing collection of videos of technology conferences.
  Submit a pull request to add your own!
* [ocharles's papers](https://github.com/ocharles/papers)
  Lots of CS papers read by [Oliver](http://ocharles.org.uk/blog/).

This is a wiki -- add your own public repository to the list!
See [[tips/centralized_git_repository_tutorial]].


code:
Some folks have built other software on top of git-annex, or that is
designed to interoperate with it.

* The [[git-annex assistant|assistant]] is included in git-annex,
  and extends its use cases into new territory.
* [git-annex-watcher](https://github.com/rubiojr/git-annex-watcher)
  is a status icon for your desktop.
* [[forum/gadu_-_git-annex_disk_usage]] is a du like utility that
  is git-annex aware.
* [sizes](http://hackage.haskell.org/package/sizes) is another du-like
  utility, with a `-A` switch that enables git-annex support.
* Emacs Org mode can auto-commit attached files to git-annex.
* [git annex darktable integration](https://github.com/xxv/darktable-git-annex)
* Emacs's [Magit mode](http://www.emacswiki.org/emacs/Magit) has
  [magit integration](http://melpa.milkbox.net/?utm_source=dlvr.it&utm_medium=twitter#/magit-annex)


code:
Required content settings can be configured to do more complicated
things than just setting the required number of [[copies]] of your data.
For example, you could require that data be archived in at least two
archival repositories, and also require that one copy be stored offsite.

The format of required content expressions is the same as
[[preferred_content]] expressions.

Required content settings can be edited using `git annex vicfg`.
Each repository can have its own settings, and other repositories will
try to honor those settings when interacting with it.

While [[preferred_content]] expresses a preference, it can be overridden
by simply using `git annex drop`. On the other hand, required content
settings are enforced; `git annex drop` will refuse to drop a file if
doing so would violate its required content settings.
(Although even this can be overridden using `--force`).


code:
git-annex is designed for scalability. The key points are:

* Arbitrarily large files can be managed. The only constraint
  on file size are how large a file your filesystem can hold.

  While git-annex does checksum files by default, there
  is a [[WORM_backend|backends]] available that avoids the checksumming
  overhead, so you can add new, enormous files, very fast. This also
  allows it to be used on systems with very slow disk IO.

* Memory usage should be constant. This is a "should", because there
  can sometimes be leaks (and this is one of haskell's weak spots),
  but git-annex is designed so that it does not need to hold all
  the details about your repository in memory.

* Many files can be managed. The limiting factor is git's own
  limitations in scaling to repositories with a lot of files, and as git
  improves this will improve. Scaling to hundreds of thousands of files
  is not a problem, scaling beyond that and git will start to get slow.

  To some degree, git-annex works around inefficiencies in git; for
  example it batches input sent to certain git commands that are slow
  when run in an enormous repository.

* It can use as much, or as little bandwidth as is available. In
  particular, any interrupted file transfer can be resumed by git-annex.

## scalability tips

* If the files are so big that checksumming becomes a bottleneck, consider
  using the [[WORM_backend|backends]]. You can always `git annex migrate`
  files to a checksumming backend later on.

* If you're adding a huge number of files at once (hundreds of thousands),
  you'll soon notice that git-annex periodically stops and say
  "Recording state in git" while it runs a `git add` command that
  becomes increasingly expensive. Consider adjusting the `annex.queuesize`
  to a higher value, at the expense of it using more memory.


code:
[[!if test="enabled(shortcut)"
     then="This wiki has shortcuts **enabled**."
     else="This wiki has shortcuts **disabled**."]]

This page controls what shortcut links the wiki supports.

* [[!shortcut name=debbug url="http://bugs.debian.org/%S" desc="Debian bug #%s"]]
* [[!shortcut name=iki url="http://ikiwiki.info/%S/"]]
* [[!shortcut name=rfc url="https://www.ietf.org/rfc/rfc%s.txt" desc="RFC %s"]]
* [[!shortcut name=cve url="https://cve.mitre.org/cgi-bin/cvename.cgi?name=%s"]]
* [[!shortcut name=hackage url="http://hackage.haskell.org/package/%s"]]
* [[!shortcut name=commit url="http://source.git-annex.branchable.com/?p=source.git;a=commitdiff;h=%s"]]


code:
[[!img logo_small.png link=no]]

* [[install]]
* [[assistant]]
* [[walkthrough]]
* [[tips]]
* [[bugs]]
* [[todo]]
* [[forum]]
* [[comments]]
* [[contact]]
* [Flattr this](http://flattr.com/thing/84843/git-annex)


code:
[[!map pages="page(*) and !*/discussion and !recentchanges
and !bugs/* and !examples/*/* and !news/* and !tips/* and !videos/*
and !design/assistant/blog/*
and !forum/* and !todo/* and !users/* and !ikiwiki/*"]]


code:
git-annex can transfer data to and from configured git remotes.
Normally those remotes are normal git repositories (bare and non-bare;
local and remote), that store the file contents in their own git-annex
directory.

But, git-annex also extends git's concept of remotes, with these special
types of remotes. These can be used just like any normal remote by git-annex.
They cannot be used by other git commands though.

* [[S3]] (Amazon S3, and other compatible services)
* [[Amazon_Glacier|glacier]]
* [[bup]]
* [[ddar]]
* [[gcrypt]] (encrypted git repositories!)
* [[directory]]
* [[rsync]]
* [[webdav]]
* [[tahoe]]
* [[web]]
* [[xmpp]]
* [[hook]]

The above special remotes are built into git-annex, and can be used
to tie git-annex into many cloud services.

Here are specific instructions
for using git-annex with various services:

* [[Amazon_S3|tips/using_Amazon_S3]]
* [[Amazon_Glacier|tips/using_Amazon_Glacier]]
* [[tips/Internet_Archive_via_S3]]
* [[Box.com|tips/using_box.com_as_a_special_remote]]
* [[Google drive|tips/googledriveannex]]
* [[Google Cloud Storage|tips/using_Google_Cloud_Storage]]
* [[Mega.co.nz|tips/megaannex]]
* [[SkyDrive|tips/skydriveannex]]
* [[OwnCloud|tips/owncloudannex]]
* [[Flickr|tips/flickrannex]]
* [[IMAP|forum/special_remote_for_IMAP]]
* [[Usenet|forum/nntp__47__usenet special remote]]
* [chef-vault](https://github.com/3ofcoins/knife-annex/)
* [hubiC](https://github.com/Schnouki/git-annex-remote-hubic)
* [pCloud](https://github.com/tochev/git-annex-remote-pcloud)

Want to add support for something else? [[Write your own!|external]]

## Ways to use special remotes

There are many use cases for a special remote. You could use it as a
backup. You could use it to archive files offline in a drive with
encryption enabled so if the drive is stolen your data is not. You could
git annex move --to specialremote large files when your local drive is
getting full, and then git annex move the files back when free space is
again available. You could have one repository copy files to a special
remote, and then git annex get them on another repository, to transfer the
files between computers that do not communicate directly.

The git-annex assistant makes it easy to set up rsync remotes using this
last scenario, which is referred to as a transfer repository, and arranges
to drop files from the transfer repository once they have been transferred
to all known clients.

None of these use cases are particular to particular special remote types.
Most special remotes can all be used in these and other ways. It largely
doesn't matter for your use what underlying transport the special remote
uses.

## Unused content on special remotes

Over time, special remotes can accumulate file content that is no longer
referred to by files in git. Normally, unused content in the current
repository is found by running `git annex unused`. To detect unused content
on special remotes, instead use `git annex unused --from`. Example:

	$ git annex unused --from mys3
	unused mys3 (checking for unused data...)
	  Some annexed data on mys3 is not used by any files in this repository.
	    NUMBER  KEY
	    1       WORM-s3-m1301674316--foo
	  (To see where data was previously used, try: git log --stat -S'KEY')
	  (To remove unwanted data: git-annex dropunused --from mys3 NUMBER)
	$ git annex dropunused --from mys3 1
	dropunused 12948 (from mys3...) ok

## Testing special remotes

To make sure that a special remote is working correctly, you can use the
`git annex testremote` command. This expects you to have set up the remote
as usual, and it then runs a lot of tests, using random data. It's
particularly useful to test new implementations of special remotes.

By default it will upload and download files of around 1MiB to the remote
it tests; the `--size` parameter can adjust it test using smaller files.

It's safe to use this command even when you're already storing data in a
remote; it won't touch your existing files stored on the remote.

For most remotes, it also won't bloat the remote with any data, since
it cleans up the stuff it uploads. However, the bup, ddar, and tahoe
special remotes don't support removal of uploaded files, so be careful
with those.


code:
This special remote type stores file contents in a bucket in Amazon S3
or a similar service.

See [[tips/using_Amazon_S3]] and
[[tips/Internet_Archive_via_S3]] for usage examples.

## configuration

The standard environment variables `AWS_ACCESS_KEY_ID` and
`AWS_SECRET_ACCESS_KEY` are used to supply login credentials
for Amazon. You need to set these only when running
`git annex initremote`, as they will be cached in a file only you
can read inside the local git repository.

A number of parameters can be passed to `git annex initremote` to configure
the S3 remote.

* `encryption` - One of "none", "hybrid", "shared", or "pubkey".
  See [[encryption]].

* `keyid` - Specifies the gpg key to use for [[encryption]].

* `embedcreds` - Optional. Set to "yes" embed the login credentials inside
  the git repository, which allows other clones to also access them. This is
  the default when gpg encryption is enabled; the credentials are stored
  encrypted and only those with the repository's keys can access them.

  It is not the default when using shared encryption, or no encryption.
  Think carefully about who can access your repository before using
  embedcreds without gpg encryption.

* `datacenter` - Defaults to "US". Other values include "EU",
  "us-west-1", and "ap-southeast-1".

* `storageclass` - Default is "STANDARD". If you have configured git-annex
  to preserve multiple [[copies]], consider setting this to "REDUCED_REDUNDANCY"
  to save money.

* `host` and `port` - Specify in order to use a different, S3 compatable
  service.

* `bucket` - S3 requires that buckets have a globally unique name,
  so by default, a bucket name is chosen based on the remote name
  and UUID. This can be specified to pick a bucket name.

* `fileprefix` - By default, git-annex places files in a tree rooted at the
  top of the S3 bucket. When this is set, it's prefixed to the filenames
  used. For example, you could set it to "foo/" in one special remote,
  and to "bar/" in another special remote, and both special remotes could
  then use the same bucket.

* `x-amz-*` are passed through as http headers when storing keys
  in S3.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 10"
 date="2013-05-23T20:04:03Z"
 content="""
You can enable a special remote on a clone by running `git annex enableremote $name`, where $name is the name you used to originally create the special remote. (Older versions of git-annex used `git annex initremote` to enable the special remote on the clone.)

(Just in case, I have verified that embedcreds does cause the cipher= to be stored in the remote.log. It does.)
"""]]


code:
[[!comment format=mdwn
 username="basak"
 ip="2001:8b0:1c8::2"
 subject="comment 11"
 date="2013-05-24T09:38:40Z"
 content="""
Thanks Joey - initremote on my slightly older version appears to work. I'll use `enableremote` when I can.

> (Just in case, I have verified that embedcreds does cause the cipher= to be stored in the remote.log. It does.)

This doesn't do what I expect. The documentation suggests that my S3 _login_ credentials would be stored. I understand that the cipher would be stored; but isn't this a separate concept? Instead, I'm being asked to set `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`; my understanding was that git-annex will keep them in the repository for me, so that I don't have to set them after running `initremote` before cloning. This works, apart from surviving the cloning. I'm using `encryption=shared`; does this affect anything? Or am I using a version of git-annex (3.20121112ubuntu3) that's too old?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 12"
 date="2013-05-24T15:33:12Z"
 content="""
Ah -- No, your AWS creds are not stored. While some other special remotes, like webdav, can store all necessary credentials, it's not done for AWS. I didn't want git-annex to be responsible for someone accidentially publishing their AWS creds to their friends, since that could cost them a lot of money.
"""]]


code:
[[!comment format=mdwn
 username="basak"
 ip="2001:8b0:1c8::2"
 subject="comment 13"
 date="2013-05-24T15:47:14Z"
 content="""
That's not what the documentation here says! It even warns me: \"Think carefully about who can access your repository before using embedcreds without gpg encryption.\"

My use case:

Occasional use of EC2, and a desire to store some persistent stuff in S3, since the dataset is large and I have limited bandwidth. I want to destroy the EC2 instance when I'm not using it, leaving the data in S3 for later.

If I use git-annex to manage the S3 store, then I get the ability to clone the repository and destroy the instance. Later, I can start a new instance, push the repo back up, and would like to be able to then pull the data back out of S3 again.

I'd really like the login credentials to persist in the repository (as the documentation here says it should). Even if I have to add a --yes-i-know-my-s3-credentials-will-end-up-available-to-anyone-who-can-see-my-git-repo flag. This is because I use some of my git repos to store private data, too.

If I use an Amazon IAM policy as follows, I can generate a set of credentials that are limited to access to a particular prefix of a specific S3 bucket only - effectively creating a sandboxed area just for git-annex:

    {
      \"Statement\": [{\"Sid\": \"Stmt1368780615583\",
                     \"Action\": [\"s3:GetObject\", \"s3:PutObject\", \"s3:DeleteObject\"],
                     \"Effect\": \"Allow\",
                     \"Resource\": [\"arn:aws:s3:::bucketname/prefix/*\"]}
                    ],
      \"Statement\": [{\"Sid\": \"Stmt1368781573129\",
                    \"Action\": [\"s3:GetBucketLocation\"],
                    \"Effect\": \"Allow\",
                    \"Resource\": [\"arn:aws:s3:::bucketname\"]}
                   ]
    }

Doing this means that I have a different set of credentials for every annex, so it would be really useful to be able have these stored and managed within the repository itself. Each set is limited to what the annex stores, so there is no bigger compromise I have to worry about apart from the compromise of the data that the annex itself manages.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 14"
 date="2013-05-24T16:45:25Z"
 content="""
I apologise for incorrect information. I was thinking about defaults when using the webapp.

I have verified that embedcreds=yes stores the AWS creds, always.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawne_amN4fko4p5cRY_9EYwaYuJKNn7LRio"
 nickname="Tobias"
 subject="different s3 storage URLs"
 date="2013-08-23T08:59:32Z"
 content="""
Is it possible to change the S3 endpoint hosts? I'm running a radosgw with S3 support which I'd like to define as S3 remote for git-annex
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.63"
 subject="comment 16"
 date="2013-08-23T17:39:56Z"
 content="""
Yes, you can specify the host to use when setting up the remote. It's actually documented earlier on this very page, if ou search for \"host\". Any S3 compatabile host will probably work -- the Internet Archive's S3 does, for example.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnoUOqs_lbuWyZBqyU6unHgUduJwDDgiKY"
 nickname="Matt"
 subject="environment variables"
 date="2012-05-29T12:40:25Z"
 content="""
Just noting that the environment variables `ANNEX_S3_ACCESS_KEY_ID` and `ANNEX_S3_SECRET_ACCESS_KEY` seem to have been changed to `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.81.112"
 subject="comment 2"
 date="2012-05-29T19:10:46Z"
 content="""
Thanks, I've fixed that. (You could have too.. this is a wiki ;)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnoUOqs_lbuWyZBqyU6unHgUduJwDDgiKY"
 nickname="Matt"
 subject="comment 3"
 date="2012-05-30T00:26:33Z"
 content="""
Thanks! Being new here, I didn't want to overstep my boundaries. I've gone ahead and made a small edit and will do so elsewhere as needed.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmX5gPNK35Dub-HzR0Yb3KXllbqc0rYRYs"
 nickname="Eduardo"
 subject="bucket/folder s3 remotes"
 date="2012-08-09T10:52:07Z"
 content="""
it'd be really nice being able to configure a S3 remote of the form `<bucket>/<folder>` (not really a folder, of course, just the usual prefix trick used to simulate folders at S3). The remote = bucket architecture is not scalable at all, in terms of number of repositories.

how hard would it be to support this?

thanks, this is the only thing that's holding us back from using git-annex, nice tool!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 subject="comment 5"
 date="2012-08-09T18:01:06Z"
 content="""
I guess this could be useful if you have a *lot* of buckets already in use at S3, or if you want to be able to have a lot of distinct S3 special remotes. Implemented the `fileprefix` setting. Note that I have not tested it, beyond checking it builds, since I let my S3 account expire. Your testing would be appreciated.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnY9ObrNrQuRp8Xs0XvdtJJssm5cp4NMZA"
 nickname="alan"
 subject="Rackspace Cloud Files support?"
 date="2012-08-23T21:00:11Z"
 content="""
Any chance I could bribe you to setup Rackspace Cloud Files support?  We are using them and would hate to have a S3 bucket only for this.

https://github.com/rackspace/python-cloudfiles
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmyFvkaewo432ELwtCoecUGou4v3jCP0Pc"
 nickname="Eric"
 subject="S3 Remote Future Proof?"
 date="2013-01-20T09:21:50Z"
 content="""
Joey, I'm curious to understand how future proof an S3 remote is.  Can I restore my files without git-annex?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="68.119.100.5"
 subject="comment 8"
 date="2013-01-20T20:37:09Z"
 content="""
If encryption is not used, the files are stored in S3 as-is, and can be accessed directly. They are stored in a hashed directory structure with the names of their key used, rather than the original filename. To get back to the original filename, a copy of the git repo would also be needed.

With encryption, you need the gpg key used in the encryption, or, for shared encryption, a symmetric key which is stored in the git repo.

See [[future proofing]] for non-S3 specific discussion of this topic.
"""]]


code:
[[!comment format=mdwn
 username="basak"
 ip="2001:8b0:1c8::2"
 subject="Recovering from a clone"
 date="2013-05-22T18:32:05Z"
 content="""
How do I recover a special remote from a clone, please? I see that `remote.log` has most of the details, but my remote is not configured on my clone and I see no obvious way to do it. And I used `embedcreds`, but the only credentials I can see are stored in .git/annex/creds/ so did not survive a clone. I'm confused because the documentation here for `embedcreds` says that clones should have access.

As a workaround, it looks like copying the remote over from `.git/config` as well as the credentials from `.git/annex/creds/` seems to work. Is there some other way I'm supposed to do this, or is this the intended way?
"""]]


code:
This special remote type stores file contents in a
[bup](http://github.com/bup/bup) repository. By using git-annex
in the front-end, and bup as a remote, you get an easy git-style
interface to large files, and easy backups of the file contents using git.

This is particularly well suited to collaboration on projects involving
large files, since both the git-annex and bup repositories can be
accessed like any other git repository.

See [[walkthrough/using_bup]] for usage examples.

Each individual key is stored in a bup remote using `bup split`, with
a git branch named the same as the key name. Content is retrieved from
bup using `bup join`. All other bup operations are up to you -- consider
running `bup fsck --generate` in a cron job to generate recovery blocks,
for example; or clone bup's git repository to further back it up.

## configuration

These parameters can be passed to `git annex initremote` to configure bup:

* `encryption` - One of "none", "hybrid", "shared", or "pubkey".
  See [[encryption]].

* `keyid` - Specifies the gpg key to use for [[encryption]].

* `buprepo` - Required. This is passed to `bup` as the `--remote`
  to use to store data. To create the repository,`bup init` will be run.
  Example: "buprepo=example.com:/big/mybup" or "buprepo=/big/mybup"
  (To use the default `~/.bup` repository on the local host, specify "buprepo=")

Options to pass to `bup split` when sending content to bup can also
be specified, by using `git config annex.bup-split-options`. This
can be used to, for example, limit its bandwidth.

## notes

[[git-annex-shell]] does not support bup, due to the wacky way that bup
starts its server. So, to use bup, you need full shell access to the server.


code:
[[!comment format=mdwn
 username="http://sekenre.wordpress.com/"
 nickname="sekenre"
 subject="Synchronizing Bup repositories"
 date="2013-05-07T16:46:34Z"
 content="""
Hi All,

I managed to answer my questions above about copying changes between local bup repositories efficiently.

You run the following commands

    git annex copy . --to bup_repo_1                      # Uses bup split in the background (slow)
    rsync -av /mnt/repodisk1/repo/ /mnt/repodisk2/repo/ \
    --exclude=config --exclude=*.bloom --exclude=*.midx   # rsync without bup-specific indices (speed depends on delta between repositories)
    BUP_DIR=/mnt/repodisk2/repo/ bup midx -a && bup bloom # rebuild bup-specific indices on the target (this is extremely fast)
    git annex copy . --to bup_repo_2                      # Records file is now available in repo2 (also extremely fast)

Now `git annex whereis` will show the correct location and `git annex get <file> --from bup_repo_2` will work.

So far in my testing I haven't found any problems...

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnAvbXOnK57sqgvZvxkbG74NUKBDwKDcuk"
 nickname="Tim"
 subject="bup location data not synced through annex assistant"
 date="2013-05-15T15:08:54Z"
 content="""
I set up 2 servers running git annex assistant, both with a ~/annex dir and an additional ~/annex-bup bup repo. There is no additional cloud repository.
For test, I added my /etc dir which uploaded correctly from server1, but which never arrived on server2

    bup@bup1:~/annex/etc$ git annex whereis updatedb.conf
    whereis updatedb.conf (3 copies)
  	    687d3a7f-4798-4dbe-8774-1785b8ab6b7d -- here (bup@bup1:~/annex)
   	    adfc1307-771f-40e9-b794-bae2e1f21b8b -- bup2-annex-bup
   	    e4e0ac0b-992a-4312-a4ac-fc8d3d9f7c0f -- bup1-annex-bup
    ok

    bup@bup2:~/annex/etc$ git annex whereis updatedb.conf
    whereis updatedb.conf (1 copy)
  	    687d3a7f-4798-4dbe-8774-1785b8ab6b7d -- bup1 (bup@bup1:~/annex)
    ok

As you can see, server 2 just doesn't know the data is already on it's own disk in it's local bup repo.
Is there a reason this data does not get synced? Should I set up a transfer repo?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnAvbXOnK57sqgvZvxkbG74NUKBDwKDcuk"
 nickname="Tim"
 subject="my bad"
 date="2013-05-15T15:39:31Z"
 content="""
Sorry, looks like I did initremote twice on the same folder, instead of enableremote the second time...
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkgbXwQtPQSG8igdS7U8l031N8sqDmuyvk"
 nickname="Albert"
 subject="Error with bup and gnupg"
 date="2012-10-22T20:56:56Z"
 content="""
Hello,

I get this error when trying to use git-annex with bup and gnupg:

<pre>
move importable_pilot_surveys.tar (gpg) (checking localaseebup...) (to localaseebup...)
Traceback (most recent call last):
  File \"/usr/lib/bup/cmd/bup-split\", line 133, in <module>
    progress=prog)
  File \"/usr/lib/bup/bup/hashsplit.py\", line 167, in split_to_shalist
    for (sha,size,bits) in sl:
  File \"/usr/lib/bup/bup/hashsplit.py\", line 118, in _split_to_blobs
    for (blob, bits) in hashsplit_iter(files, keep_boundaries, progress):
  File \"/usr/lib/bup/bup/hashsplit.py\", line 86, in _hashsplit_iter
    bnew = next(fi)
  File \"/usr/lib/bup/bup/helpers.py\", line 86, in next
    return it.next()
  File \"/usr/lib/bup/bup/hashsplit.py\", line 49, in blobiter
    for filenum,f in enumerate(files):
  File \"/usr/lib/bup/cmd/bup-split\", line 128, in <genexpr>
    files = extra and (open(fn) for fn in extra) or [sys.stdin]
IOError: [Errno 2] No such file or directory: '-'
</pre>

I was able to work-around this issue by altering /usr/lib/bup/cmd/bup-split (though I don't think its a bup bug) to just pull from stdin:

files = [sys.stdin]

on ~ line 128.

Any ideas? Also, do you think that bup's data-deduplication does anything when gnupg is enabled, i.e. is it just as well to use a directory remote with gnupg?

Thanks! Git annex rules!

Albert
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.23"
 subject="comment 2"
 date="2012-10-23T20:01:43Z"
 content="""
@Albert, thanks for reporting this bug (but put them in [[bugs]] in future please).

This is specific to using the bup special remote with encryption. Without encryption it works. And no, it won't manage to deduplicate anything that's encrypted, as far as I know.

I think bup-split must have used - for stdin in the past, but now, it just reads from stdin when no file is specified, so I've updated git-annex.
"""]]


code:
[[!comment format=mdwn
 username="http://sekenre.wordpress.com/"
 nickname="sekenre"
 subject="Bup remotes in git-annex assistant"
 date="2013-03-13T12:54:56Z"
 content="""
Hi,

Is the bup remote available via the Assistant user interface?

Unrelated question;

If you are syncing files between two bup repos on local usb drives, does it use git to sync the changes or does it use \"bup split\" to re-add the file? (Basically, is the syncing as efficient as possible using git-annex or would I have to go to a lower level)

Many Thanks,
Sek
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 4"
 date="2013-03-13T16:05:50Z"
 content="""
I don't plan to support creating bup spefial remotes in the assistant, currently. Of course the assistant can use bup special remotes you set up.

Your two bup repos would be synced using bup-split.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnZWCbRYPVnwscdkdEDwgQHZJLwW6H_AHo"
 nickname="Tobias"
 subject="bup fail?"
 date="2013-03-31T21:05:32Z"
 content="""
I've run into problems storing a huge number of files in the bup repo. It seems that thousands of branches are a problem. I don't know if it's a problem of git-annex, bup, or the filesystem.

How about adding an option to store tree/commit ids in git-annex instead of using branches in bup?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 7"
 date="2013-04-02T21:24:06Z"
 content="""
`bup-split` uses a git branch to name the objects stored in the bup repository. So it will be limited by any scalability issues affecting large numbers of git branches. I don't know what those are.

Yes, it would be possible to make git-annex store this in the git-annex branch instead.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlsXhOlsW6RaGR83VNSMxPh159l5dFau70"
 nickname="Yung-Chin"
 subject="re scaling issue"
 date="2013-05-03T14:57:51Z"
 content="""
Tobias/joey,

I think there are at least two scaling issues that may be causing you trouble. One is that bup writes pack+idx files rather than bare objects, and if you send 1 file per call to bup-split, you end up with a pair of pack and idx files for each such call. When you later try to retrieve a blob, bup currently just calls git, and git will have to traverse all these tiny idx files looking for the right hash (bare objects you could at least find by name). You can probably ameliorate the pain by calling git repack (look at the -a and --max-pack-size switches) on your bup repository. The other is the \"thousands of branches\" issue, and I think \"git pack-refs --all\" (that's again on your bup repository) might help a little bit.

It would certainly help performance if you could store blob/tree ids in git-annex instead of branch names. For small files, all bup would need to store is a blob, but currently you end up storing a blob, a tree, and a commit (and looking-up all of those, plus the ref too, on calling bup-join). (you might want to patch bup-split, so it would allow you to ask it for \"--blob-or-tree\", because currently if you say you pass it -b for blob-ids, then for bigger files you get a series of IDs, whereas you'd be much better off with a tree-id there)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlsXhOlsW6RaGR83VNSMxPh159l5dFau70"
 nickname="Yung-Chin"
 subject="comment 9"
 date="2013-05-03T16:34:05Z"
 content="""
Thinking about this some more, a very elegant way to make a bup remote could actually be to just pass the whole .git/annex tree into bup-index/save (you could avoid sending some files by only bup-indexing select subtrees, or by using --exclude-*'s, but you'd run bup-save over the whole .git/annex tree). You could then use bup-restore to retrieve files or whole subtrees, and you'd refer to the files you're retrieving by their actual pathname under which they live in .git/annex (if that doesn't make sense it's because I've misunderstood how git-annex is organised!), so something like \"bup restore branch_name/latest/.git/annex/aa/bb/sha-of-some-sort\" would work - that's cute, right? And you'd only have 1 branch.

However... somebody who is good with lazy-evaluation would need to rework bup.vfs: currently, if you'd call bup-restore on a path like that, it would instantiate a _lot_ of vfs-nodes you don't need - to begin with, it would make a node for every commit you ever made (on any branch!) - on a big repository you'd wait ages for it to just find the commit objects...
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkQqKSVY98PVGDIaYZdK9CodJdbh7cFfhY"
 nickname="Ashwin"
 subject="Rackspace Cloud Files support"
 date="2013-03-22T08:20:40Z"
 content="""
It'd be really cool to have Rackspace cloud files support. Like the guy above me said, I would submit a patch but not if I have to learn Haskell first :)
"""]]


code:
[[!comment format=mdwn
 username="andy"
 ip="99.48.75.171"
 subject="Re: Webhook special remote"
 date="2013-04-12T08:54:47Z"
 content="""
@Alex: You might see if the newly-added [[todo/wishlist: allow configuration of downloader for addurl]] could be made to do what you need... I've not played around with it yet, but perhaps you could set the downloader to be something that can sort out the various URLs and send them to the correct downloading tool?
"""]]


code:
[[!comment format=mdwn
 username="http://yarikoptic.myopenid.com/"
 nickname="site-myopenid"
 subject="How to establish local preference for (special) remotes"
 date="2013-05-22T14:06:48Z"
 content="""
Sorry if it is RTFM... If I have multiple original (reachable) remotes, how could I establish my preference for which one to be used in any given location?

usecase: if I clone a repository within amazon cloud instance -- I would now prefer if this (or all -- user-wide configuration somehow?) repository 'get's load from URLs originating in the cloud of this zone (e.g. having us-east-1.s3.amazonaws.com/ in their URLs).
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8"
 nickname="develop"
 subject="Remote costs"
 date="2013-05-22T14:15:03Z"
 content="""
This should be implemented with costs

I refer you too: http://git-annex.branchable.com/design/assistant/blog/day_213__costs/

This has been implemented in the assistant, so if you use that, changing priority should be as simple as changing the order of the remotes on the web interface. Whichever remote is highest on the list, is the one your client will fetch from.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 14"
 date="2013-05-22T14:30:00Z"
 content="""
You do not need to use the assistant to configure the costs of remotes. Just set `remote.<name>.annex-cost` to appropriate values. See also the documentation for the `remote.<name>.annex-cost-command` which allows your own code to calculate costs.
"""]]


code:
[[!comment format=mdwn
 username="http://yarikoptic.myopenid.com/"
 nickname="site-myopenid"
 subject="remotes costs"
 date="2013-05-22T18:33:11Z"
 content="""
Thank you -- that is nice!

Could costs be presented in 'whereis' and 'status' commands? e.g. like we know APT repositories priorities from  apt-cache policy -- now I do not see them (at least in 4.20130501... updating to sid's 0521 now)

"""]]


code:
[[!comment format=mdwn
 username="http://yarikoptic.myopenid.com/"
 nickname="site-myopenid"
 subject="compression -- storage and transfer"
 date="2013-05-22T18:48:59Z"
 content="""
Is there any remote which would not only compress during transfer (I believe rsync does that, right?) but also store objects compressed?

I thought bup would do both -- but it seems that git annex receives data uncompressed from a bup remote, and bup remote requires ssh access.

In my case I want to make publicly available files which are binary blobs which could be compressed very well.  It would be a pity if I waste storage on my end and also incur significant traffic, which could be avoided if data load was transferred compressed.  May be HTTP compression (http://en.wikipedia.org/wiki/HTTP_compression) could somehow be used efficiently for this purpose (not sure if load then originally could already reside in a compressed form to avoid server time to re-compress it)?
"""]]


code:
[[!comment format=mdwn
 username="http://yarikoptic.myopenid.com/"
 nickname="site-myopenid"
 subject="Re: compression -- storage and transfer"
 date="2013-05-22T19:17:33Z"
 content="""
ha -- apparently it is trivial to configure apache to serve pre-compressed files (e.g. see http://stackoverflow.com/questions/75482/how-can-i-pre-compress-files-with-mod-deflate-in-apache-2-x) and they arrive compressed to client with

Content-Encoding: gzip

but unfortunately git-annex doesn't like those (fails to \"verify\") -- do you think it could be implemented for web \"special remotes\"? that would be really nice -- then I could store such load on another website, and addurl links to the compressed content
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 18"
 date="2013-05-23T23:25:02Z"
 content="""
All special remotes store files compressed when you enable encryption. Not otherwise, though.

As far as the web special remote and pre-compressed files, files are downloaded from the web using `wget` or (of wget is not available) `curl`. So if you can make it work with those commands, it should work.
"""]]


code:
[[!comment format=mdwn
 username="http://yarikoptic.myopenid.com/"
 nickname="site-myopenid"
 subject="compressed storage/transfer -- gzip Content-Type"
 date="2013-05-25T06:41:37Z"
 content="""
FWIW -- eh -- unfortunately it seems not that transparent.  wget seems to not support decompression at all, curl can do with explicit --compressed, BUT it doesn't distinguish url to a \"natively\" .gz file and pre-compressed content.  And I am not sure if it is possible to anyhow reliably distinguish the two urls.  In the case of obtaining pre-compressed file from my sample apache server the only difference in the http response header is that it gets \"compound\" ETag:
compare ETag: \"3acb0e-17b38-4dd5343744660\" (for directly asking zeros100.gz) vs \"3acb0e-17b38-4dd5343744660;4dd5344e1537e\" (requesting zeros100) where portion past \";\" I guess signals the caching tag for gzipping, but not exactly sure on that since it seems to be not a part of standard.  Also for zeros100 I am getting \"TCN: choice\"... once again not sure if that is any how reliably indicative for my purpose.  So I guess there is no good way ATM via Content-Type request.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawk9nck8WX8-ADF3Fdh5vFo4Qrw1I_bJcR8"
 nickname="Jon Ander"
 subject="MediaFire"
 date="2013-01-17T12:17:54Z"
 content="""
MediaFire offers 50GB of free storage (max size 200MB). It would be great to support it as a new special remote.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnWvnTWY6LrcPB4BzYEBn5mRTpNhg5EtEg"
 nickname="Bence"
 subject="Testing a special remote"
 date="2013-11-24T08:24:36Z"
 content="""
Is there a unit test or integration test to check for the behavior of a special remote implementation and/or validity?

I don't speak Haskell, so maybe there are some in the source but maybe I wouldn't recognize, so I haven't checked. If there are any tests how should I use it?

Thank you,
Bence
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 21"
 date="2013-11-24T15:58:30Z"
 content="""
@Bence the closest I have is some tests of particular special remotes inside Test.hs. The shell equivilant of that code is:

[[!format sh \"\"\"
set -e
git annex copy file --to remote # tests store
git annex drop file # tests checkpresent when remote has file
git annex move file --from remote # tests retrieve and remove
\"\"\"]]
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmp1ThsNNAbSn46ju-gwFELfStlhl8usJo"
 nickname="donkeyicydragon"
 subject="Enhancing special remotes as backup spaces"
 date="2014-04-22T21:08:49Z"
 content="""
Hi Joey,

I am thinking about using google drive as an encrypted backup for my important files. However, I fear that if all my git annex repositories are unrecoverable that the encrypted data on the special remote will not help me much. Assuming I have backed up my gpg key I still get a bunch of decrypted files but the folder structure is lost. Would it be possible to implement something like a safety feature that also uploads an (encrypted) tar of all symlinks (pointing to the respective encrypted files) of the (current/or master-branch) git working tree?

I am almost sure this is already implementable using hooks however I could not find information on which types of hooks are available. I am looking for one that is triggered once after all copy/move operations to a special remote are finished. Can you point me in the right direction?

Marek

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 23"
 date="2014-05-16T19:11:00Z"
 content="""
@donkeyicydragon one way to accomplish that would be to just tar up `.git` -- excluding `.git/annex/objects` and add that to git-annex like any other file. You could make a git post-commit hook that does that, but that seems overboard.

Or, you could just make a git clone of your repo to a local removable drive, and use that as a local backup.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.238"
 subject="comment 2"
 date="2013-01-17T16:44:25Z"
 content="""
Mediafire does not appear to offer any kind of API for its storage.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawk9nck8WX8-ADF3Fdh5vFo4Qrw1I_bJcR8"
 nickname="Jon Ander"
 subject="MediaFire REST API"
 date="2013-01-17T16:53:41Z"
 content="""
Wouldn't this be enough? http://developers.mediafire.com/index.php/REST_API
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmRFKwny4rArBaz-36xTcsJYqKIgdDaw5Q"
 nickname="Andrew"
 subject="JABOF special remote"
 date="2013-01-19T08:34:32Z"
 content="""
Similar to a JABOD, this would be Just A Bunch Of Files. I already have a NAS with a file structure conducive to serving media to my TV. However, it's not capable (currently) of running git-annex locally. It would be great to be able to tell annex the path to a file there as a remote much like a web remote from \"git annex addurl\". That way I can safely drop all the files I took with me on my trip, while annex still verifies and counts the file on the NAS as a location.

There are some interesting things to figure out for this to be efficient. For example, SHAs of the files. Maybe store that in a metadata file in the directory of the files? Or perhaps use the WORM backend by default?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.3.194"
 subject="comment 5"
 date="2013-01-19T16:05:13Z"
 content="""
The web special remote is recently able to use file:// URL's, so you can just point to files on some arbitrary storage if you want to.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlBia1J9-PoXgZYj2LASf7Bs__IqK3T8qQ"
 nickname="Greg"
 subject="Rackspace US/UK"
 date="2013-01-30T11:33:12Z"
 content="""
It'd be awesome to be able to use Rackspace as remote storage as an alternative to S3, I would submit a patch, but know 0 Haskell :D
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn-UoTjMBsVh6q4HNViGwJi-5FNaCVQB7E"
 nickname="Nico"
 subject="Rapidshare"
 date="2013-02-02T16:49:58Z"
 content="""
Would it be possible to support Rapidshare as a new special remote?
They offer unlimited storage for 6-10€ per month. It would be great for larger backups.
Their API can be found here: http://images.rapidshare.com/apidoc.txt
"""]]