code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="64.134.31.139"
 subject="comment 1"
 date="2013-10-16T15:19:25Z"
 content="""
Adding a filename would defeat deduplication. You suggest adding the \"first seen\" filename, but then git-annex would need to maintain a lookup between hashes and hashes with the first seen filename to use when adding new files.

There is already [[todo/wishlist: option to print more info with 'unused']].
"""]]


code:
[[!comment format=mdwn
 username="orb"
 ip="178.11.228.69"
 subject="comment 2"
 date="2013-10-17T21:12:09Z"
 content="""
thanks! (i hope a thanks is wanted, i know people who are just distracted and consider \"thanks\" posts unnecessary)
"""]]

repo_name: kakafarm/git

license: unknown


code:
I'm sitting on a pretty fast connection, and often git-annex isn't using my entire upstream to fill the remotes.

Lets say i have an annex with 10 x 100mb files. and my git-annex has 5 special remotes that needs to be filled.

I would like to have git-annex do 5 concurrent uploads(one to each remote). Currently git-annex only uploads to one special-remote at a time (meaning the 4 remaining servers are just waiting).

It would also be nice to have a per remote number of threads. Especially if adding a directory with 1000 small files in it(say 100bytes each), having 5 transfer threads to each remote would make the sync complete much faster.

For now i've made a shell script that i call:

    # for j in `seq -w 0 10`; do echo DOING $j; for i in `curl "http://127.0.0.1:$1/?auth=$2" | grep "continue" | gawk -F\"  ' { print $8 } '`; do curl "http://127.0.0.1:$1$i"; sleep 0; done; done

But it is very rough, and basically just starts all transfers on the page. Which means i currently have 315 active transfers running. whoops.

I could improve the shell script. But it really would be quite a bit niftier to have it as settings in git-annex.

Sincerely
Tobias


code:
I'm not sure of the right place for feature requests; sorry if this is incorrect.

It occurred to me that it would be neat if running git annex copy --auto copied enough copies of the file to connected remotes that don't have it to fulfil numcopies (if possible). Further coolness could come from choosing the remotes smartly (based, for example, on availability, access cost, remote disk space, trust level... you get the idea)

Having an option like this would allow the user to use numcopies to backup files to wherever possible, if they don't really care about where those copies go.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.183"
 subject="comment 1"
 date="2013-02-09T02:32:23Z"
 content="""
You can kind of do this now by:

for remote in $(git remote); do git annex copy --auto --to $remote; done

Although without some of the potential smarts of course.
"""]]


code: I would like it if the webapp could support setting up [bare remotes](http://git-annex.branchable.com/tips/centralized_git_repository_tutorial/) to assist with syncing between two machines that are never online at the same time.


code:
Is there any way to find all files that do not have a certain field assigned in metadata. E.g. I want to find all files that do not have an author field set and

    git-annex find --not --metadata "author=*"

doesn't give any results.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.172"
 subject="comment 1"
 date="2014-02-21T22:36:51Z"
 content="""
--metadata does not support globs, so your example is asking for all files that don't have an author field with a literal \"*\" value. When I try that command, it lists all files ... as expected.

It seems that adding glob support to it would get to the result you want, and makes sense to parallel git annex view. Change made in git!
"""]]


code:
I've made a first attempt at a .app launcher for git-annex webapp for OSX, please see <https://github.com/jcftang/git-annex/commits/master> the icon needs to be resized and made to look nicer at somepoint, the launcher assumes that git-annex is in the run time path.

I would imagine it might be possible to fiddle with the paths and get all the needed components into the .app file for OSX users.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.14.141"
 subject="comment 1"
 date="2012-09-23T15:14:20Z"
 content="""
Aha! I had been fumbling trying to find the right way to do this. Looks very promising. Found the documentation for the file format: <http://developer.apple.com/library/ios/#documentation/General/Reference/InfoPlistKeyReference/Articles/CoreFoundationKeys.html>

Where do I install this thing? If I put the git-annex.app directory inside ~/Desktop/, will that put it on the user's desktop?

Could you make a screenshot?

I'm guessing that the NSAppleScriptEnabled is not needed, and perhaps the LSMinimumSystemVersionByArchitecture block could also be removed.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.14.141"
 subject="comment 2"
 date="2012-09-23T15:17:42Z"
 content="""
Also, is there any global location I can install it when installing as root?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 3"
 date="2012-09-23T15:52:31Z"
 content="""
The .app file can be placed anywhere, typically you would want to bundle it in a .dmg file for the user and the user just \"drags\" it to the /Applications folder. It should be possible to bundle statically linked binaries inside the .app bundle, you might end up having to ship parts of coreutils and a few other applications to make it self contained.

Anyhow, here is a quick screencast <http://screencast.com/t/pnk8bk3hyUJM> and a screengrab of the app <http://screencast.com/t/4jjeRFx4g>
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 4"
 date="2012-09-23T16:00:35Z"
 content="""
Thinking this out a little, I reckon the script _MacOS/git-annex_ could be a bit more clever and get the basedir/basename of where it app is and execute git-annex from the .app bundle, then you wouldn't need to install it as root at all. It would require a bit of work to collect the executables needed.

Also, I just quickly checked my Lion install, it seems to come with git and uuidgen once xcode and the command line tools are installed.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.14.141"
 subject="comment 5"
 date="2012-09-23T16:29:19Z"
 content="""
For now I have it generating the shell script, so it contains the path to whereever git-annex's binary is actually installed.

For root, the app is installed in /Applications/ , for non-root installs I put it in ~/Desktop/
"""]]


code:
I was wondering how does one recover from...

<pre>
(Recording state in git...)
error: invalid object 100644 8f154c946adc039af5240cc650a0a95c840e6fa6 for '041/5a4/SHA256-s6148--7ddcf853e4b16e77ab8c3c855c46867e6ed61c7089c334edf98bbdd3fb3a89ba.log'
fatal: git-write-tree: error building trees
git-annex: failed to read sha from git write-tree
</pre>

The above was caught when i ran a "git annex fsck --fast" to check stash of files"


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 subject="comment 1"
 date="2012-07-24T22:00:35Z"
 content="""
This is a corrupt git repository. See [[tips/what_to_do_when_a_repository_is_corrupted]]
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 2"
 date="2012-07-25T06:52:32Z"
 content="""
Ah I was looking at the walkthrough on how to fix the issue, I had not thought about looking at the tips section. That tip fixed the issue for me, thanks.
"""]]


code:
The _views_ functionality has made the file structure dynamic.

With this in mind would it be possible/suitable to make drag and drop targets for `git-annex get` and `git-annex drop` commands?
This would make available git annex most interesting feature to assistant users.

I'm not entirely sure it's a good idea but it avoids file manager scripts and should be discoverable and easy to use.

I see two possibilities:

1. A folder in the annex root that contains a mirror of the tree but with only the unavailable files. Unavaliable files would then not be displayed in the live tree. Perhaps the folder can be called `remote`? Files can then be dragged from this tree to the live tree to trigger transfer of files. Any error/info gets displayed in the assistant
2. Flat droptargets in the annex root for `drop` and `get` commands.

Number one above makes discovering available files a bit cumbersome as you have to navigate down a parallell tree. It does make the available/remote distinction clearer which may be good for assistant users.

The mechanism could be extended to include a folder or droptarget for each remote.

The creation of droptargets/action folders could be triggered by a big visible button in the assistant interface **Manage Files**

Is it messy?


code: With numcopies it is possible to control how many copies there should be of a file overall in all of the remotes. Is it also possible, using .gitattributes or something else, to force one of the repos (e.g. a server) to contain a copy of all files, or does this have to be done manually?


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="99.54.57.201"
 subject="comment 1"
 date="2013-09-14T20:36:53Z"
 content="""
That is what the \"backup\" preferred content group is for.  Depending on how your repos connect to each other it might take some more trickery to make all content migrate to it, but that will make it \"want\" all the files if it can get them.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.14.105"
 subject="Ed's right"
 date="2013-09-19T18:56:39Z"
 content="""
What we don't have is something to enforce a particular preferred content setting. Ie, refuse to allow dropping a file from a repo if it's preferred content there. Currently you have to remember to use `git annex drop --auto` to cause the preferred content to be honored.
"""]]


code:
[[!meta title="GPG passphrase handling on OSX"]]

Hello!
I'm using OSX 10.9 and have installed gpg (and gpg2, if it matters) through
homebrew and git-annex through cabal. I also installed
https://github.com/joeyh/git-remote-gcrypt like the UI told me.

Whenever I'm trying to add an encrypted remote through the web UI I get a
lot of "You need a passphrase to unlock the secret key for user:" on stdout
and, obviously, I can't enter my passphrase (If I could I wouldn't make this
post to begin with :))
Is this behavior normal? What should I do to work around it?
I did also try to not use the web UI by using this command:
git annex initremote rsync.net type=gcrypt gitrepo=user@host:directory encryption=pubkey keyid=X

Because of this I can't copy files to my remotes. All I get is:
-----
$ git annex copy --to rsync.net
copy MySecretFile (gpg)
You need a passphrase to unlock the secret key for
user: "user"
4096-bit RSA key, ID X, created 2013-10-01 (main key ID Y)

(checking rsync.net...) (to rsync.net...) gpg: no valid addressees
gpg: [stdin]: encryption failed: No user ID
failed
-----

Yes, I am using gpg-agent. When other applications ask for my passphrase I get
the pinentry dialog from GPGTools, just like I've configured it in
~/.gnupg/gpg-agent.conf, but this isn't the case with git-annex.

If I remove GPGTools from /usr/local/bin with: ``brew link --overwrite gnupg &&
brew link --overwrite gnupg2'' it works *slightly* better.
I get that pinentry dialog I want but when I do a copy I get:
-----
$ git annex copy --to rsync.net
copy MySecretFile (gpg) (checking rsync.net...) (to rsync.net...) gpg: no valid addressees
gpg: [stdin]: encryption failed: no such user id
failed
-----

--debug shows me it is executing gpg llke so:
-----
gpg ["--batch","--no-tty","--use-agent","--quiet","--trust-model","always","--batch","--encrypt","--no-encrypt-to","--no-default-recipient","--force-mdc","--no-textmode"]
-----

$ git annex version
git-annex version: 4.20131024
build flags: Assistant Webapp Pairing Testsuite S3 WebDAV FsEvents XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav glacier hook

$ gpg --version
gpg (GnuPG) 2.0.22
libgcrypt 1.5.3
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: ~/.gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA, ?, ?
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

 $ gpg-agent --version
gpg-agent (GnuPG/MacGPG2) 2.0.22
libgcrypt 1.5.3
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkzwmw_zyMpZC9_J7ey--woeYPoZkAOgGw"
 nickname="dxtrish"
 subject="comment 1"
 date="2013-10-31T00:06:22Z"
 content="""
In fact I just tried this with a key without a passphrase and I STILL get:

[2013-10-31 01:00:50 CET] Pusher: Syncing with rsync.net
gpg: no valid addressees
gpg: [stdin]: encryption failed: no such user id

In addition I also noticed a lot of:
git-annex: fd:28: hPutBuf: resource vanished (Broken pipe)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkzwmw_zyMpZC9_J7ey--woeYPoZkAOgGw"
 nickname="dxtrish"
 subject="comment 2"
 date="2013-10-31T02:42:47Z"
 content="""
I think I have found two ways I can make this happen.

1) If a filename contains UTF-8 it seems to happen (At least with this specific filename I have right here)

2) Finder.app does not seem to play nice with git-annex (I got the same error when I tried using Finder to copy files to my git-annex thing)

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 3"
 date="2013-11-02T20:39:36Z"
 content="""
There seem to be a lot of different ways to install gpg on OSX, with different results in what works and not. I don't know what's going on with that. Advice from those who understand Mac OS would be appreciated.

As far as the \"gpg: no valid addressees\" bug goes, I can reproduce it on Linux, and have fixed it. Only affected using gcrypt with encryption=pubkey.

(Please file bug reports in [[bugs]] in the future, not in the forum!)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkzwmw_zyMpZC9_J7ey--woeYPoZkAOgGw"
 nickname="dxtrish"
 subject="comment 4"
 date="2013-11-03T13:10:11Z"
 content="""
I would have filed it in the bug section if I knew it was a bug. I just thought I had done something wrong :)
"""]]


code: Is it possible to generate a view of files currently available on the annex? My use case is that I have pretty large repo (couple of TBs) and I have partial checkouts on multiple machines instead of seeing 100s of broken symlinks I would like to just filter filter files that are present on the machine?


code:
I'm just getting started with git-annex and trying to wrap my head around using it with Amazon S3. I am familiar with using git, but things are a bit different as we can't init a repo at S3 directly.

I've followed http://git-annex.branchable.com/tips/using_Amazon_S3/, and performed:

`git init`

Initialized empty Git repository in /home/

`git annex init`

init  ok

`git annex initremote s3 type=S3 encryption=FOOBAR bucket=foo`

initremote s3 (encryption setup with gpg key YGTVT51715TFR) (checking bucket...) (gpg) ok

`git annex describe s3 "Amazon S3"`

describe s3 ok

`git annexx add foo/`

add foo/bar.txt

add foo/bar.png

...etc

`git annex sync`

51 files changed, 51 insertions(+)

create mode 120000 foo/bar.txt

create mode 120000 foo/bar.png

...etc

Looking at http://git-annex.branchable.com/git-annex/, I thought the files added would then be pushed to S3 by git annex sync, but that doesn't seem to be the case. I've also tried variations of got annex copy, like `git annex copy . --to s3`, without any luck.

Is there a way to push to s3?

Any help is appreciated!


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.81.112"
 subject="comment 1"
 date="2012-05-29T19:09:50Z"
 content="""
`git annex sync` only syncs git metadata, not file contents, and metadata is not stored on S3, so it does notthing (much).

`git annex move . --to s3` or `git annex copy . --to s3` is the right way to send the files to S3. I'm not sure why you say it's not working. I'd try it but Amazon is not letting me sign up for S3 again right now. Can you show what goes wrong with copy?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnoUOqs_lbuWyZBqyU6unHgUduJwDDgiKY"
 nickname="Matt"
 subject="comment 2"
 date="2012-05-30T00:40:45Z"
 content="""
It's strange. I've done some testing on another machine, and this one, and the issue seems to be with adding only certain sub-directories of the git-annex directory. Would it cause an issue with git-annex if a sub-directory was a git repo?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.81.112"
 subject="comment 3"
 date="2012-05-30T00:54:38Z"
 content="""
If the subdirectory has a .git, then it's a separate git repo, and inside the directory, all git (and git-annex) commands in it will operate on that nested repo and ignore the outside one.
"""]]


code:
Hello,

I am trying to wrap my head around annex still.  I'm creating a source local git repo, editing an annex file, and then syncing in a second repo.  In that second repo I'm trying to figure out how I can get a status notifying me that a file is out of date.

If I use 'annex sync --content' the files are all up to date, as expected, but what I kind of expected is 'annex status' to say something like 'files out of date, blah blah'.  I am spelling out my example below.

Annex version is 5.20140613

## I create a source and target repo, and I add a git managed file and an annex managed file

    $ mkdir source target
    $ cd source
    $ git init
    $ git annex init
    $ cat > gitfile
    hi this is my git file
    $ cat > annexfile
    hi this is my annex file
    $ git add gitfile
    $ git annex add annexfile
    $ git commit -m 'init commit'
    $ cd ../target
    $ git clone ../source/ .
    $ ls -al
    total 16
    drwxr-xr-x  170 Jul 15 15:55 .
    drwxr-xr-x  136 Jul 15 15:54 ..
    drwxr-xr-x  442 Jul 15 15:55 .git
    lrwxr-xr-x  180 Jul 15 15:55 annexfile -> .git/annex/objects/Qp/F0/SHA256E-s25--045cf30cb201c6723cb6fad9ca539f639de7f242b87775b876ef9ccb1f577ccf/SHA256E-s25--045cf30cb201c6723cb6fad9ca539f639de7f242b87775b876ef9ccb1f577ccf
    -rw-r--r--  23 Jul 15 15:55 gitfile
    $ git annex sync
    ...
    To [base path]/target/../source/
     * [new branch]      git-annex -> synced/git-annex
     * [new branch]      master -> synced/master
    ok
    $ cat gitfile
    hi this is my git file
    $ cat annexfile
    cat: annexfile: No such file or directory
    $ git annex sync --content
    $ cat annexfile
    hi this is my annex file

## So far so good.  Now I'm going to edit my annexed file in source repo and see if I can find out that the file was edited in the target

    $ cd ..
    $ cd source
    $ git annex edit annexfile
    $ cat > annexfile
    wow I changed my annex file
    $ git annex add annexfile
    $ git commit -m 'changed an annex file'
    $ cd ..
    $ cd target
    $ git pull
    $ cat annexfile
    cat: annexfile: No such file or directory
    $ git annex sync
    $ ls
    annexfile gitfile
    $ cat annexfile
    cat: annexfile: No such file or directory
    $ git annex status
    [no output]

## Here is where I'd expect something saying 'annexfile is out of date', etc.
## I can infer it because the link is missing, but I'm sure there's a more logical way.

    $ git annex sync --content
    $ cat annexfile
    wow I changed my annex file

## After I synced content all is well.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 1"
 date="2014-07-16T18:18:00Z"
 content="""
As soon as git-annex syncs with a remote that has made changes to the repository, any new files or files that have been modified, and the content is not yet available in the local repository will be broken symlinks. You can generally easily see a broken symlink by running `ls`, at least with a good `ls` on eg Linux that supports colorization. (Look for the red filenames.) Many file managers also represent broken symlinks in a visual way.

Or, you can run a command like `git annex whereis $file` and see if it includes \"here\" in the list of locations.

Or, you can run `git annex find --not --in here` to find all files whose current content is not present.

The reason `git annex status` doesn't say is that it's focused on showing you which files in the local repository have not yet been committed to git. The `git annex list` command has a similar one line per file output, but puts a \"_\" in the first column (under \"here\") if the file is not locally present.
"""]]


code:
I'd like to use git-annex with my Asustor 302t NAS. It should be treated like a "simple" removable drive. I don't really know how to do this however, when scanning for "removable drives", it can't be found, when I try connecting via SSH, I receive an Error: Permission denied (publickey,password). As there is no way to insert my password before connecting.

How do I setup a repository on my NAS to be synced with my laptop?
I am using MAC OS X and the git-annex web interface currently.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 1"
 date="2014-05-16T16:32:05Z"
 content="""
If your NAS is accessed over SSH, it is not a removably drive, clearly.

Try starting git-annex webapp from a console, you will probably see the ssh password prompt there. Or, install ssh-askpass, or, install a current daily build of git-annex, which will prompt for the ssh password from within the webapp.
"""]]


code:
Hi,

I accidentally removed the .git/ folder inside a git annex assistant-monitored folder. When I open the assistant web app it still shows the repository but when I try to select it the page request never returns. What can I do at this point? I was thinking I can remove the repository from the git annex assitant configuration file if it is a plain-text file, but I don't know of its location. Please advise.

Regards,
Blake


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 1"
 date="2013-12-17T16:07:00Z"
 content="""
The file you need is .config/git-annex/autostart in your home directory.

Once deleted, re-running the assistant should let you re-create the git repository and it'll re-add your files.
"""]]


code:
Is it possible to use Gitblit as a centralized repository using git-annex?
thanks Gary


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 1"
 date="2014-05-16T17:40:20Z"
 content="""
I assume gitblit is something like gitosis or gitolite. These let a user ssh in to a server to access a git repository, but they limit the commands they can run to `git-shell`. In order to use git-annex get/copy to such a repository, it needs to be able to run `git-annex-shell`, which is like git-shell but with more commands available. If gitblit can be made allow git-annex-shell to be run, it will work.
"""]]


code:
Hi,

I am trying to use Git Annex Assistant on my Fedora Linux computer. I currently have a local repository that assistant monitors, but now I want to use my personal server as a git remote for the repository. I click on Configuration > Remote server and it prompts me for my server login credentials and then click the "Check this server" button. After this I am shown a screen that assistant is "Ready to add remote server" and then states:
> The server <MY IP ADDRESS HERE> can be used as is, but installing git-annex on it would make it work better, and provide more options below.
>
> If you're able to install software on the server, do so and click Retry

The information shown below this message is regarding encrypting the data (if I have git-annex installed on the server, which I don't). So my question is, what do I do now? There is no button to just add the remote from within git annex assistant? I created a bare repository on my server and added a remote manually inside the repository on my local machine, but git annex assistant doesn't notice it. I'm clueless on what git annex assistant is expecting from me at this point. I would think what I'm trying to do is one of the most common use cases for git annex, but I can't find any documentation or materials on the correct procedure for this. Any help is appreciated.

Regards,
Blake


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="64.134.31.139"
 subject="comment 1"
 date="2013-10-21T22:20:53Z"
 content="""
You have two choices:

1. Install git-annex on your server. You can then use the webapp to set up a git repository, or use the git repository you've already set up.

2. Don't install git-annex on your server. Select the option in the webapp to use an encrypred rsync repository on your server. (The screen you described the webapp showing had a button to use this option, but you seem to have misread it.)
"""]]


code:
I'm using Git Annex Assistant for the first time, trying to get it to mirror a directory of files from my computer to an external USB drive. I've tried setting up the second repo as a "Removable Drive" in the "full backup" repository group. I've also tried putting it in the "client" group. The result is the same, and Assistant doesn't backup any of my files to the external drive.

What I get in the external drive in the directory that should contain the repo are the set of files "annex, objects, refs, config, HEAD" and they are NOT in a .git subdirectory. Aside from this I don't see any of the files Git Annex Assistant says it has synced. As this is my first time using the Assistant, I don't know if this is a bug, or if I'm just using it incorrectly. If anyone knows what is going on here, please share.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawk3HGoDpnOPob5jOjvIootmkve1-nCpRiI"
 nickname="Kalle"
 subject="This is the expected default behavior"
 date="2014-05-17T20:49:06Z"
 content="""
Your files should be on the removable drive but the file structure and file names are following git annex logic and is not meant to be human readable.

The default assumption (which I question) is that you will use the removable drive to transfer files between git-annex repositories. So by default you cannot read the files on devices that do not have git-annex installed.

You can configure git-annex to do what you want using the `post-receive` hook. It's a bit complicated and might leave git-annex hogging your drive when you wan't to eject it but see the following:

 - [[forum/USB_backup_with_files_visible/]]
 - [[forum/Can't_get_git-annex_merge_to_work_from_git_hook/#comment-d93181e69b8bea54bc03cb335dd1d7c8]]

The last post on that second link tells you exactly what to put in the `post-receive` hook in the .git/hooks folder on your removable drive.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnov5q9_Cl4Ps5NoYE08yE01NLSvBANnY8"
 nickname="Eric"
 subject="comment 2"
 date="2014-05-18T09:35:45Z"
 content="""
Okay, I understand now. Probably Assistant should only treat the external drive that way if it's in the \"transfer\" group.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnov5q9_Cl4Ps5NoYE08yE01NLSvBANnY8"
 nickname="Eric"
 subject="comment 3"
 date="2014-07-10T05:50:32Z"
 content="""
Actually, I understand the logic behind encouraging bare repositories.
"""]]


code:
So, I'm using git-annex to manage a group of external hard drives that I use to store a bunch of big files.

When syncing tonight, though, it decided to take a whole subdirectory and replace the symlinks with the content directly when merging, or something.

The remote side is not like that, and I didn't tell it to do anything like that.
It just seems to feel like that's the thing to do, then it syncs and fails because suddenly git has many gigs of objects, which is what git-annex is supposed to solve.

I've reset master a bunch of times, and it keeps choosing that path, but I don't know why.
I've also reset master and synced/master a few times. I'm not fully sure what that one actually encodes, so I didn't want to touch it too much, but setting it to the same thing as master didn't seem to fix things either.

Any idea why?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 1"
 date="2013-12-31T18:06:15Z"
 content="""
Are you using direct mode?

Before the direct mode guard was put in place, it was not uncommon for users to mistakenly run some command like \"git commit -a\" in a direct mode repository. This happily checks the full large files into git.

It sounds to me like you've done that. But, you have not provided enough information to make anything more than a vague guess.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawk_W9oOMDiBaU_xLIbeaTF8wW2wJzJ7bfw"
 nickname="Christopher"
 subject="comment 2"
 date="2013-12-31T18:56:58Z"
 content="""
I am not in direct mode.
I don't believe I did anything in the repo. I reset back to a commit and look in the folder and it's all symlinks and nothing in git status says anything about it.

Then I run `git annex sync` and one of the thing it does is \"Checking out files\" which takes a long time, since it seems to be copying the data into the working directory and commiting it.
I don't know why it's decided to do that.

What steps can I take to either get you more information or fix things?

Perhaps relatedly, if I make a mistake with git-annex, is git-reset master and git-reset synced/master the right approach?
Are there other things I should try, etc?

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 3"
 date="2013-12-31T19:19:15Z"
 content="""
\"Checking out files\" is a message printed by git (not by git-annex) when it is updating the work tree.

It still seems that you have somehow committed large files directly to git. Perhaps you accidentially ran \"git add\" on a large file.

git annex sync is probably merging this commit from one of your other repositories.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawk_W9oOMDiBaU_xLIbeaTF8wW2wJzJ7bfw"
 nickname="Christopher"
 subject="comment 4"
 date="2014-01-01T15:23:17Z"
 content="""
I know that checking of files is a git message, I was just providing it for context.

I know that my other repos do not have these things checked in, because one of them is offline and not involved in this, and the other repo literally can't contain that much data, which is the biggest reason this is an issue.

Here are the commits I don't like:

    commit 1ae2410a4bc3e35edbfd9d48a577bcfc7e3a8836
    Author: Christopher Vollick <0@psycoti.ca>
    Date:   Mon Dec 30 23:22:47 2013 -0500

        git-annex automatic sync

    :120000 100644 185b311... fd03ae5... T	subdir1/other_file0309
    :120000 100644 515e6df... 44f8383... T	subdir1/other_file0309

    commit 7403e3116eb34e2b192dfa3ea4c16e35f82210ba
    Author: Christopher Vollick <0@psycoti.ca>
    Date:   Sat Dec 21 19:12:56 2013 -0500

        git-annex automatic sync

    :120000 100644 185b311... fd03ae5... T	subdir1/other_file0309
    :120000 100644 515e6df... 357d9a7... T	subdir1/other_file0309
    :120000 100644 079f69f... 97c785a... T	subdir1/file01
    :120000 100644 c69574a... 1589721... T	subdir1/file02
    :120000 100644 01d98a4... 154a059... T	subdir1/file03
    :120000 100644 6ee02d5... 6857f73... T	subdir1/file04
    :120000 100644 aab78e7... 19bae04... T	subdir1/file05
    :120000 100644 9f8d11b... bd5b501... T	subdir1/file06
    :120000 100644 ca824c1... a9e0b9e... T	subdir1/file07
    :120000 100644 c85f526... a6383fc... T	subdir1/file08
    :120000 100644 a6d957d... 9b99c99... T	subdir1/file09
    :120000 100644 d3de258... d51da9a... T	subdir1/file10
    :120000 100644 3cc803e... 8f7945f... T	subdir1/other_file0005
    :120000 100644 cd94cca... 1c06cf8... T	subdir1/other_file0006
    :120000 100644 1442c98... fee3c1d... T	subdir1/other_file0201
    :120000 100644 4131025... f732332... T	subdir1/other_file0202
    :120000 100644 865214f... 1178931... T	subdir1/other_file0203
    :120000 100644 a0f84e2... 6c39a4a... T	subdir1/other_file0204
    :120000 100644 21bf0c1... bbb3c93... T	subdir1/other_file0205
    :120000 100644 5f6a5f3... d298a97... T	subdir1/other_file0206
    :120000 100644 3d7f034... 4cac0d1... T	subdir1/other_file0207
    :120000 100644 ee75d44... d769e8e... T	subdir1/other_file0208
    :120000 100644 f381401... 5f1a7f3... T	subdir1/other_file0209
    :120000 100644 8b6a409... 1db8c37... T	subdir1/other_file0210
    :120000 100644 c6a23f1... 5a9bbf5... T	subdir1/other_file0211
    :120000 100644 17a34c7... 068c619... T	subdir1/other_file0212
    :120000 100644 3c7fe0a... 356315b... T	subdir1/other_file0213
    :120000 100644 4b61d45... e9747e5... T	subdir1/other_file0214
    :120000 100644 1c0d845... f7a11d3... T	subdir1/other_file0301
    :120000 100644 cdd57b2... fb5512f... T	subdir1/other_file0302
    :120000 100644 9bb8ffc... 9b3897d... T	subdir1/other_file0303
    :120000 100644 007cfa6... 93e36d7... T	subdir1/other_file0304
    :120000 100644 24cf5d3... 9eecff4... T	subdir1/other_file0305
    :120000 100644 d9be965... 6c612f4... T	subdir1/other_file0306
    :120000 100644 20cd474... 9d1c714... T	subdir1/other_file0307
    :120000 100644 3f24532... 9ff7770... T	subdir1/other_file0308
    :120000 100644 3939cbe... de9fefd... T	subdir1/other_file0308
    :120000 100644 9de87f9... 2242892... T	subdir1/other_file0309
    :120000 100644 7250b17... a544af6... T	subdir1/other_file0310
    :120000 100644 9de87f9... 2242892... T	subdir1/other_file0312
    :120000 100644 14fbe43... f643564... T	subdir1/other_file0401
    :120000 100644 0f28f22... e4b5e2c... T	subdir1/other_file0402
    :120000 100644 6704647... 4529fb0... T	subdir1/other_file0403
    :120000 100644 65b3742... dacc2ce... T	subdir1/other_file0404
    :120000 100644 14ad1d6... a4f7dab... T	subdir1/other_file0405
    :120000 100644 87bda81... a3b2a0a... T	subdir1/other_file0406
    :120000 100644 23fd37b... 7ea3182... T	subdir1/other_file0407
    :120000 100644 a82d025... aa0e1f6... T	subdir1/other_file0408
    :120000 100644 cd350d3... c225b3d... T	subdir1/other_file0409
    :120000 100644 2ce6c3b... 1efffa6... T	subdir1/other_file0410
    :120000 100644 3f780d5... 207f8af... T	subdir1/other_file0411
    :120000 100644 27ecf10... 4a34b2d... T	subdir1/other_file0412
    :120000 100644 5cf1ba0... ab0b15d... T	subdir1/other_file0413
    :120000 100644 a1cccd7... 274dbd1... T	subdir1/other_file0004

Now, the most interesting part here is that commit 7403e3116eb is dated Sat Dec 21.
That is not when I was attempting this. The 30th, shown in 1ae24 is.

That makes me think that I have failed to reset properly, and so when I'm telling it to sync it's not recomputing the thing, it's just grabbing the thing it did last time.

Now, I still don't know why it chose to do that, but perhaps it wouldn't choose to do that again if I could figure out how to get it to forget that path.

Thoughts?

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawk_W9oOMDiBaU_xLIbeaTF8wW2wJzJ7bfw"
 nickname="Christopher"
 subject="comment 5"
 date="2014-01-01T16:37:38Z"
 content="""
Ok, so, looking into it a little more I think I've found that I screwed up updating a ref, so I didn't reset synced/master like I thought I did.

So, I don't know why it chose to make this choice in the first place, but I was able to reset and manually change my way around it.

That being said, I don't want to do this a lot, because I'm not fully clear on what the synced branches represent.
Is it that, on repo1, repo2/synced/master is the last thing that pushed over, and that after a sync local synced/master is always master, and before that it's the last sync to anywhere?

How safe is it to screw around with these things?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 6"
 date="2014-01-01T20:59:32Z"
 content="""
The ` T ` in your `git show --raw` indicates that what was a symlink has been replaced with a regular file, and this has been committed to git.

It's certainly possible to do that when using git-annex, but you have to go a bit out of your way to so shoot yourself in the foot, because normally the pre-commit hook will detect that, and fix up your commit to not change the type of the file. Something like this:

[[!format sh \"\"\"
joey@darkstar:~/tmp/r>git annex unlock foo
unlock foo (copying...) ok
joey@darkstar:~/tmp/r>git commit -a --no-verify -m oops
[master 9b63e4e] oops
 1 file changed, 1 deletion(-)
 rewrite foo (100%)
 mode change 120000 => 100644
joey@darkstar:~/tmp/r>git show --raw
commit 9b63e4efad1f229ae7713749e9ac2d0f9c286008
Author: Joey Hess <joey@kitenet.net>
Date:   Wed Jan 1 16:55:19 2014 -0400

    oops

:120000 100644 ea46194... e69de29... T	foo
\"\"\"]]

The fact that you seem to have made 2 commits that did this, on the 21st and 30th, makes me wonder if your .git/hooks/pre-commit does not exist, or perhaps you are making some other mistake repeatedly.

The synced/* branches never have any data that is not stored somewhere else (another branch, possibly in the remote repository), so it should always be ok to delete them.
"""]]


code:
Hi,

May I know, which processes git-annex is using to move/copy/get file content between repositories?
Is it using same processes which git uses, Like send-pack, receive-pack.
I want to use Aspera to move file contents between repositories.
Is it enough if I customize send-pack, receive-pack of git code to do Aspera file transfer or git-annex uses any other transfer mechanism.

Many Thanks,
Royal Pinto


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.2.186"
 subject="rsync over ssh"
 date="2012-05-10T18:18:01Z"
 content="""
Some other protocols such as S3 for special remotes.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlYu7QmD7wrbHWkoxuriaA9XcijM-g5vrQ"
 nickname="Royal"
 subject="Protocols to transfer file content"
 date="2012-05-10T18:48:40Z"
 content="""
Thanks, Is git annex is using same protocols as normal git to transfer content between normal git repositories?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.2.186"
 subject="comment 3"
 date="2012-05-10T18:51:56Z"
 content="""
git-annex doesn't transfer git content between git repositories. You use git for that. Well, git-annex sync can run a few git commands for you to do it.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlYu7QmD7wrbHWkoxuriaA9XcijM-g5vrQ"
 nickname="Royal"
 subject="Git annex content transfer protocols"
 date="2012-05-10T19:13:48Z"
 content="""
Sorry if I am not clear. Actually i meant to ask, if i have 2 git repositories which are not special remotes and I am transferring annexed file content between these repositories using git annex command (move or copy) then, which protocol it uses to transfer content? Is it uses git-send-pack git-recieve-pack or some other protocols.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.2.186"
 subject="comment 5"
 date="2012-05-10T19:17:22Z"
 content="""
rsync over ssh is used to transfer file contents between repositories. (You can use the -d option to see the commands git-annex runs.)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlYu7QmD7wrbHWkoxuriaA9XcijM-g5vrQ"
 nickname="Royal"
 subject="comment 6"
 date="2012-05-10T19:21:08Z"
 content="""
Ok. This helped me a lot. Thank you
"""]]


code:
Hi,

Since the I updated to `2013-10-24` `git annex` has corrupted itself twice on my setup.  I'm not claiming causality here and thus I haven't filled it as a bug (probably the mistake is mine).

I use three laptops and one remote ssh server.  The motherboard on laptop *T* seems to have broken and I haven't fixed it yet.  I have disabled sync to *T*  on the webapp dashboard.

First, one computer, called *W*, broke on Friday.  I got the dreaded [serious problem](http://git-annex.branchable.com/devblog/day_41__onward/) screen. `git fsck` reported lots of missing blobs and tree problem, though my data seemed OK.  I git cloned the repo from computer *X*.  Now, last night both X and W reported the same problem (log below).  *W*  recovered itself somehow, but *X*  is now in a cycle of re-adding all files in the repo and the posting the below error that has to do with computer *T* (IP ending with 107).

I would appreciate hints on why this suddenly started happen.  Computer T has been out of sync for almost 14 days (since the 24th of October I think).

    error: refs/remotes/192.168.1.107_annex/git-annex does not point to a valid object!
    error: refs/remotes/192.168.1.107_annex/master does not point to a valid object!
    error: refs/remotes/192.168.1.107_annex/synced/git-annex does not point to a valid object!
    error: refs/remotes/192.168.1.107_annex/synced/master does not point to a valid object!
    error: refs/synced/2ed58ecf-8e8c-44b8-ad34-d42ddfb35315/cmFzbXVzQGNvZGVyb2xsZXJzLmNvbQ==/git-annex does not point to a valid object!
    error: refs/synced/2ed58ecf-8e8c-44b8-ad34-d42ddfb35315/cmFzbXVzQGNvZGVyb2xsZXJzLmNvbQ==/master does not point to a valid object!
    error: refs/synced/f54b8200-8ec1-43d2-8710-b73ec118addc/cmFzbXVzQGNvZGVyb2xsZXJzLmNvbQ==/git-annex does not point to a valid object!
    error: refs/synced/f54b8200-8ec1-43d2-8710-b73ec118addc/cmFzbXVzQGNvZGVyb2xsZXJzLmNvbQ==/master does not point to a valid object!
    error: refs/synced/f54b8200-8ec1-43d2-8710-b73ec118addc/git-annex does not point to a valid object!
    error: refs/synced/f54b8200-8ec1-43d2-8710-b73ec118addc/master does not point to a valid object!
    error: refs/remotes/192.168.1.107_annex/git-annex does not point to a valid object!
    error: refs/remotes/192.168.1.107_annex/master does not point to a valid object!
    error: refs/remotes/192.168.1.107_annex/synced/git-annex does not point to a valid object!
    error: refs/remotes/192.168.1.107_annex/synced/master does not point to a valid object!
    error: refs/synced/2ed58ecf-8e8c-44b8-ad34-d42ddfb35315/cmFzbXVzQGNvZGVyb2xsZXJzLmNvbQ==/git-annex does not point to a valid object!
    error: refs/synced/2ed58ecf-8e8c-44b8-ad34-d42ddfb35315/cmFzbXVzQGNvZGVyb2xsZXJzLmNvbQ==/master does not point to a valid object!
    error: refs/synced/f54b8200-8ec1-43d2-8710-b73ec118addc/cmFzbXVzQGNvZGVyb2xsZXJzLmNvbQ==/git-annex does not point to a valid object!
    error: refs/synced/f54b8200-8ec1-43d2-8710-b73ec118addc/cmFzbXVzQGNvZGVyb2xsZXJzLmNvbQ==/master does not point to a valid object!
    error: refs/synced/f54b8200-8ec1-43d2-8710-b73ec118addc/git-annex does not point to a valid object!
    error: refs/synced/f54b8200-8ec1-43d2-8710-b73ec118addc/master does not point to a valid object!
    error: invalid object 100644 1ca66de3cdd9c79cde26a7555cf3b8d26d0e371d for '000/147/SHA256E-s347--1ab8084bf9ae06407ce0a7260a83638ea6e9a028dc59b4815fd60aec61dbd747.txt.log.log'
    fatal: git-write-tree: error building trees
    TransferScanner crashed: failed to read sha from git write-tree
    [2013-11-04 09:45:41 CET] TransferScanner: warning TransferScanner crashed: failed to read sha from git write-tree


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlEXAlu8qKv_FHYOv-uEohdGgMY_Lw-Ges"
 nickname="Jonas"
 subject="Just happened for me as well."
 date="2013-11-12T08:10:56Z"
 content="""
This was the first search result when I googled the error message, I'm not sure what has happened (or if it is a problem since I don't change my files that often).

git-annex version: 4.20131101

Also running git annex with one ssh remote ( laptop1 & laptop2 synching through a ssh remote server ).

Got a bunch of
    error: refs/synced/dbb70cea-4b83-416f-93af-e59082c4d633/git-annex does not point to a valid object!
    error: refs/synced/dbb70cea-4b83-416f-93af-e59082c4d633/master does not point to a valid object!
    error: refs/synced/dbb70cea-4b83-416f-93af-e59082c4d633/git-annex does not point to a valid object!
    error: refs/synced/dbb70cea-4b83-416f-93af-e59082c4d633/master does not point to a valid object!
when I tried a manual git annex sync, and noticed that I had a bunch of those in the log in the webapp as well...

The gui doesn't show that anything is wrong, and looking at the log it seems that things are atleast uploading

    [2013-11-12 09:05:46 CET] Committer: Committing changes to git
    [2013-11-12 09:05:46 CET] Pusher: Syncing with born
    error: refs/synced/dbb70cea-4b83-416f-93af-e59082c4d633/git-annex does not point to a valid object!
    error: refs/synced/dbb70cea-4b83-416f-93af-e59082c4d633/master does not point to a valid object!
    Everything up-to-date

    FILE

       32768  42%    0.00kB/s    0:00:00
       76972 100%   42.16MB/s    0:00:00 (xfer#1, to-check=0/1)

    sent 77056 bytes  received 31 bytes  51391.33 bytes/sec
    total size is 76972  speedup is 1.00
    [2013-11-12 09:05:48 CET] Transferrer: Uploaded FILE
    [2013-11-12 09:05:48 CET] Pusher: Syncing with born
    error: refs/synced/dbb70cea-4b83-416f-93af-e59082c4d633/git-annex does not point to a valid object!
    error: refs/synced/dbb70cea-4b83-416f-93af-e59082c4d633/master does not point to a valid object!
    error: refs/synced/dbb70cea-4b83-416f-93af-e59082c4d633/git-annex does not point to a valid object!
    error: refs/synced/dbb70cea-4b83-416f-93af-e59082c4d633/master does not point to a valid object!
    To ssh://me@born/annex
      54e04cf..333cf10  git-annex -> synced/git-annex
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlEXAlu8qKv_FHYOv-uEohdGgMY_Lw-Ges"
 nickname="Jonas"
 subject="comment 3"
 date="2013-11-13T09:03:24Z"
 content="""
I did a change of one of the files and that totally broke all other repositories, git fsck reports a lot of missing trees (I'm not sure if it is normal), and git annex fsck started reporting invalid object for another file for some reason, also got a (git annex repair still reports everything is fine though.)

    fatal: git-write-tree: error building trees
    git-annex: failed to read sha from git write-tree

I tried following those instructions but I don't think I had a good repository, git annex repair reported everything was ok but git fsck said a lot of trees where missing and it stopped updating.
I noticed I was running different git version (git version 1.7.9.5 on the server and git version 1.8.4.2 on one of the clients), so maybe that was part of the reason.

I couldn't find out what remote dbb70cea-4b83-416f-93af-e59082c4d633 was supposed to be either, it wasn't any of the active ones and it didn't show up with git annex status either (I've lost a few repositories when I reinstalled my computer).

I ended up recreating the repository (by copying the raw files) from what I think is the most recent version from one of my computers and hope that it was a error that happened due to upgrades.
"""]]


code:
[[!comment format=mdwn
 username="http://id.clacke.se/"
 nickname="clacke"
 subject="comment 3"
 date="2013-11-12T17:19:32Z"
 content="""
I have been getting this too. Here is what I have done to fix it:

[[tips/recovering_from_a_corrupt_git_repository]]
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.43"
 subject="comment 4"
 date="2014-01-13T20:27:34Z"
 content="""
Many problems of this sort can happen if the disk that the git reposotory is on is losing data that's written to it, or is just removed (or the computer crashes) at the wrong time. `git-annex repair` can be run in the repository to recover from many problems like this, and it's getting better all the time.
"""]]


code: How can I clone a git annex remote (bare repository) created with the webapp in command line ? My goal is to be able to access the files using an ssh connection.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.254.222"
 subject="comment 1"
 date="2013-07-07T17:16:27Z"
 content="""
There is no difference between a repository created using the assistant and one created by hand. So you can just `git clone ssh://server/path` or `git remote add foo  ssh://server/path` and use it at the command line like any other repository.

The assistant interoperates perfectly with the command line; you can switch between them at will at any time.
"""]]


code:
With Amazon EC2, it is possible to take a snapshot of the virtual machine and then bring up multiple instances based on the snapshot.

If I configure git annex assistant to sync a folder to S3, will it work correctly if I spin up multiple instances, all identical git annex assistant configurations?

My use case is to allow users of my forums to upload attachments with their posts. The attachments are stored locally on disk, but I'm hoping that git annex assistant can help synchroise the attachments across all forum instances (I usually run only 1 instance, but their could be as many as 3 under heavy load).


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
 nickname="Justin"
 subject="comment 1"
 date="2013-09-06T22:57:05Z"
 content="""
If you are using ec2 and s3 why don't you just serve the attachments directly from s3?  Not sure how git-annex adds anything here.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.251.174"
 subject="comment 2"
 date="2013-09-07T17:08:27Z"
 content="""
You are going to need to make each repository have a distinct UUID. Otherwise, the assistant will get confused.
"""]]


code:
Hi,
I want to download some files in git and git annex in Windows OS.
I have installed msysgit for git-scm in D:\Git directory,
Now I download git-annex assistant for Windows, and install it
in D:\Git\cmd directory.
But when I run the command "git annex" in git-bash, it says it can not
found the "git annex" command.
My question is that, how should I use git annex in Windows to download files?
For example, git annex get <filename>.
Doea anyone has any idear to help me figure it out?
Thanks in advance!


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.251.174"
 subject="comment 1"
 date="2013-09-07T17:00:03Z"
 content="""
Are you able to run git commands? IIRC msysgit prompts when it's installed about whether you want to put it in the path or not. git-annex just piggybacks on this path setting. It is installed in the same folder as the git.exe and should work as long as that is in path.
Otherwise, consult windows documentation to add git-annex to the path, or you can copy the git-annex.exe to somewhere else I suppose.

I built a clean Windows system yesterday and have just re-tested it and it does work.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkQrsXK-icnYXg6kJNd-jDjMgOixnhwE34"
 nickname="x"
 subject="reply to Comment 1"
 date="2013-09-10T13:46:45Z"
 content="""
Thanks for your helpful comment.I add the path to windows, and the
problem disappears now.
But I meet another problem when I download the data using git annex.
I use the command \"git annex get foo.nc\" to download data. The foo.nc
is actually symlink. For example:
foo.nc -> ../../.git/annex/objects/jK/v7/WORM-s16009120-m1337754158--
foo.nc/WORM-s16009120-m1337754158--foo.nc
Could you help me to figure it out?
Thank you.

The erro information when using git annex in Windows, is as below:

$ git annex get tau.nc

   Detected a crippled filesystem.

   Enabling direct mode.

   Detected a filesystem without fifo support.

   Disabling ssh connection caching.

*** Please tell me who you are.

Run

    git config --global user.email \"you@example.com\"

    git config --user.name \"Your Name\"

to set your account's default identity.

Omit --global to set the identity only in this repository.

fatal: unable to auto-detect email address (got 'xshao@DELL-PC.(none')

git-annex :tau.nc not found

(Recording state in git ...)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkQrsXK-icnYXg6kJNd-jDjMgOixnhwE34"
 nickname="x"
 subject="Comment3"
 date="2013-09-10T15:36:57Z"
 content="""
In addition, because the file is actually symlink direct to the data stored in Amazon S3
How could I get the URL, given the symlink for git annex. For example:
foo.nc -> ../../.git/annex/objects/jK/v7/WORM-s16009120-m1337754158--foo.nc/WORM-s16009120-m1337754158--foo.nc
Thank you.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.51"
 subject="comment 4"
 date="2013-09-12T20:58:30Z"
 content="""
Once `git annex get` has downloaded the file, it will replace the symlink with the contents of the file.

If that is not happening, you might try running `git annex fsck`

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.2.134"
 subject="comment 5"
 date="2013-09-13T18:03:17Z"
 content="""
Turns out that there was a bug in the Windows port that could lead to the behavior described, and another bug that prevented `git annex fsck` from fixing it. I have fixed both bugs, so you should upgrade and then run `git annex fsck`.
"""]]


code:
[[!comment format=mdwn
 username="http://alerque.com/"
 nickname="Caleb"
 subject="path issue"
 date="2014-03-20T13:05:34Z"
 content="""
It seems the use of the cmd folder in the Windows git-annex installer no longer meshes with the git install from msysgit. Changing \"cmd\" to \"bin\" during the install process made it work for me.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.102"
 subject="comment 7"
 date="2014-03-20T16:04:36Z"
 content="""
Hmm, putting git-annex.exe in to bin and not cmd with mysysgit 1.8.5.2 will make \"git annex\" work, but \"git-annex\" will not.

What version of msysgit has the problem with bin?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.102"
 subject="comment 8"
 date="2014-03-20T19:50:26Z"
 content="""
I upgraded to msysgit 1.9.0 and git annex works in cmd.

Did you tell the msysgit installer to \"Run git from the windows command prompt\"? This is the default. If you chose \"Use git bash only\" then git-annex will only work from within git bash.
"""]]


code:
Hello

I'm trying Git annex latest builds available in Linux & Android
Here is the scenario :-

Syncing 2 peer devices running Linux & Android
Devices can access each other directly over Cisco VPN encrypted link

Currently, syncing files via rsync over webdav (over Cisco VPN)
Everything is working, manual file tracking is an issue.

Here's where Git annex steps in,
I've configures external jabber account so they can talk to each other.

But the transfers are super-slow over internet,
I'm guessing Cisco VPN adding 15-20% overhead + SSH adding another 20-30% overhead.

Since the network link is trusted, is there any way to speed things up assuming Cisco solution
remains in place?

looking to solve this issue


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
 nickname="Justin"
 subject="comment 1"
 date="2013-07-12T20:09:32Z"
 content="""
Not sure if it applies to your devices, but I considerably speed up SSH transfers on my raspberry pi by using

    Ciphers arcfour

in the ssh configuration.
"""]]


code:
Hello,

A mode of operation that I find comfortable, and from various online posts it seems that so do some others, is putting whole git repositories in dropbox.

It seems that currently this mode of operations if not supported in a simple manner. (i.e. just add and let assistant do the work). Is there any workaround or possibility to do this?

Thanks!


code:
[[!comment format=mdwn
 username="EvanDeaubl"
 ip="68.230.56.163"
 subject="comment 1"
 date="2013-07-28T02:40:24Z"
 content="""
I'll give this warning upfront: this works for me, but it requires manual intervention, and I wouldn't be surprised if this spectacularly broke with the assistant running in full auto mode.

I'm currently storing git repos inside of my annex by using a separated git dir, so that git doesn't detect the nested repository and balk at adding it. Doing a git clone or git init with a --separate-git-dir=<git dir> argument puts what would normally be in .git in that directory, and creates a simple .git file in its place which references the separated git dir.

When I'm not using the repo, I move the .git file out of the directory, and everything looks like regular files to git-annex. When I want to use it, I move it back in place, and any git operations inside the repo directory use the inner git repo itself.

Another option I used for a while was to store bare repos in the annex, and doing checkouts from those repos.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmNu4V5fvpLlBhaCUfXXOB0MI5NXwh8SkU"
 nickname="Adam"
 subject="Many months later..."
 date="2014-05-08T16:49:26Z"
 content="""
Here is the relevant bug: <http://git-annex.branchable.com/bugs/Can__39__t_add_a_git_repo_to_git_annex:___34__Invalid_path_repo__47__.git__47__X__34___for_many_X>

Here's my scenario: I have been using Dropbox for several years.  I want to move to git-annex.

For some time I used Bazaar repos inside Dropbox.  A couple of times I accidentally modified files on one computer before another finished syncing, and the repo became mildly corrupted, but I recovered it okay.

Recently I switched to git, and so far, so good.  I'm careful not to make any changes to my files before Dropbox is finished syncing, so everything works fine.

But when I add a directory containing a git repo to an annex, git-annex ignores the .git directory, so the repo isn't synced.

I read this forum thread, and I read the bug report, but there don't appear to be any real solutions.

This seems like what should be a fairly common use case: a user has a directory of simple text files, like shell scripts or org-mode files.  He uses git to version-control them, and he uses Dropbox to sync the files and the git repo automatically between computers.  Now the user wants to use git-annex instead, but if he does this, it will mean having to choose between:

* No longer using git to version-control the files
* Only using git on a single system, which means that he can't make commits or do any other git operations when using other systems
* Keeping separate git repos on each system, and making the same commits manually on each one...which would be a mess
* Keeping separate git repos on each system, and using a script to make commits automatically--which would mean commits wouldn't be any more useful than a simple timed backup script

I guess I could just use git instead of Dropbox or git-annex, but then I must choose between either:

* Only being able to sync repos when both systems are online
* Having to store repos in plaintext on remote servers

This is one of my primary use cases for Dropbox/git-annex.  I'd really appreciate any help in figuring this out.  Thanks.
"""]]


code:
First of all thanks for the great work. git-annex looks very nice already!

I wonder if it's possible to add Git repositories to the annex. In my current Dropbox setup I drop some small Git repositories into the Dropbox to keep them in sync on two computers.

This is obviously not possible with git-annex, as the annex itself is a Git repository and sub-repositories (directories with .git folders) are ignored. But in some cases people might want to be able to drop "anything" into the annex, maybe even Git repositories (e.g. the "nomad" usecase)...


code:
I am having some issues trying to make git-annex work with the latest version of Gitolite,
When running git annex sync, or git annex copy --to I get.
"""
Failed to get annex.uuid configuration of repository origin

Instead, got: "annex.uuid=\ncore.gcrypt-id=\n"
"""

Has anyone found any solution to this? Also, I am using the latest version of Gitolite, because I wanted to see whether it would work.
Thanks, Zack.
EDIT: It might also be worthy to note that I did have a go at following the guide on this site.


code:
[[!comment format=mdwn
 username="Renaud"
 ip="126.10.66.235"
 subject="comment 1"
 date="2014-01-11T14:29:08Z"
 content="""
I had that problem when I tried to access a newly created remote repository.
It seems git-annex doesn't recognize anymore if the remote repository has been initialized or not.
The workaround I used is create a local repository, initialize it with git annex, `clone --bare` it, add the bare repository as a remote and `git annex sync` it to initialize the bare repository. Then, you can put the bare repository where gitolite usualy keep its repositories and use it normally.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmqaNwDQ367zpW6cIRviLz6zJZZFODgoEI"
 nickname="Zack"
 subject="Almost there!"
 date="2014-01-12T19:13:29Z"
 content="""
Thanks for the help (I can now clone the repo.), but I now get this.
get file FATAL: bad git-annex-shell command: git-annex-shell 'configlist' '/annex2.git' at /opt/gitolite2/bin/commands/git-annex-shell line 25, <DATA> line 1.
"""]]


code:
I am in the process of organising all of my documents (pdf/ps/etc.) and putting them into an annex.  It seems like the easiest (and smartest?) way for me to manage my 15,000+ document library...  (Yeah, I've not read them all!)

However, one of the issues I have run into is when I want to include something like a software manual...

I'm not even sure if git-annex is the correct way to handle these sort of document, but...

What would you do if the URL stays the same, but the file content changes over time?

For example, the administration manual for R gets updated and re-released for each release of R, but the URL stays the same.

    http://cran.r-project.org/doc/manuals/R-admin.pdf

I'm not particularly worried about whether my annex version is the most recent, just that if I want to 'annex get' it, it will pull *a* version from somewhere.

Any thoughts?

I'd bet that the SHA-hash would change between releases(!) so would a WORM backend be the best approach?  It would mess up my one-file-in-multiple-directories (i.e. multiple simlinks to the one SHA-...-.....) approach.


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2012-01-03T00:57:55Z"
 content="""
The web special remote will happily download files when you `git annex get` even if they don't have the same content that they did before.

`git annex fsck` will detect such mismatched content to the best ability of the backend (so checking the SHA1, or verifying the file size at least matches if you use WORM), and complain and move such mismatched content aside. `git annex addurl --fast` deserves a special mention; it uses a backend that only records the URL, and so if it's used, fsck cannot later detect such changes. Which might be what you want..

For most users, this is one of the reasons `git annex untrust web` is a recommended configuration. Once you untrust the web, any content you download from the web will be kept around in one of your own git-annex repositories, rather than the untrustworthy web being the old copy.
"""]]


code:
[[!comment format=mdwn
 username="http://a-or-b.myopenid.com/"
 ip="203.45.2.230"
 subject="comment 2"
 date="2012-01-03T02:49:18Z"
 content="""
Thanks!  `git annex addurl --fast` does exactly what I want it to do.

Wow.  Yet another special backend for me to play with.  :-)
"""]]


code:
Hello,

i am trying to run the walkthrough on Windows 7. When i try to get the contents of a file, i only get a some git annex text string and not the real file. Both repositories are on the same ntfs filesystem.

C:\tmp>git annex version
git-annex version: 4.20130827-g4f18612
build flags: Pairing Testsuite S3 WebDAV DNS
local repository version: 4
default repository version: 3
supported repository versions: 3 4
upgrade supported from repository versions: 2

C:\tmp\server>git --version
git version 1.8.3.msysgit.0

# walkthrough.bat

    doskey /history > commands.log
    mkdir laptop
    cd laptop
    git init
    git annex init laptop
    cd ..

    git clone laptop server
    cd server
    git annex init server
    git remote add laptop c:\tmp\laptop

    cd ..\laptop
    git remote add server c:\tmp\server
    copy ..\1.pdf .
    git annex add 1.pdf
    git commit -m add
    dir

    cd ..\server
    dir
    git fetch laptop
    git merge laptop/master
    git annex get 1.pdf
    dir
    type 1.pdf

# walkthrough.log

    C:\tmp>walkthrough.bat

    C:\tmp>doskey /history  1>commands.log

    C:\tmp>mkdir laptop

    C:\tmp>cd laptop

    C:\tmp\laptop>git init
    Initialized empty Git repository in C:/tmp/laptop/.git/

    C:\tmp\laptop>git annex init laptop
    init laptop
      Detected a crippled filesystem.

      Enabling direct mode.

      Detected a filesystem without fifo support.

      Disabling ssh connection caching.
    ok
    (Recording state in git...)

    C:\tmp\laptop>cd ..

    C:\tmp>git clone laptop server
    Cloning into 'server'...
    done.
    warning: remote HEAD refers to nonexistent ref, unable to checkout.

    C:\tmp>cd server

    C:\tmp\server>git annex init server
    init server
      Detected a crippled filesystem.

      Enabling direct mode.

      Detected a filesystem without fifo support.

      Disabling ssh connection caching.
    ok
    (Recording state in git...)

    C:\tmp\server>git remote add laptop c:\tmp\laptop

    C:\tmp\server>cd ..\laptop

    C:\tmp\laptop>git remote add server c:\tmp\server

    C:\tmp\laptop>copy ..\1.pdf .
            1 file(s) copied.

    C:\tmp\laptop>git annex add 1.pdf
    add 1.pdf (checksum...) ok
    (Recording state in git...)

    C:\tmp\laptop>git commit -m add
    [master (root-commit) 7ad1514] add
     1 file changed, 1 insertion(+)
     create mode 120000 1.pdf

    C:\tmp\laptop>dir
     Volume in drive C has no label.
     Volume Serial Number is x

     Directory of C:\tmp\laptop

    09/01/2013  11:03 AM    <DIR>          .
    09/01/2013  11:03 AM    <DIR>          ..
    08/30/2013  12:43 PM            37,500 1.pdf
                   1 File(s)         37,500 bytes
                   2 Dir(s)   7,698,817,024 bytes free

    C:\tmp\laptop>cd ..\server

    C:\tmp\server>dir
     Volume in drive C has no label.
     Volume Serial Number is x

     Directory of C:\tmp\server

    09/01/2013  11:03 AM    <DIR>          .
    09/01/2013  11:03 AM    <DIR>          ..
                   0 File(s)              0 bytes
                   2 Dir(s)   7,698,817,024 bytes free

    C:\tmp\server>git fetch laptop
    remote: Counting objects: 9, done.
    remote: Compressing objects: 100% (6/6), done.
    remote: Total 8 (delta 1), reused 0 (delta 0)
    Unpacking objects: 100% (8/8), done.
    From c:\tmp\laptop
     * [new branch]      git-annex  -> laptop/git-annex
     * [new branch]      master     -> laptop/master

    C:\tmp\server>git merge laptop/master

    C:\tmp\server>git annex get 1.pdf
    get 1.pdf (merging laptop/git-annex origin/git-annex into git-annex...)
    (Recording state in git...)
    (from laptop...)
    1.pdf
           37500 100%    4.51MB/s    0:00:00 (xfer#1, to-check=0/1)

    sent 37573 bytes  received 31 bytes  75208.00 bytes/sec
    total size is 37500  speedup is 1.00
    ok
    (Recording state in git...)

    C:\tmp\server>dir
     Volume in drive C has no label.
     Volume Serial Number is x

     Directory of C:\tmp\server

    09/01/2013  11:03 AM    <DIR>          .
    09/01/2013  11:03 AM    <DIR>          ..
    09/01/2013  11:03 AM               194 1.pdf
                   1 File(s)            194 bytes
                   2 Dir(s)   7,698,767,872 bytes free

    C:\tmp\server>type 1.pdf
    .git/annex/objects/kM/0q/SHA256E-s37500--32d8190c7e189d45f48245a100e4cc981ea1bbc
    02ac8bfa6188db73e41ce06f3.pdf/SHA256E-s37500--32d8190c7e189d45f48245a100e4cc981e
    a1bbc02ac8bfa6188db73e41ce06f3.pdfC:\tmp\server>
    C:\tmp\server>


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.8.7"
 subject="comment 1"
 date="2013-09-03T17:59:03Z"
 content="""
The walkthrough assumes a system that uses indirect mode by default, so it won't work quite right on Windows, which is forced to use direct mode.

Running `git annex fsck` in the server repository will fix up this situation, but the right thing on Windows is to use `git annex sync` rather than the manual `git fetch + git merge` the walkthrough shows.

Guess I'll make the walkthrough use sync, although it may make it harder for people to understand what's going on internally.
"""]]


code:
I am using git-annex assistant for a few months now and still don't think I have it all setup correctly. My setup is:

* Laptop Computer with low storage. Content creator, assistant: file source
* Desktop Computer with medium storage capacity. Consumer, assistant: manual Mode
* 2TB USB HD (EXT4) Backup, assistant: full backup
* 2.5" portable 500GB drive NTFS formatted. transfer to backup: assistant: tried transfer and partial backup
* Root Server on the web, not yet in the mix. Planned: Index only / content creator

My primary goal is getting the stuff from the Laptop to the 2TB USB drive, syncing the annex with the desktop so I can get files on the desktop from the backup.
What works:

* Checking stuff into the annex on the Laptop
* When the 2TB drive is connected to the Laptop, stuff is automatically moved there too
* Bare Repo on the 2.5" configured as partial backup automatically backups the data

What doesn't work:

* Connecting the 2.5" bare repo to the Desktop to move stuff to the backup drive automatically
* Having a human readable folder Structure on the 2.5" NTFS drive. I'd be ok to have a bare repo plus a special remote on the drive, but I don't know how to configure it
* How do I drop stuff from the bare repo on the 2.5" drive? I think a partial backup should drop content that is in a full backup, but it doesn't

Thanks for reading this long post. If anyone has any tips on how to configure the assistant, thats much appreciated.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.246"
 subject="comment 1"
 date="2013-11-09T18:18:49Z"
 content="""
An incremental backup is supposed to drop content once it reaches a full backup. But, the assistant can only do that if it's able to verify that the full backup drive (or some other repository it can access) has a file, at the same time it's dropping it from the incremental backup drive. If you don't have both the incremental and full backup drives plugged in at the same time, you could consider making git-annex [[trust]] the full backup drive, which would avoid this explicit check.

It sounds like you want a non-bare git repository on the 2.5\" drive. The assistant does not currently support setting that up, but you can do it by just using `git clone` as shown in the [[walkthrough]].

I'm not clear on what doesn't work when you connect the 2.5\" drive to your desktop.
"""]]


code:
Hi everyone,

everyday I understand more and more how git and git-annex work but I need help with this one.
I guess I have two questions but let me describe the scenario first:

I have one local repository of mp3s (assume just one file: file1.mp3).
I clone that repository into a remote git-annex repository over ssh and "git annex get" the file contents for file1.mp3.

I unlock some mp3's locally and modify some mp3 tags.
Then I notify git-annex of these changes
with "git annex add *"
and commit them with "git commit -m 'mp3 tags changed'".
[git annex locks them again and changes the symlinks to point to the changed file in the annex, git commits the changed symlink]

At this point in time there are two objects in my git annex repository:
hash(file1.mp3)
hash(file1.mp3|with modified tags)
The symlink points now to hash(file1.mp3|with modified tags)

At this point the remote still does not now of this commit and of the new file contents.
Thus I do "git push" to send the changes to the remote.
The remote now has a BROKEN symlink because it already points to hash(file1.mp3|with modified tags)
but the remote annex's object directory only contains hash(file1.mp3).

Then I want my remote repository to also have the updated mp3 tags.
The only way I see (without scripting) to have the updated mp3 tags in the remote repository is to do an "git annex get file1.mp3" on the remote repository or an "git annex copy --to remote file1.mp3" at my local repository. However, although the binary differences between both files
file1.mp3 and file1.mp3|with modified tags are small the latter is transferred completely from the local repository to the remote repository.

This is not a problem when just changing one file, but a problem when I have 10GB's of files and when it takes 2days to upload them to the remote because of a low bandwidth.

First question: Did I miss something? Does git-annex already provide means to only transmit the diff between the two objects?

Second question regarding disk space.
I now have a complete history of all changes to file1.mp3 in my git-annex repository. I have the objects that represent every state of file1.mp3 and I can go back to these states when I checkout the respective commits and thereby the symlinks that link to these "old" objects. This history can take up a lot of space. What is the clean way to forget the past? AFAIK "git drop unused" only drops file contents that are not referenced in any commit?

If one wanted to preserve the entire history but save disk space one could also only store the current content and the patches that allow to reconstruct older versions from the current one. I understand that applying several patches consecutively takes more cpu time but for me going back to an older commit with my binary files only happens rarely.

This is the algorithm I have in mind for an optimized "git annex get file1":

On that repository where the file is missing:

1. Find the newest object that represented the contents of file1 in file1's commit history.

2. Transmit this object identification hash(object) to the remote that has the current version (the one I am getting from).

At that remote:

If the history contains full versions of file contents:

Create a binary diff between the object identified by hash(object) and the current content of file1.

If the history contains only the current version and patches to older versions:
Collect all patches that represent the change from hash(object) to the current content of file1.

If the list of patches is bigger than the current content of file 1 transmit the current content of file1. Otherwise transmit the patch(es)

On that repository where the file is missing:

Apply the patch(es) to the latest object to obtain the current object.

What's your opinion on this?

Marek


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmp1ThsNNAbSn46ju-gwFELfStlhl8usJo"
 nickname="donkeyicydragon"
 subject="Possible quick solution with rsync"
 date="2013-05-28T10:52:30Z"
 content="""
One could achieve the effect of only transmitting the changes of file contents using rsync.

On the repository that lacks the current version:

cp latestversionavailable currentversion
rsync remoterepository/currentversion ./currentversion

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmp1ThsNNAbSn46ju-gwFELfStlhl8usJo"
 nickname="donkeyicydragon"
 subject="I should have read git annex unused first"
 date="2013-05-28T22:14:46Z"
 content="""
Apparently \"git annex unused\" finds previous versions of the object that are not symlinked anymore. However, if a file is renamed then the oldest version before the rename is not marked as unused.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 3"
 date="2013-05-29T16:50:35Z"
 content="""
Treating each version of a file as a separate object that is stored and transmitted separately is one of the core simplifying assumptions of git-annex. Among other things it makes it easy to have dumb special remotes that are mere key/value stores. And it keeps it easy to understand.

I don't consider this a problem as far as storage goes. Disk space is cheap; git-annex allows dropping unused versions of files from repositories that are not on large cheap storage.

For file transfer, it would be nice to find optimisations that don't require changing this simplifying assumption. This can be done at least for rsync I think. The [[design/assistant/deltas]] page is where the design of this is or will be worked on.
"""]]


code:
Hello.
How can I use git-annex get a list of files in directory that have content? For each such file the result of command

     git-annex get [file]

will be empty.


code:
[[!comment format=mdwn
 username="arand"
 ip="130.243.226.21"
 subject="comment 1"
 date="2013-12-30T13:15:20Z"
 content="""
    git annex find <path>

that's what you meant?
"""]]


code:
[[!comment format=mdwn
 username="Alexcei"
 ip="92.255.239.77"
 subject="comment 2"
 date="2013-12-30T13:25:45Z"
 content="""
Yes, this is what I need. I have not seen this command before. Thank you.
"""]]


code: I had accidentally chosen the wrong directory for the repository. What should I do to clean all of git-annex?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.63"
 subject="comment 1"
 date="2013-08-24T18:43:08Z"
 content="""
It should be sufficient to delete `/sdcard/git-annex.home` and delete the .git directory inside whatever directory you set up as the repository.
"""]]


code: There is .gitrefs directory in my annexed repo while I am unaware. Should I do `git add .gitrefs/` or `git annex add .gitrefs/` or just ignore?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.35"
 subject="comment 1"
 date="2014-01-06T15:44:15Z"
 content="""
AFAIK neither git nor git-annex creates a .gitrefs directory.
"""]]


code:
[[!comment format=mdwn
 username="EskildHustvedt"
 ip="80.203.53.96"
 subject="comment 2"
 date="2014-01-11T07:51:57Z"
 content="""
I have noticed a .gitrefs/ directory being added to my git-annex repositories as well.
Two direct-mode git-annex assistant-managed repos have turned up a .gitrefs directory the past month or so, with the directory often returning after I remove it. First occurrance according to git logs is on the 3rd of December.

The directory looks a lot like .git/refs, as if some tool is trying to do something to .git/refs but has a bug that removes the first /, and then proceeds to copy or recreate the files.

    [0 zerodogg@firefly annexed]$ du -hs .gitrefs .git/refs
    172K    .gitrefs
    192K    .git/refs
    [0 zerodogg@firefly annexed]$ find .gitrefs > /tmp/gitr1
    [0 zerodogg@firefly annexed]$ find .git/refs > /tmp/gitr2
    [0 zerodogg@firefly annexed]$ perl -pi -e 's/^\.git.?refs//' /tmp/gitr*
    [0 zerodogg@firefly annexed]$ diff -BrayN --suppress-common-lines /tmp/gitr1 /tmp/gitr2
                                                                  > /tags
                                                                  > /remotes/browncoats
                                                                  > /remotes/browncoats/master
                                                                  > /remotes/browncoats/synced
                                                                  > /remotes/browncoats/synced/master
                                                                  > /remotes/browncoats/synced/git-annex
                                                                  > /remotes/browncoats/git-annex
                                                                  > /remotes/browncoats/annex
                                                                  > /remotes/browncoats/annex/direct
                                                                  > /remotes/browncoats/annex/direct/master
    /remotes/serenity/master                                      <
    /remotes/zoe                                                  | /remotes/faerun/annex
    /remotes/zoe/master                                           | /remotes/faerun/annex/direct
    /remotes/zoe/synced                                           | /remotes/faerunpi
    /remotes/zoe/synced/master                                    | /remotes/faerunpi/synced
    /remotes/zoe/synced/git-annex                                 | /remotes/faerunpi/annex
    /remotes/firefly                                              | /remotes/faerunpi/annex/direct
    /remotes/firefly/master                                       <
    /remotes/firefly/synced                                       <
    /remotes/firefly/synced/master                                <
    /remotes/firefly/synced/git-annex                             <
    /remotes/firefly/git-annex                                    <
    /remotes/firefly/annex                                        <
    /remotes/firefly/annex/direct                                 <
    /remotes/firefly/annex/direct/master                          <
    /remotes/river/master                                         <
                                                                  > /remotes/origin
                                                                  > /remotes/origin/synced
                                                                  > /remotes/hufsa
                                                                  > /remotes/hufsa/synced
                                                                  > /synced

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.43"
 subject="comment 3"
 date="2014-01-15T20:40:35Z"
 content="""
A bug in the git repository repair code caused packed-refs files to be exploded to .gitrefs instead of .git/refs. I have fixed that bug.

To fix up a repository that this happened to, I'd recommend checking if there are any tags and branches in the .gitrefs directory that have gone missing from .git/refs.. If so, you could move the files back into place (unannexing them first if necessary). Otherwise, deleting .gitrefs would seem to be the thing to do.
"""]]


code:
`git annex dropunused` is simple enough.

But how do I perform the equivalent procedure on an rsync remote?  I'd presume that `git annex copy --to remote` is not sufficient.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmUJBh1lYmvfCCiGr3yrdx-QhuLCSRnU5c"
 nickname="Justin"
 subject="comment 1"
 date="2013-05-12T23:06:34Z"
 content="""
Oh, sheesh, it's right there at the bottom of http://git-annex.branchable.com/special_remotes/.  Oops.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-05-13T18:03:23Z"
 content="""
Well, in your defense it's documented there because I can't think of a better place to put it. Maybe you looked in some other places first before finding it there, and I should add a pointer to those places?

(For the curious, the magic run is just: git annex unused --from remote)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmUJBh1lYmvfCCiGr3yrdx-QhuLCSRnU5c"
 nickname="Justin"
 subject="comment 3"
 date="2013-05-13T18:56:32Z"
 content="""
I think in general the `git annex command --help` bits could be more thorough.  If `git annex dropunused --help` indicated that it took `--from`, that would probably have helped me.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 4"
 date="2013-05-13T18:59:08Z"
 content="""
You might be using an older version. Here's the output of that command in the current version:

<pre>
Usage: git-annex dropunused NUM|RANGE ... [option ...]
  -f REMOTE  --from=REMOTE  drop content from a remote

To see additional options common to all commands, run: git annex help options
</pre>
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmUJBh1lYmvfCCiGr3yrdx-QhuLCSRnU5c"
 nickname="Justin"
 subject="comment 5"
 date="2013-05-13T19:07:11Z"
 content="""
Ah, rebuilding got me that.  That's pretty good!
"""]]


code:
Hello,

I had some trouble adding a remote (the files would not appear when I was copying them to the remote), so I started over and cloned an existing repository.

Of course, as I started over, I had a duplicate uuid for the remote, which would cause problems when trying to copy (I would have an error "git-annex-shell: expected repository UUID 70582c7b-0b57-4087-a9d1-77b5f5f3c75e but found UUID 335699ea-d5b8-49ff-b207-1571b5969afe").

I finally managed to find the wrong uuid in the .git/config file (there was a duplicated entry for the remote) and I'm now able to copy things to the remote repository, and "git annex sync" works well. However I still see a mention of this repository when I do a "git annex whereis":

    cody:games schmitta$ git annex whereis
    whereis dungeon_keeper_1.1.0.11.dmg (3 copies)
      	1cdfb490-0660-41fb-b7ce-74b89abb9aac -- top
       	335699ea-d5b8-49ff-b207-1571b5969afe -- here (cody)
       	70582c7b-0b57-4087-a9d1-77b5f5f3c75e

Where can I find where this last line come from, and how can I get rid of it? I tried saying that this uuid is dead, but git annex tells me it's not a remote name.


code:
[[!comment format=mdwn
 username="http://alan.petitepomme.net/"
 nickname="Alan Schmitt"
 subject="comment 1"
 date="2014-03-08T06:42:29Z"
 content="""
If it helps, I had a look at \"git annex vicfg\", and there is no mention of the extra UUID there.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.154"
 subject="comment 2"
 date="2014-03-12T19:35:26Z"
 content="""
Probably this will fix it:

`git annex dead 70582c7b-0b57-4087-a9d1-77b5f5f3c75e`
"""]]


code:
[[!comment format=mdwn
 username="http://alan.petitepomme.net/"
 nickname="Alan Schmitt"
 subject="comment 3"
 date="2014-03-13T20:05:53Z"
 content="""
Unfortunately git annex tells me there is no such remote:

    cody:games schmitta$ git annex dead 70582c7b-0b57-4087-a9d1-77b5f5f3c75e
    dead 70582c7b-0b57-4087-a9d1-77b5f5f3c75e git-annex: there is no available git remote named \"70582c7b-0b57-4087-a9d1-77b5f5f3c75e\"
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.154"
 subject="comment 4"
 date="2014-03-14T17:46:23Z"
 content="""
It seems that the git-annex branch's uuid.log must somehow not list this uuid, but it's used in the location tracking log files.

The only way I can think of that this could happen is if you had set up a repository, run git-annex init, and then went in and changed the annex.uuid setting to this other uuid, and added files with that misconfiguration. Does that sound like what happened?

The fix is just as evil as the cause -- you can edit .git/config to add a new, dummy git remote that has annex-uuid set to the problem uuid, and then `git-annex dead` can be used to kill the uuid via that remote. For example:

[[!format sh \"\"\"
joey@darkstar:~/tmp/x>git annex whereis eep
whereis eep (1 copy)
  	00000a6d-e770-4ab9-a640-7d6272e9ffff
ok
joey@darkstar:~/tmp/x>git annex dead 00000a6d-e770-4ab9-a640-7d6272e9ffff
dead 00000a6d-e770-4ab9-a640-7d6272e9ffff git-annex: there is no available git remote named \"00000a6d-e770-4ab9-a640-7d6272e9ffff\"
- exit 1
joey@darkstar:~/tmp/x>git remote add dummy dummy
joey@darkstar:~/tmp/x>git config remote.dummy.annex-uuid 00000a6d-e770-4ab9-a640-7d6272e9ffff
joey@darkstar:~/tmp/x>git annex dead dummy
dead dummy ok
(Recording state in git...)
joey@darkstar:~/tmp/x>git annex whereis eep
whereis eep (0 copies) failed
\"\"\"]]
"""]]


code:
[[!comment format=mdwn
 username="http://alan.petitepomme.net/"
 nickname="Alan Schmitt"
 subject="comment 5"
 date="2014-03-14T18:11:52Z"
 content="""
Thanks, it worked!

Regarding how I got into that state, here is what happened:
- I configured a new repository (git init, git annex init) on machine B
- I added that repository as a remote from machine A
- I started pushing files from machine A to machine B, starting with the file that later showed this problem
- I then noticed that, although the files were said to be successfully sent, I could not see them on machine B (no symbolic link was created), even after a \"git annex sync\" there
- I decided to start over, deleting the annex, and creating it from a clone of an existing annex
- This resulted in a duplicated UUID for the same path, which led to this problem

Maybe some documentation on how to cleanly start over when things get in a bad state would be very useful.

Thanks again,

Alan
"""]]


code:
Hi Joey,
    I was wondering how you know when a file in your repo fails a fsck.  I'm having trouble finding out since I can't change any of the objects (which is great!).  I'd just like to know since I have thousands of files in one of my annexes, and scrolling through my term buffer is unwieldy.  I'd assume it would show up when the fsck was done, or in 'git annex status'...

Thanks!


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
 nickname="Justin"
 subject="comment 1"
 date="2013-06-10T14:23:25Z"
 content="""
    justin@box:/tmp/f$ git init
    Initialized empty Git repository in /tmp/f/.git/
    justin@box:/tmp/f$ git-annex init
    init  ok
    (Recording state in git...)
    justin@box:/tmp/f$ cp /etc/motd .
    justin@box:/tmp/f$ git-annex add .
    add motd (checksum...) ok
    (Recording state in git...)
    justin@box:/tmp/f$ git-annex fsck
    fsck motd (checksum...) ok
    justin@box:/tmp/f$ chmod +w motd
    justin@box:/tmp/f$ echo hi >> motd
    justin@box:/tmp/f$ git-annex fsck
    fsck motd
      Bad file size (3 B larger); moved to /tmp/f/.git/annex/bad/SHA256-s354--2e724dde1a5dc33bc15580b2aef1ee541ca8047d746fff9bb7917062b871c0bf

      ** No known copies exist of motd
    failed
    (Recording state in git...)
    git-annex: fsck: 1 failed

You can also use fsck -q which will only show errors.

"""]]


code:
When starting git-annex assistant with:

    git-annex webapp

an unwanted web browser is opened.

How does one change the web browser with which the web interface is displayed?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.60"
 subject="comment 1"
 date="2012-09-17T15:32:07Z"
 content="""
The web browser is started by running the `xdg-open` program (or just `open` on Mac OS X).

xdg-open, in turn, runs the desktop environment's kde-open, gvfs-open, exo-open. Which browser these run is configured in the desktop environment's settings. Which is my goal for the web app; it should open the same browser that other applications in the desktop use.

When there's no desktop environment in use, xdg-open falls back to some default browser, or to what's configured by the BROWSER environment variable.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlZvtBCVyJw4O71OPsdwGpVh6iJ1W-xaPc"
 nickname="Kilian"
 subject="comment 2"
 date="2012-10-11T21:56:26Z"
 content="""
On my Ubuntu 10.04 machine, ./git-annex-webapp starts Firefox whereas xdg-open starts my configured standard browser, Chrome.

-ke
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.246.8"
 subject="comment 3"
 date="2012-10-11T23:41:36Z"
 content="""
Recently the webapp has started honoring git's web.browser setting, so if that's set in ~/.gitconfig, it'll use a different browser than xdg-open does.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnBJ6Dv1glxzzi4qIzGFNa6F-mfHIvv9Ck"
 nickname="Jim"
 subject="Doesn't use the configured web browser app..."
 date="2012-12-04T23:10:43Z"
 content="""
On my system, `xdg-open http://google.com/` opens firefox as desired, but `xdg-open file:///tmp/foo.html` tries to load **wine**!
I'm using xfce, and it seems like exo-open is noticing the `file://` URL, finding the mime type, and looking it up in `~/.local/share/applications/wine-extension-html.desktop`, which was (annoyingly) created by wine.  Deleting those files fixes it.

Not sure how you'd make this better.  Maybe give xdg-open a URL like `http://localhost:54896/?authfile=...` that redirects to `file:///tmp/webapp12345.html` that redirects to `http://localhost:54896/?auth=...` ?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.72"
 subject="comment 5"
 date="2012-12-05T01:21:03Z"
 content="""
@Jim, that would be a good technique.. except many browsers prohibit redirecting to file:// urls.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnZEanlyzay_QlEAL0CWpyZcRTyN7vay8U"
 nickname="Carlo"
 subject="comment 6"
 date="2013-11-15T13:43:06Z"
 content="""
Since git web--browse defaults to firefox on my system (Ubuntu/XMonad), I did this on Ubuntu to get the system default browser.

  git config --global web.browser xdg-open

"""]]


code:
[[!comment format=mdwn
 username="rupi"
 ip="80.110.152.13"
 subject="xdg-open"
 date="2014-02-28T17:48:23Z"
 content="""
Since I discovered the same behavior mentioned above (git annex webapp
tries to use 'winebrowser'). There is an easy way to work with the difference
between http
url and html files: enter preferences for both:

In ~/.local/share/applications/mimeapps.list:

		[Added Associations]
		..
		x-scheme-handler/http=iceweasel.desktop;
		x-scheme-handler/https=iceweasel.desktop;
		text/html=icewesel.desktop;

		[Default Applications]
		x-scheme-handler/http=iceweasel.desktop
		x-scheme-handler/https=iceweasel.desktop
		text/html=iceweasel.desktop

Your preferred browser might be a different one off course. And I agree that this has to be the most logical place to store that information ever..

"""]]


code:
I want to upload and download more than one file at a time to and from the various remotes.

How do I do that?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="71.80.94.56"
 subject="comment 1"
 date="2014-02-07T19:12:10Z"
 content="""
This is not currently supported. See [[todo/Slow_transfer_for_a_lot_of_small_files.]]

 You can click on the button next to individual files in the transfer queue to force concurrent transfers.

"""]]


code:
What is the safest way to cancel a `git annex add` before the first commit?

Say I first added a directory with a lot of small files to the annex and then (while watching the endless list of checksum operations) decide I should actually put them in a tar file and annex that instead.

The problem seems to be that `git annex add` immediately replaces the files with symlinks instead of waiting for a commit, so I can't just `git reset largedir/; tar cvfz largedir.tgz largedir/; git annex add largedir.tgz`.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 1"
 date="2013-11-03T00:50:48Z"
 content="""
Run `git annex unannex` on the files.
"""]]


code:
git-annex created a gitannexxxxxxxxxxxxxxxxxxxxx folder on S3 and I guess I'm a bit OCD but I'd like to rename it to git-annex.
Of course I can rename it in my S3 console but where does git-annex store that name so I can also change it on my local install of git-annex.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 1"
 date="2014-05-16T16:28:40Z"
 content="""
If you look at the [[S3_documentation|special_remotes/S3]] you will see a `bucket` setting that does what you want. You can use `git annex enableremote` to change it.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkAUMhKOSkh9JaBA6xst3XxQIIsDEq5Zd4"
 nickname="Ovidiu"
 subject="comment 2"
 date="2014-05-16T18:08:11Z"
 content="""
Thanks but I realized there's no way to rename an S3 bucket so giving up on this idea...
"""]]


code:
I am in direct mode (managing music) and have no other repositories than the one I'm in.  I would like to delete a file and have git no longer track it at all.

So far I've manually deleted the file and run "git annex add .", but when I run "git annex status" the file is still marked as deleted.  I would like to receive no message whatsoever.

I tried "git annex drop path" , "git annex unannex path", and everything else I can think of but nothing will get git annex to forget the file.

I'm sure this is easy, but how do you remove a deleted file in direct mode?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 1"
 date="2014-07-02T16:57:46Z"
 content="""
git annex sync
"""]]


code:
Hello,

I want to be safe and have two copies of my files on two different backend. Currently I only have a SSH backend, that stores all my data. I have full(root) access to that machine/backend. On my laptop I have only a few bytes of data, because all is moved/copied to that SSH backend. Now, I want to duplicate the data on the SSH backend to a Google Drive account (or any other). How could I do that (without downloading all data from the SSH backend)??? Encryption is not a must.

I looked into the annex/objects folder on the SSH backend, but there are 3 char length directories compared to what I see on a test Google Drive backend, where only 2 char length directory names are.

Example SSH backend: [git-annex root]/annex/objects/c10/90a/SHA256E-s445227--14c3f85d6dd3464f116f6a5bbd411012781d36794549d136b18d1914c4158820.jpg/SHA256E-s445227--14c3f85d6dd3464f116f6a5bbd411012781d36794549d136b18d1914c4158820.jpg

Example Google Drive: [Google Drive root]/annex/W7/xQ/SHA256E-s913904--29f9800b0dd34d4200c4e9ee152b79c3556a9a473848720be7cf83d20eff65a4.JPG

Is there a way to convert these directory names and do a simpe copy???

Thank you,
Bence


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.251.174"
 subject="comment 1"
 date="2013-09-07T17:11:03Z"
 content="""
Just run `git annex copy --all --to remote`

(Needs git-annex 4.20130709 or newer.)
"""]]