code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmU_2tE75oyG0h2ZPN4lcroIKEMC8G-otE"
 nickname="Michael"
 subject="comment 2"
 date="2012-08-30T11:32:24Z"
 content="""
That helped now the compile stops at:

    Utility/WebApp.hs:106:9:
        Ambiguous occurrence `liftIO'
        It could refer to either `Yesod.liftIO',
                                 imported from `Yesod' at Utility/WebApp.hs:14:1-12
                                 (and originally defined in `transformers-0.2.2.0:Control.Monad.IO.Class')
                              or `Control.Monad.IO.Class.liftIO',
                                 imported from `Control.Monad.IO.Class' at Utility/WebApp.hs:18:1-29
    make: *** [git-annex] Fehler 1

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.246.16"
 subject="comment 3"
 date="2012-08-30T17:07:44Z"
 content="""
I've fixed this.. I think. I don't have this problem with the version of yesod I have here.
"""]]

repo_name: kakafarm/git

license: unknown


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmU_2tE75oyG0h2ZPN4lcroIKEMC8G-otE"
 nickname="Michael"
 subject="comment 4"
 date="2012-08-31T08:04:18Z"
 content="""
Success! Now it compiles fine on Ubuntu 12.04 with make. Thank you.
"""]]


code:
I have a repository synchronized between two PCs and box.com . I chose encryption for box.com. Today on my work computer I changed files, on box.com I see that there are changed files today. When I got home, I still see files with yesterday's date.
One thing to mention is that currently the work PC is not accessible.

1. How do I debug this problem? I have logging set to debug via web console, saved, restarted daemon, but after restart the debug checkbox is unchecked.
From logs it looks like that after failing to connect to work pc via ssh it gives up:

    [2013-05-09 21:42:52 CEST] main: Syncing with box.com
this is first and last line mentioning box today, I restarted the daemon several times around 22:14 :

    [2013-05-09 22:11:40 CEST] TransferScanner: Syncing with [work pc repo]
    Already up-to-date.

    (scanning...) [2013-05-09 22:11:40 CEST] Watcher: Performing startup scan
    Already up-to-date.

    (started...) ssh: connect to host [work pc] port 22: No route to host
    fatal: The remote end hung up unexpectedly

2. where is the configuration for box.com stored? How can I check gpg key ID mentioned at http://git-annex.branchable.com/tips/using_box.com_as_a_special_remote/
3. how do I manually trigger sync with box.com? git fetch box.com neither git annex sync box.com doesn't work.

---
Version: 4.20130501
Build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-05-13T17:56:35Z"
 content="""
From what you've described, the box.com special remote is the only remote you have connecting your two computers.

But special remotes are only used to transfer the contents of files. They do not have the content of the git repository.

You need to set up a git remote that both computers can access. Or you can use the webapp to set up XMPP to sync between the computers directly. If you start the webapp, it will detect you're missing a git remote, and suggest setting up XMPP. [[/assistant/xmppnudge.png]]
"""]]


code:
[[!comment format=mdwn
 username="tomas"
 ip="188.167.111.235"
 subject="Problems syncing with box.com"
 date="2013-05-19T19:56:09Z"
 content="""
Thanks for the answer Joey.
My setup is really 2 PCs that are usually not turned on at the same time (usually) and are unable to connect directly (usually). I've dropped box.com because there are bugs. When I sync a file, I see a broken link sometimes. That happens even in situation when the two PCs see each other. Other repos work as expected. So I assume this is a box.com issue. I tried to remove the box repo from both sides, but it is still in 'cleaning out' state even after few days and a lot of daemon restarts. I also have a queue to box. If I delete the transfers, they would appear later. Any way of sorting that out?
I know that reporting bugs like this is not really efficient. What would be the best way to make it reproducible? E.g. using vagrant and two VMs?

"""]]


code: Are there any problems using submodules with git-annex? I have not tried it yet, I'm just asking.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 subject="comment 1"
 date="2012-07-05T17:04:34Z"
 content="""
I haven't tried it either, but I think it should work ok, as long as you bear in mind that to git-annex, each submodule will be treated as a separate git repository.
"""]]


code:
[[!comment format=mdwn
 username="http://mildred.fr/"
 nickname="Mildred"
 subject="submodules with git-annex in direct mode"
 date="2014-04-27T14:05:20Z"
 content="""
Does this work well even with direct mode ?
How is this managed in direct mode ?

Coming back from direct mode to indirect mode (required when I want to run git commands) I found that the .gitmodules file was an annex symlink (to .git/annex/objects/...). I just recovered from corrupt repository and may have made mistakes, but is this a risk under normal curcumstances ?
"""]]


code:
Hi,

On a linux server I created a (direct) git annex repository:

    $ ls -a
    .  ..  apps  attic  devel  games  .git  portable  tools

On a Windows client, I want to check it out:

    me@client /cygdrive/c/tmp
    $ git clone ssh://me@server/srv/data/bin bin
    Cloning into 'bin'...

But git annex checks only half of it out :-(

    me@client /cygdrive/c/tmp
    $ ls -a bin
    .  ..  .git  attic  games

Any ideas? Sorry, I am very new to git-annex (and git).

Thanks!


code:
[[!comment format=mdwn
 username="Xyem"
 ip="178.79.137.64"
 subject="comment 1"
 date="2014-05-08T12:50:58Z"
 content="""
What files are in the directories that are missing?
Are they committed into the repository?
"""]]


code:
[[!comment format=mdwn
 username="divB"
 ip="171.67.174.99"
 subject="comment 2"
 date="2014-05-09T01:02:22Z"
 content="""
Thanks.

As you can see everything except attic and games is missing.

As far as I understand, yes, they are committed. How can I check?
On the server I issued
    git annex add .

\"git annex status\" does not output anything. Remember: I use direct mode.
"""]]


code:
[[!comment format=mdwn
 username="divB"
 ip="171.67.174.99"
 subject="comment 3"
 date="2014-05-09T06:06:54Z"
 content="""
Finally I found it: https://git-annex.branchable.com/bugs/assistant_doesn__39__t_sync_empty_directories/

Empty directories.

Wow! I was not aware that git is actually a step BACKWARDS in this regard :-(

That was already a big problem half a century ago in CVS, finally fixed in SVN and git re-introduces the issue? Sad :-(

"""]]


code:
I'm trying to use git-annex to archive scientific data. I'm often dealing with large numbers of files, sometimes 10k or more. When I try to git-annex add these files I get this error:

    Stack space overflow: current size 8388608 bytes.
    Use `+RTS -Ksize' to increase it.

This is with the latest version of git-annex and a current version of git on OS 10.6.7. After this error occurs, I am unable to un-annex the files and I'm forced to recover from a backup.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 1"
 date="2011-04-05T07:27:46Z"
 content="""
Heh, cool, I was thinking throwing about 28million files at git-annex. Let me know how it goes, I suspect you have just run into a default limits OSX problem.

You probably just need to up some system limits (you will need to read the error messages that first appear) then do something like

<pre>
# this is really for the run time, you can set these settings in /etc/sysctl.conf
sudo sysctl -w kern.maxproc=2048
sudo sysctl -w kern.maxprocperuid=1024

# tell launchd about having higher limits
sudo echo \"limit maxfiles 1024 unlimited\" >> /etc/launchd.conf
sudo echo \"limit maxproc 1024 2048\" >> /etc/launchd.conf
</pre>

There are other system limits which you can check by doing a \"ulimit -a\", once you make the above changes, you will need to reboot to make the changes take affect. I am unsure if the above will help as it is an example of what I did on 10.6.6 a few months ago to fix some forking issues. From the error you got you will probably need to increase the stacksize to something bigger or even make it unlimited if you feel lucky, the default stacksize on OSX is 8192, try making it say 10times that size first and see what happens.
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 2"
 date="2011-04-05T17:46:03Z"
 content="""
This message comes from ghc's runtime memory manager. Apparently your ghc defaults to limiting the stack to 80 mb.
Mine seems to limit it slightly higher -- I have seen haskell programs successfully grow as large as 350 mb, although generally not intentionally. :)

Here's how to adjust the limit at runtime, obviously you'd want a larger number:

<pre>
# git-annex +RTS -K100 -RTS find
Stack space overflow: current size 100 bytes.
Use `+RTS -Ksize -RTS' to increase it.
</pre>

I've tried to avoid git-annex using quantities of memory that scale with the number of files in the repo, and I think in general successfully -- I run it on 32 mb and 128 mb machines, FWIW. There are some tricky cases, and haskell makes it easy to accidentally write code that uses much more memory than would be expected.

One well known case is `git annex unused`, which *has* to build a structure of every annexed file. I have been considering using a bloom filter or something to avoid that.

Another possible case is when running a command like `git annex add`, and passing it a lot of files/directories. Some code tries to preserve the order of your input after passing it through `git ls-files` (which destroys ordering), and to do so it needs to buffer both the input and the result in ram.

It's possible to build git-annex with memory profiling and generate some quite helpful profiling data. Edit the Makefile and add this to GHCFLAGS: `-prof -auto-all -caf-all -fforce-recomp` then when running git-annex, add the parameters: `+RTS -p -RTS` , and look for the git-annex.prof file.
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 3"
 date="2011-04-05T18:02:05Z"
 content="""
Oh, you'll need profiling builds of various haskell libraries to build with profiling support. If that's not easily accomplished, if you could show me the form of the command you're running, and also how git annex unannex fails, that would be helpful for investigating.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkhdKAhe3l_UyGt5SdfRBPYVwe-9f8P2dM"
 nickname="Justin"
 subject="comment 4"
 date="2011-04-05T21:14:12Z"
 content="""
@joey

OK, I'll try increasing the stack size and see if that helps.

For reference, I was running:

git annex add .

on a directory containing about 100k files spread over many nested subdirectories. I actually have more than a dozen projects like this that I plan to keep in git annex, possibly in separate repositories if necessary. I could probably tar the data and then archive that, but I like the idea of being able to see the structure of my data even though the contents of the files are on a different machine.

After the crash, running:

git annex unannex

does nothing and returns instantly. What exactly is 'git annex add' doing? I know that it's moving files into the key-value store and adding symlinks, but I don't know what else it does.

--Justin

If

"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 5"
 date="2011-04-07T16:41:00Z"
 content="""
I think what is happening with \"git annex unannex\" is that \"git annex add\" crashes before it can \"git add\" the symlinks. unannex only looks at files that \"git ls-files\" shows, and so files that are not added to git are not seen. So, this can be recovered from by looking at git status and manually adding the symlinks to git, and then unannex.

That also suggests that \"git annex add .\" has done something before crashing. That's consistent with you passing it < 2 parameters; it's not just running out of memory trying to expand and preserve order of its parameters (like it might if you ran \"git annex add experiment-1/ experiment-2/\")

I'm pretty sure I know where the space leak is now. git-annex builds up a queue of git commands, so that it can run git a minimum number of times. Currently, this queue is only flushed at the end. I had been meaning to work on having it flush the queue periodically to avoid it growing without bounds, and I will prioritize doing that.

(The only other thing that \"git annex add\" does is record location log information.)
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 6"
 date="2011-04-07T18:09:13Z"
 content="""
I've committed the queue flush improvements, so it will buffer up to 10240 git actions, and then flush the queue.

There may be other memory leaks at scale (besides the two I mentioned earlier), but this seems promising. I'm well into running `git annex add` on a half million files and it's using 18 mb ram and has flushed the queue several times. This run
will fail due to running out of inodes for the log files, not due to memory. :)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 7"
 date="2011-04-08T21:55:36Z"
 content="""
http://xfs.org/index.php/XFS_FAQ#Q:_Performance:_mkfs.xfs_-n_size.3D64k_option
"""]]


code:
I have a backups repository with a few files in it, that at one point had some huge filesets erroneously added to it.  As a result, even though there are only 23,334 annexed files, the number of non-dangling Git objects in the repository comes to 593,584.

Normally I would use `git filter-branch` to clear out the deadwood in situations like this, since it is a completely private repository.  What I'm wondering is, is any such thing possible with git-annex, or is the best option just to start over, copy all the files into the new repository, and then `git-add` them all?


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/2grhJvAC049fJnvALDXek.6MRZMTlg--#eec89"
 nickname="John"
 subject="comment 1"
 date="2013-08-29T07:23:13Z"
 content="""
Maybe one way to solve this that would be general is to have some kind of `prune-history` command, which keeps only the HEAD and drops everything else.  Because there are some repositories that I want to manage with `git-annex` for many reasons, but I don't care about keep history around at all.
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/2grhJvAC049fJnvALDXek.6MRZMTlg--#eec89"
 nickname="John"
 subject="comment 2"
 date="2013-08-30T06:18:42Z"
 content="""
This was answered quite thoroughly in:http://git-annex.branchable.com/forum/safely_dropping_git-annex_history/
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4830:1600:187::2"
 subject="comment 3"
 date="2013-09-04T06:36:15Z"
 content="""
`git annex forget` automates this now. Needs a version of git-annex supporting it installed on *all* the computers you use the repo on.
"""]]


code:
How could I delete and purge a remote? I want to remove all traces of it
but I can't find out how to achieve this.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 1"
 date="2013-11-05T16:17:22Z"
 content="""
In the git-annex webapp, each repository has a settings menu with \"delete repository\".
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkzwmw_zyMpZC9_J7ey--woeYPoZkAOgGw"
 nickname="dxtrish"
 subject="comment 2"
 date="2013-11-05T17:46:02Z"
 content="""
When I use that Delete button it seems to still have records of it
because when I delete the repo on the other end and create a new bare
one it complains about some UUID mismatch.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 3"
 date="2013-11-05T18:03:26Z"
 content="""
Hmm well, it's true that deleting a repository does not clean out any git remotes that other repositories might have configured using it, and the uuid of the old remote is cached in there. This is only a problem if you put back another repository in the same location as the old one. `git remote remove` should clean that up for you.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkzwmw_zyMpZC9_J7ey--woeYPoZkAOgGw"
 nickname="dxtrish"
 subject="comment 4"
 date="2013-11-06T14:24:21Z"
 content="""
That is exactly what I have wanted to do a couple of times now and 'git remote remove' does not seem to remove it all either.
When I try to create a new repository in the same place it still complains about that darn UUID :)

What is working is to manually add the new UUID to .git/config
"""]]


code: If I use git-annex with a centralized bare git repository as [described here](http://git-annex.branchable.com/tips/centralized_git_repository_tutorial/), will the Assistant automatically `git push` and `git pull` the master and git-annex branches? Or does it basically just do a `git annex sync`?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.252.11.120"
 subject="comment 1"
 date="2012-11-17T20:25:03Z"
 content="""
The assistant will push, pull, and merge the master and git-annex branches (which is exactly what git annex sync does).

Additionally, the assistant will transfer file contents to the bare repository, either to make it have a copy of everything, or as needed to transfer to other repositories that also use that bare repository. This behavior is configurable using the webapp.
"""]]


code:
[[!comment format=mdwn
 username="annexuser"
 ip="85.159.236.214"
 subject="comment 2"
 date="2012-11-17T20:44:53Z"
 content="""
When I make a change in the repository and run a plain old `git annex sync`, it pushes out to any other available annex, but not to my bare hub. If I run `git annex sync` a second time it tells me that I'm ahead of the hub:

    commit
    # On branch master
    # Your branch is ahead of 'hub/master' by 1 commit.
    #
    nothing to commit, working directory clean
    ok

In order to push the changes out to the hub I have to do `git push --all`.

The same thing with pulling. If I make a change on the annex, push it to the hub, and then go to a second annex that knows about the hub but not the first annex, a `git annex sync` does not pull the changes in from the hub. I have to do `git pull` to get them.

Do I have something setup incorrectly?
"""]]


code:
I want to setup my remote git repo on S3
Referred to
http://git-annex.branchable.com/tips/using_Amazon_S3/
http://git-annex.branchable.com/special_remotes/S3/

I need support for AWS.config param proxy_uri
http://docs.amazonwebservices.com/AWSRubySDK/latest/AWS.html

Can some one help?


code:
Hi,

Whenever I do a git annex vicfg (as pasted below) the long gone repo 40da403e-6f74-4705-aae0-433aa656b55e keeps automatically being added back into the configuration.  Worse in the webapp this manifests itself as a repository list item that's empty of text (and when buttons are clicked there's a complaint about missing UUIDs).

In addition to removing the lines via git-annex vicfg, I've switched my repository into indirect mode, manually checked out the the git-annex branch and deleted the offending key from the preferred content and group files (it oddly didn't appear in the uuid file).  However they keep coming back :-(

Everything seems to be okay bar this rogue entry which in the web-app irrationally bothers me.

I'm running the very latest git-annex release.

Any thoughts?

Cheers,

Matt.

    # git-annex configuration
    #
    # Changes saved to this file will be recorded in the git-annex branch.
    #
    # Lines in this file have the format:
    #   setting uuid = value

    # Repository trust configuration
    # (Valid trust levels: trusted semitrusted untrusted dead)
    # (for mattford63 (matt@descartes.rss.mhs.man.ac.uk:~/current))
    trust 88733342-45ef-420f-8f1b-9a5ed8d8e070 = semitrusted

    # Repository groups
    # (Standard groups: client transfer backup incrementalbackup smallarchive archive source manual public unwanted)
    # (Separate group names with spaces)
    # (for )
    group 40da403e-6f74-4705-aae0-433aa656b55e = client
    # (for mattford63 (matt@descartes.rss.mhs.man.ac.uk:~/current))
    group 88733342-45ef-420f-8f1b-9a5ed8d8e070 = client

    # Repository preferred contents
    # (for )
    content 40da403e-6f74-4705-aae0-433aa656b55e = standard
    # (for mattford63 (matt@descartes.rss.mhs.man.ac.uk:~/current))
    content 88733342-45ef-420f-8f1b-9a5ed8d8e070 = standard


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.193"
 subject="comment 1"
 date="2013-07-02T16:56:42Z"
 content="""
Run: `git annex dead 40da403e-6f74-4705-aae0-433aa656b55e`
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnXybLxkPMYpP3yw4b_I6IdC3cKTD-xEdU"
 nickname="Matt"
 subject="comment 2"
 date="2013-07-02T19:48:19Z"
 content="""
Hi,

I should have mentioned I can't do this either:

    ~/current $ git annex dead 40da403e-6f74-4705-aae0-433aa656b55e
    dead 40da403e-6f74-4705-aae0-433aa656b55e git-annex: there is no available git remote named \"40da403e-6f74-4705-aae0-433aa656b55e\"

I can't quite remember how I removed the remote - it was a whilst back, I think maybe I just edited it out of the git config file....
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.193"
 subject="comment 3"
 date="2013-07-02T19:58:30Z"
 content="""
It seems that the uuid is present in the group.log and preferred-content.log files in the git-annex branch, but not in the uuid.log file. I missed that in your description before. That's weird, don't know how that could happen.

However, when I manually configure things this way, I do not see the repository in the webapp. Which makes sense; uuid.log is the canonical list of repositories.

I guess if I were you I'd manually edit the uuid into uuid.log and then mark it dead.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnXybLxkPMYpP3yw4b_I6IdC3cKTD-xEdU"
 nickname="Matt"
 subject="comment 4"
 date="2013-07-02T20:45:29Z"
 content="""
I tried as you suggested.  It worked as far in so far that I was able to mark the repository dead.  However, I still see an empty line in the web app so perhaps this odd repo isn't the cause (but I'm deeply suspicious of it).  Looking at the HTML source the line in the web-app does not have any UUID at all (see the screen shot [here](http://dancingfrog.hopto.org/~matt/git-annex.png)).

What causes the code to add back a previously deleted repository (I vaguely remember something about this being due to the merge algorithm...and why one can only mark repos as deleted)?  Why would it only add back to the group and trust logs and not the UUID?  The code that generates the web-app HTML must use something else or things in addition to the UUID log?

The logs don't really say much - how can I help debug this further?
"""]]


code:
With git it is easy to create one repository per project, and it almost always makes sense to do so.  When using git-annex, what is the recommended setup?

Should one have a single annex containing all files, or is it recommended to create different repositories for things like 'photos', 'music', 'isos' ?


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="depends ..."
 date="2011-11-04T19:59:24Z"
 content="""
It makes sense to have separate repositories when you have well-defined uses for them.

I have a separate repository just for music and podcasts, which I can put various places where I have no need of the overhead of a tree of other files.

If you're using it for whatever arbitrary large files you accumulate, I find it's useful to have them in one repository. This way I can rearrange things as makes sense. It might make sense to have \"photos\" and \"isos\" as categories today, but next year you might prefer to move those under 2011/{photos,isos}. It would certainly make sense to have different repositories for home, work, etc.

How to split repositories up for a home directory is a general problem that the [vcs-home](http://vcs-home.branchable.com)
project has surely considered at one time or another.
"""]]


code:
I have a directory with 6TB of data in it. I tried to use git annex to back it up to three 3TB drives, I didn't want to use RAID as it sucks, and I didn't want to use tar as I wanted my files easily available.

I added my remotes successfully, then I ran ``git annex add .``

That mostly worked, although it understandably took ages, although it missed several GB of files here and there.

Next I tried to do ``git commit -a -m added``, hoping that this would copy all of my files to the remotes.  It didn't it just died with the error

    fatal: No HEAD commit to compare with (yet)
    fatal: No HEAD commit to compare with (yet)
    Stack space overflow: current size 8388608 bytes.
    Use `+RTS -Ksize -RTS' to increase it.

So I freaked out and decided to undo the mess and just go with tar instead, since at this point every git command takes multiple minutes and fails with the same error as above.

I tried to run ``git anne unannex .``, but I got this error:

``
unannex GWAS/by-download-number/27081.log.gz fatal: No HEAD commit to compare with (yet)
``

So now I can't do anything without committing the files it seems, and I somehow need to grow the git cache, although when I search online for `+RTS -Ksize -RTS', I get nothing.

Does anyone know how to increase the cache size, or how to unannex the files without this HEAD error?

Thanks,

Mike


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmwjQzWgiD7_I3zw-_91rMRf_6qoThupis"
 nickname="Mike"
 subject="comment 10"
 date="2014-06-18T20:15:42Z"
 content="""
``git ls-files --cached | wc -l`` returns: 1882028

As far as I can tell, the largest objects in .git/objects are 65kb, there are just a bunch of them (257). Also, my repo contains 1,886,125 files and directories total, most in a single directory (after git annex add completed, that one directory contained 8.3GB of symlinks.

``git-annex add .`` just completed successfully, I am now running ``git commit -a -m added``, it is chugging away and taking its time.

Is there an obvious upper limit to the number of files or the total size of files that git annex can handle? For example, is 1 million files too many? How about 6TB? or 9TB? For this repo I think I have a little less than 2m files, and the total size of the repo is greater than 6TB. Is that too much? Should I split it into multiple repos?

I also have a question just about the utility of git-annex for this purpose. I don't need to backup this data, I just want to have it off the big hard drive and onto a bunch of small drives. I have added 3 4TB drives as remotes and I want all of the data stored on them, I will take them offline and put them in a safe. Ideally my file and directory structure will remain intact as symlinks, and then when I want to access a file in the future, I can run ``git annex get <file>``, connect the drive that git annex tells me to, and then get that file, use it, and then drop it when I am done. From all of my reading it seems like that is a good usage for git annex, but I want to check with you and see if that makes sense to you. Also, can I just run ``git annex drop --auto --numcopies=1``  to get git annex to move all of the files to my remote repositories?

Thanks for all of your help, and let me know if there are any other debug steps you would like me to run. I am still waiting for git commit to run, and for an exact repo size for you.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.203"
 subject="comment 11"
 date="2014-06-18T20:54:31Z"
 content="""
Ok, my suspicion is that the root problem is having a large number of file in a single directory. That would cause the git tree objects to get big, and it may
be that git-annex somewhere buffers a whole tree object in memory, although I cannot think of where off the top of my head.

git-annex scales to any size of files (limited only by checksumming time unless you use the WORM backend to avoid checksumming). git-annex tries to scale at least as good as git does to a large quantity of files in a repository. git doesn't handle a million files in a repository very fast, due to a number of issues including how the index works. I have never tested git-annex with more than 1 million files, and not all in the same directory.

Other than the number of files, your use case seems reasonable. `git annex drop` will drop files that you have already copied to enough of the remotes (using eg, `git-annex copy`).

Above you show a git-annex add failing after 5 files. I suspect you truncated that output, and it processed rather more files. git-annex only says \"(Recording state in git...)\" once it's added all the files, or after it's added around 10 thousand files and still has more to do). It seems to have failed at the point where the files are staged into the index.

I'm building a 2 million file in one directory repo on a fast server now to see if I can reproduce this.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmwjQzWgiD7_I3zw-_91rMRf_6qoThupis"
 nickname="Mike"
 subject="comment 12"
 date="2014-06-18T22:28:27Z"
 content="""
Yes, you are right. ``git-annex add`` got through almost all of the files in the first run, which I did a week ago, I am not sure how long it took, several days I think (which is fine, time isn't that important here).

I re-ran ``git-annex add .`` yesterday after having trouble with ``git commit``, which is when I uncovered these problems. I think you are right that the problem appears when git-annex is staging files into the index. No problems occurred during checksumming or moving files.

Also, the repo isn't as large as I thought, it if 4.1TB, so it makes sense that the issue is number of files, not files size.

``git add`` and ``git commit`` are now working fine, all git operations (e.g. ``git status``) are now taking around 30s to 1 min, which is acceptable.

I am going to try and move the data to the remotes now. Is there anything special I need to do since the remotes are smaller than the current repo? The remotes are just single drives with ext4 filesystems and an empty repo on them. I ideally want to fill each drive as much as possible, and have the current repo contain no files, how do I do that? Can I just run ``git-annex move --to mito_backup1`` and then when it is full run a second command of ``git-annex move --to mito_backup2``.  Is it better to use ``git-annex copy`` instead of ``move`` and then use ``drop`` after?

Thanks!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.203"
 subject="comment 13"
 date="2014-06-18T22:43:08Z"
 content="""
It will be faster to use `git annex move`, assuming you want to only have 1 copy of each file, and not more. git-annex will stop storing files on a drive once it gets close to full (annex.diskreserve), and you can safely interrupt it and switch to the next drive.

Do you have any special git configuration? In particular I'm curious about any annex.queuesize setting, which if set to something really high would make `git annex add` buffer a lot of filenames and stage them all at once. (However, I just noticed that annex.queuesize didn't cause as large a queue to be used as intended, so it would need to have been set to some really enormous value to run it out of stack space.)

Also, see [[scalability]].
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmwjQzWgiD7_I3zw-_91rMRf_6qoThupis"
 nickname="Mike"
 subject="comment 14"
 date="2014-06-18T22:51:28Z"
 content="""
As far as I know I am using the defaults, I didn't customize anything. I am thinking of switching to the WORM backend though as I think it will make things a little faster, but I haven't done that yet.

Also, I actually compiled cabal-install with the ghc flag ``-rtsopts`` and git-annex with flags ``-rtsopts -with-rtsopts=-K1000m``. Due to the amount of memory available, I am not worried if git-annex leaks memory and uses 1GB of memory during operations, but I have been watching it with htop, and its memory usage is usually very small.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmwjQzWgiD7_I3zw-_91rMRf_6qoThupis"
 nickname="Mike"
 subject="comment 15"
 date="2014-06-19T21:19:01Z"
 content="""
Everything seems to be working fine now, but I have another question:

Is there any way to speed up the copying of many small files? It looks like git-annex is calling rsync for each individual file, which is very fast for large files, but on my directories with many small files, the total speed is working on to just a few MB an hour - it has only transferred 1GB in the last 4 hours.

I am using the WORM backed with the ``-b WORM`` flag, but I wonder if there is a different move method implemented? For example many calls to ``mv`` will be much faster than many calls to ``rsync``.

Thanks!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.203"
 subject="comment 16"
 date="2014-06-19T22:16:21Z"
 content="""
git-annex could be made to always use cp for local transfers, see Remote/Git.hs rsyncOrCopyFile and change `ifM (sameDeviceIds src dest) (docopy, dorsync)` to just `docopy`

However, I doubt that will be a significant speedup. It's more likely that the overhead around copying a file and updating the location tracking etc adds up with millions of small files.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmwjQzWgiD7_I3zw-_91rMRf_6qoThupis"
 nickname="Mike"
 subject="comment 17"
 date="2014-06-21T01:56:48Z"
 content="""
Would git-annex be able to detect if I manually moved some files? At this point it looks like the transfer will take multiple weeks... if I just moved the files from .git/annex directly from one repo to the other I could make the transfer significantly faster, but would that corrupt the repo?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.55"
 subject="comment 18"
 date="2014-07-04T18:03:54Z"
 content="""
You can manually move files and use `git annex fsck`, but it is not likely to be any faster.

---

After letting a 2 million file import run while I was away on vacation, I came back to it, and it indeed ran out of memory:

<pre>
add 999996 ok
add 999997 ok
(Recording state in git...)
[2014-06-21 11:49:28 JEST] feed: xargs [\"-0\",\"git\",\"--git-dir=/home/joey/tmp/r/.git\",\"--work-tree=/home/joey/tmp/r\",\"add\",\"--\"]
add 999998 ok
add 999999 ok
[2014-06-21 11:49:49 JEST] read: git [\"--git-dir=/home/joey/tmp/r/.git\",\"--work-tree=/home/joey/tmp/r\",\"diff\",\"--name-only\",\"--diff-filter=T\",\"-z\",\"--\",\".\"]
(Recording state in git...)
[2014-06-21 11:52:24 JEST] feed: xargs [\"-0\",\"git\",\"--git-dir=/home/joey/tmp/r/.git\",\"--work-tree=/home/joey/tmp/r\",\"add\",\"--\"]
Stack space overflow: current size 8388608 bytes.
Use `+RTS -Ksize -RTS' to increase it.
</pre>
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.55"
 subject="comment 19"
 date="2014-07-04T18:18:41Z"
 content="""
[[bugs/runs_of_of_memory_adding_2_million_files]]
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.203"
 subject="comment 1"
 date="2014-06-18T16:22:13Z"
 content="""
What version of git-annex are you using? And what version of git? Is your git-annex repository in direct mode or indirect mode?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmwjQzWgiD7_I3zw-_91rMRf_6qoThupis"
 nickname="Mike"
 subject="Frustrating"
 date="2014-06-18T16:29:31Z"
 content="""
OK, this is a little frustrating. I found this post from three years ago: [[http://git-annex.branchable.com/forum/Problems_with_large_numbers_of_files/]] and I decided to try a newer version of git-annex.

I uninstalled ghc and haskell from Scientific Linux because all of these Red Hat based distros have ancient packages.

I installed the latest git from source, the latest ghc linux x86_64 binary, and then the latest haskell platform from source. Then I used cabal to install all dependencies for git annex with ``cabal install --only-dependencies git-annex``. Finally I installed git annex from source.

I then tried to run ``git-annex add .`` in my directory and got the following error:

    #> git annex add .
    add AllStudies.txt.csv.gz ok
    add GWAS/action.txt.gz ok
    add GWAS/archive/dump2.txt.gz ok
    add GWAS/archive/dump3.txt.gz ok
    add GWAS/by-download-number/27080.log.gz ok
    (Recording state in git...)
    Stack space overflow: current size 8388608 bytes.
    Use `+RTS -Ksize -RTS' to increase it.

Ok, I was hoping that the latest version would just work, no luck. So I did what it told me to:

``git-annex +RTS -K1000000 -RTS add .``

That gave the error:

``
    git-annex: Most RTS options are disabled. Link with -rtsopts to enable them.
``

Grr.

So I went into the Makefile and added the line ``-rtsopts -with-rtsopts=\"-K1000m\"`` after every call to ghc I could find. I also added ``ghc-options: -with-rtsopts=\"-K100000\"`` to my ~/.cabal/config file.

Now when I run ``make`` I get this error:

    if [ \"cabal \" = ./Setup ]; then ghc --make Setup; fi
    cabal  configure
    cabal: Most RTS options are disabled. Link with -rtsopts to enable them.
    make: *** [Build/SysConfig.hs] Error 1

Do I have to manually compile the entire Haskell Platform with the -rtsopts flag in order to get this to work?

I can't find any easy-to-follow information anywhere that shows me how to just increase the memory limit. My server has 48 cores, 192GB of memory, over 1TB of scratch space, and over 60TB of storage. I really want to be able to use git-annex to easily move files from our large RAID arrays onto archive drives, and be able to intelligently get that data back whenever I want, I don't understand why I am being limited to 8MB of memory for this.

Any advice would be fantastic, thank you.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmwjQzWgiD7_I3zw-_91rMRf_6qoThupis"
 nickname="Mike"
 subject="Version"
 date="2014-06-18T16:32:32Z"
 content="""
Hi Joeyh,

Thanks for the reply. I am using git version 2.0.0.390.gcb682f8, not sure what version of git-annex, but I downloaded it from github about 20 minutes ago.

Thanks!

-Mike
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.203"
 subject="comment 4"
 date="2014-06-18T16:35:29Z"
 content="""
You can find out the version of git-annex by running: git-annex version

You can find out if your repository is in direct or indirect mode by running: git config annex.direct
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmwjQzWgiD7_I3zw-_91rMRf_6qoThupis"
 nickname="Mike"
 subject="Versions"
 date="2014-06-18T16:40:14Z"
 content="""
``git-annex version`` returns:

    git-annex version: 5.20140618-gc2f1c63
    build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus DesktopNotify XMPP DNS Feeds Quvi TDFA CryptoHash
    key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
    remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar hook external
    local repository version: unknown
    supported repository version: 5
    upgrade supported from repository versions: 0 1 2 4

``git config annex.direct`` exits with error code 1 and doesn't return any information, however I never explicitly set direct mode, and the repository is all symlinked, so my assumption is that it is in indirect mode. Would direct mode be better for such a large repo?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmwjQzWgiD7_I3zw-_91rMRf_6qoThupis"
 nickname="Mike"
 subject="Reinstall GHC or Cabal?"
 date="2014-06-18T16:49:35Z"
 content="""
Do I need to reinstall ghc or cabal with rtsopts enabled somehow in order to be able to compile git-annex with -K1000m?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.203"
 subject="comment 7"
 date="2014-06-18T17:14:40Z"
 content="""
Ok, so the repository is in indirect mode, and this rules out a large quantity of problems that could have been caused by direct mode (no, I don't recommend using direct mode).

If you want to build git-annex with the +RTS option enabled, you just need to pass -rtsopts to ghc when building git-annex. (Not -with-rtsopts ...)
That *might* let you pump up the memory and bypass whatever the problem is, or at least find out how much memory it's trying to allocate, which might be a useful clue. But I would be much more interested in debugging and fixing the actual problem, since git-annex should not normally need to allocate a 8+ mb chunk of memory.

The  \"No HEAD commit to compare with (yet)\" failure mode was removed from git in 2011. You must have been using old versions of git and git-annex before you upgraded. Perhaps they have left the repository in some broken state.

What size does `du -hsc .git/objects` report? How about `du -h .git/index`?

Are git commands that do not involve git-annex still taking a long time to run or failing in some way? (Note that `git commit` has a hook that runs git-annex; you can bypass that with `git commit --no-verify`)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmwjQzWgiD7_I3zw-_91rMRf_6qoThupis"
 nickname="Mike"
 subject="comment 8"
 date="2014-06-18T17:31:58Z"
 content="""
The git available through yum is git 1.7.1, which looks like it was released in 2010 or earlier. (I really wish I had a different version of linux on this server). It is possible that in some way screwed up the repo.

I figured out how to compile cabal and git-annex with rtsopts, so I can now set higher memory levels, but I am happy to help debug the problem too, as I would really love a fully functional git-annex.

git commands now run quickly, thanks to the new git I think.

``du -hsc .git/objects`` returns: ``8.1G    .git/objects``

``du -h .git/index`` returns: ``437M    .git/index``

I am currently running the command ``git-annex +RTS -K1000m -RTS add .``, it is chugging away doing something, but is not printing any messages yet after 11 minutes of running, it is a 6TB directory though, and there are a lot of concurrent IO operations on that disk right now.

I am also running ``du -h --max-depth=1`` on the root repo directory, and also ``find | wc -l``, so that I can tell you the exact size of the dir and the total number of files too.  These operations combined may take more than an hour though, I will send details when the commands complete.

Let me know if you want me to stop the ``git-annex +RTS -K1000m -RTS add .`` command and run git-annex some other way.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.203"
 subject="comment 9"
 date="2014-06-18T17:36:44Z"
 content="""
Both of those du's look extremely large. How many files are listed by `git ls-files --cached | wc -l` ?

I don't think that there's any point in running `git annex add` while you're still having some problem. I am curious though how much memory the git-annex add you have running has used.

If I were you, I'd look in .git/objects for large files (> 100kb, say).
"""]]


code: There are some settings (rsync bandwidth settings are my use case) I'd like to reload after assistant starts, possibly mid-sync. Does assistant support SIGHUP, etc? What's the correct way to change bandwidth settings?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="24.159.78.125"
 subject="comment 1"
 date="2014-07-23T15:02:04Z"
 content="""
Assistant does not support SIGHUP. There is a menu item (top right menu) that can restart it.
"""]]


code: I have around 70 GBs of data spread around 4 repositories on a Linux (Ubuntu) box. My problem is I need to reformat the drive they are on. I would like to move them to an external usb drive temporarily during reinstall then move them back to their original location. When I started with annex I did try to mv annexFolder/ toNewLoc/ which failed leaving behind a corrupt repo. What is a safe way to move an annex folder? My primary connection is a 3G modem so I am trying to avoid re downloading everything. Another thing I am trying to avoid is cloning the repos to the external drive, reinstall, clone it back to the internal drive and mark original and external repos as dead, but AFAIK those dead repos will show up in the output of whereis, so everytime I reformat I gonna have two extra dead repos.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
 nickname="Justin"
 subject="comment 1"
 date="2013-08-16T03:42:00Z"
 content="""
moving the annex should work fine, provided:

* You are moving it to a proper unix filesystem - NOT fat32
* The assistant is shutdown. If it is running, bad things will happen.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.63"
 subject="comment 2"
 date="2013-08-24T18:56:11Z"
 content="""
`git annex whereis` does not show dead repositories.

Anyway Justin is of course right: Provided the assistant is not running in a repository, the repository is just a collection of files in a directory, and can be moved around, tarred up, untarred, etc just like any other repository. If the assistant is running it may become unhappy if its repository vanishes out from underneath it.
"""]]


code:
Hi, I'm using git-annex 5.2014012 with the webapp and I've added two repositories: a remote server and a box.com account. When I add a file it is uploaded properly to both remotes as you would expect.

The problem is that I would like the files to only be uploaded to box.com, and the remote server only store the git repository. Is there any way of saying to git-annex to not sync files to a remote server?


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/FHnTlSBo1eCGJRwueeKeB6.RCaPbGMPr5jxx8A--#ce0d8"
 nickname="Hamza"
 subject="comment 1"
 date="2014-02-06T13:30:33Z"
 content="""
You can tell git-annex to ignore the repo (assuming git repo is called origin)

     git config remote.origin.annex-ignore true
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawk9nck8WX8-ADF3Fdh5vFo4Qrw1I_bJcR8"
 nickname="Jon Ander"
 subject="comment 2"
 date="2014-02-07T11:36:32Z"
 content="""
Thanks, that seams to be exactly what I need.
Any change of adding this option to the webapp?
"""]]


code:
Hi,

I've removed some large files with git remove, but seem to be unable to remove the corresponding annex content.

Example:

kheymann@corax:~/annex$ find -name "*s24576--10daa3d9007edad720dc057e4272a9c6cda930bef34a83e3bc1d93f1c55b9cac"

./.git/annex/objects/jF/j7/SHA256-s24576--10daa3d9007edad720dc057e4272a9c6cda930bef34a83e3bc1d93f1c55b9cac

./.git/annex/objects/jF/j7/SHA256-s24576--10daa3d9007edad720dc057e4272a9c6cda930bef34a83e3bc1d93f1c55b9cac/SHA256-s24576--10daa3d9007edad720dc057e4272a9c6cda930bef34a83e3bc1d93f1c55b9cac

kheymann@corax:~/annex$ git annex dropkey -vvv --backend SHA256 s24576--10daa3d9007edad720dc057e4272a9c6cda930bef34a83e3bc1d93f1c55b9cac

No output but the files remain in the annex. Git annex fsck and git annex unused run without listing files to be removes. What can I do apart from deleting the files manually from the annex?

Some info:

    kheymann@corax:~/annex$ git annex version
    git-annex version: 3.20121017
    local repository version: 3
    default repository version: 3
    supported repository versions: 3
    upgrade supported from repository versions: 0 1 2

Any hints?

Best,
Karsten


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn7gQ1zZDdWhXy9H51W2krZYShNmKL3qfM"
 nickname="Karsten"
 subject="comment 1"
 date="2012-11-09T19:37:24Z"
 content="""
Please ignore my previous post, by following the instructions in the comment in http://git-annex.branchable.com/walkthrough/unused_data/ I was able to get rid of the data.
"""]]


code:
Hi,

I'm trying to figure out if it's possible to have a remote that allows access from _clients without git-annex_. That is a remote type that presents itself as a client repo. I'm probably missing something obvious.

Ideally I'd like to use my Nokia N900 phone as a ssh remote from which I can open files on any computer when the device is connected via usb in mass storage mode. This would allow me to access my files when I'm using a computer at work or at a friends house without installing git-annex. The phone has an ext3 partition so links are possible. Naturally I'd have to ensure the files I want are on the device before I leave the house.

I'd also like to be able to scp files from my ssh remote when on git-annex less devices. Unfortunately the ssh/rsync repo presents no links or normal filenames.

Git annex, and the assistant is working really smoothly between my local computers and my server on the internet. Setting things up with the webapp is super simple. Great work!


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="99.54.57.201"
 subject="comment 1"
 date="2013-01-20T20:20:06Z"
 content="""
This is off the top of my head, so I may be wrong on details:

An ssh remote, or a USB remote, when you create it with the assistant, is a bare repo.  The files in it are not visible or accessible except through a clone of it (such as your client repo).

You *could* make an SSH remote or USB remote as a *non-bare repo*, in which case the files within it would be visible and accessible just like on a client repo.  You could set that up using the command line; the assistant doesn't do it, but once you got it set up on the command line the assistant would use it just like normal.

There is a catch though.  Though potentially available, the files in that repo don't actually show up until you run \"git annex sync\" in that repo.  Because git annex does not update the checked out branch of a remote.  So you'd have to cd into that repo manually and run \"git annex sync\" to make the latest-and-greatest files available.

Here's how I'd do this...

If the git annex assistant is running, turn it off.

follow the instructions here: http://git-annex.branchable.com/walkthrough/  under \"adding a remote\" on your usb drive.  You now have a non-bare remote.

Fire up the assistant and you'll see that that is now one of your remotes.  Configure it to \"backup\" or whatever you want.  Let the assistant fill it up with content.

When you want to take it someplace, get on the command line and \"cd /wherever/my/usb/drive/is/annex\" and then do: \"git annex sync\" -- that will update the links to the content so you can actually see it.

You now have all your content available on your USB drive.

You can do a similar process to manually create a non-bare SSH remote, and you can ssh in and run \"git annex sync\" to update its links and make the content available, which you can then retrieve va scp or sftp, or for that matter, you could configure a web server or WebDAV to serve it up if you like.

The only complication is running git-annex sync to update your links.

Again, this is all off the top of my head, I haven't looked up or tested anything here so I welcome correction.  But I think what you want to do can be pretty straightforward if you're willing to go behind Assistant's back to do the setup work, and run \"git annex sync\" when you need to.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn7gQ1zZDdWhXy9H51W2krZYShNmKL3qfM"
 nickname="Karsten"
 subject="comment 2"
 date="2013-01-21T07:02:17Z"
 content="""
An alternative to running git annex sync in the non-bare remote is running an assistant there.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawk3HGoDpnOPob5jOjvIootmkve1-nCpRiI"
 nickname="Kalle"
 subject="Thanks for the help"
 date="2013-01-21T20:40:39Z"
 content="""
I take it all the above requires mounting an sshfs filsystem of the remote client to run the git-annex or the assistant \"locally\" on the desktop or laptop. Or am I missing something.

For the usb usecase a normal mount would suffice of course.

The lack of automation means running a rsync script from (or to) the phone is easier. Not as clever though :( and means running a separate \"system\" for distributing files to the phone.

Thanks again!

"""]]


code:
Greetings.  Here is my desired usage scenario:

I have setup a server and three laptops which I am syncing over ssh. Each laptop may access the server, but because different people are using each of the laptops, they may not sync between laptops for security/privacy.

Each of the laptops will be adding content to the repository, and each of the laptops will be using the new content that the others have added.

Currently, in order to make all content available to all users, I'm having to ssh into the server and use `git annex get .` every time new content is added from one of the laptops, which is a pain. Is there a way to require a local copy of all content in a given repository so that is pulled/pushed automatically? I would like for each laptop to be able to add new content to their copies of the repository and sync with the server, after which each other laptop can access the content through the server.

Thanks!


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 1"
 date="2014-07-21T19:46:44Z"
 content="""
This is generally handled by using either `git annex sync --content`, which will make the laptops upload new files to the server before they push the git repository to it, or by using the git-annex assistant (on the laptops or perhaps on the server) to automatically copy files. Either way, you configure [[preferred_content]] settings to control which repositories want which  files.
"""]]


code:
Just two very simple questions:

1.) Is there a way to restrict the SSH key for git annex by supplying a command= ? Even better, is it also possible to supply a directory in which the repository is? (I do not want chroot - too complicated but a soft check would be sufficient for me).

2.) Can I tell git and git-ssh which pubkey to use WITHOUT changing system-/user wide config (e.g.., .ssh/*)? If it is indeed not possible, what's the best way to do it in Windows?

Thanks!


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="ooooh"
 date="2014-07-16T22:21:36Z"
 content="""
Thank *you* for taking the time to figure that out!

I agree on moving the .vbs files. I have done so. (Well, you have to run the git-annex-uninstall.exe to remove the old ones, but that shouldn't matter.)
Testing of an autobuild would be appreciated.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 1"
 date="2014-07-10T19:41:54Z"
 content="""
1. Yes, use [[git-annex-shell]].

2. The best way to do this is to use a dummy hostname in the git url for the remote. Then in .ssh/config, you can add a Host stanza that sets the real Hostname and also specifies the IdentityFile to use for that host.

Incidentially, the git-annex webapp takes care of both of these things for you automatically when setting up a remote on a ssh server.
"""]]


code:
[[!comment format=mdwn
 username="divB"
 ip="171.67.173.91"
 subject="comment 2"
 date="2014-07-11T21:43:11Z"
 content="""
Hey Joey,

Cool, that's great! Thanks!

As for the client side. I assume you probably don't know too much about the git setup on Windows ... but maybe (or someone else has an idea :)

1.) The ssh client config for git is \"c:\Program Files (x86)\Git\etc\ssh\ssh_config\" which is *very* unhandy since it's system wide! If it would at least be in the user profile ... (as mentioned, within the \".git\" directory of the repository would be the best)

2.) Even if I create a \"Host\" and use \"IdentityFile\", ssh still queries the agent! This is absolutely unwanted and slows down things! I have a particular key \"git-annex.key\" and *only* this should be tried - no agent. The reason why the agent is especially problematic for me is that I use a special agent which \"locks\" itself after some inactivity and requires to re-enter the passwords.

Thank you!

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 3"
 date="2014-07-14T20:20:28Z"
 content="""
Re the ssh config on Windows, my windows VM has a per-user .ssh/config inside c:\Documents and Settings\$user. This is where the assistant stores configurations so I know it works.

The soution to ssh still trying to use the agent is probably to set \"IdentitiesOnly yes\" in the stanza for a host. This is what the assistant does, anyway.
"""]]


code:
[[!comment format=mdwn
 username="divB"
 ip="128.12.90.218"
 subject="comment 4"
 date="2014-07-15T03:11:18Z"
 content="""
Hi Joey,

Sorry that I bother so much with this.
Thank you so much for your answer. At least with the command line it works now. I had it in %USERPROFILE%/.ssh/ssh_config (where I think it's supposed to be) rather than %USERPROFILE%/.ssh/config. In this file I have a stanza \"Host annex\" with Hostname, Port and IdentityFile set. When I call \"ssh annex\" from the command line everything works. It seems that it also works when I use e.g. \"git annex sync\" from the command line.

However, if I use the webapp, the daemon.log is full of:

    Please make sure you have the correct access rights
    and the repository exists.
    ssh: Could not resolve hostname annex: hostname nor servname provided, or not known
    fatal: Could not read from remote repository.

Is it possible that the assistant ignores the ssh config or does something differently?

Thanks again!

"""]]


code:
[[!comment format=mdwn
 username="divB"
 ip="128.12.90.218"
 subject="comment 5"
 date="2014-07-15T04:03:16Z"
 content="""
After some debugging I found another weird thing which is I think the reason. If I execute
    git annex get file.jpg

everything works. But if I do

    \"c:\program files (x86)\git\bin\git.exe\" annex get file.jpg

it fails, claiming it can't connect to host \"annex\". I found that there are 3 (!) git.exe installed. When I choose

    \"c:\program files (x86)\git\cmd\git.exe\" annex get file.jpg

it works again. When I use \"which git\" in cygwin, it also points me to the \"cmd/git.exe\" version.
So I think this is a bug and git annex assistant should call the executeable in \"cmd\" rather than \"bin\".
Why are there three versions of git.exe at all (one more is in directory libexec/git-core)?

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 6"
 date="2014-07-15T19:20:13Z"
 content="""
msgit seems to install it in both places, I am not sure why. Either one works ok when I try to use them. The msgit installer puts \"c:\program files (x86)\git\cmd\\" into PATH so I assume that's the one you're supposed to use.
"""]]


code:
[[!comment format=mdwn
 username="divB"
 ip="128.12.90.218"
 subject="comment 7"
 date="2014-07-15T20:33:07Z"
 content="""
Thanks Joey,

The problem is more that git-annex assistant takes the wrong (which is in \"bin\" rather than \"cmd\"). I think this is a bug. Because this way the connection does not work in git-annex assistant the same way it does not work with the version in \"bin\" ...

I think when git-annex assistant just calls the git.exe from path (which should be cmd/git.exe) then it should work.

Regards
Niki

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 8"
 date="2014-07-16T18:10:09Z"
 content="""
git-annex only ever runs git from PATH; I never hardcode paths to programs.

You can verify this by running it with the --debug flag to see the exact commands it runs.
"""]]


code:
[[!comment format=mdwn
 username="divB"
 ip="204.17.143.10"
 subject="comment 9"
 date="2014-07-16T20:54:54Z"
 content="""
Hi Joey,

Thanks for taking care about all these Windows troubles (Windows and POSIX is unfortunately a big mess).

I finally found the issue now and maybe the bug is related to msysgit. I guess you don't know too much about it so I will report to the msysgit folks.

However, there is still an (easy to fix) \"bug\" related to git-annex I think. I shortly describe the issue and a possible fix:

1.) As mentioned above, from the 3 git.exe, only the one in \"cmd\" should be called! It seems to be a wrapper for the \"bin\"-version. If the git.exe from \"bin\" is called something with the environment is wrong (e.g., ssh_config can not be found)

2.) cmd/git.exe is in %PATH% so usually no problem

3.) However, git-annex-autostart.vbs is in \"bin\" folder. Therefore, when called from there PWD is the \"bin\" folder and when calling \"git.exe\" without absolute path, this overwrites %PATH% because it's the current directory (of course, such behavior does not appear on UNIX).

4.) Now the git-annex assistant daemon always calls the wrong git.exe resulting in a broken config

Short term fix for users: Create a shortcut to git-annex-autostart.vbs and change the working directory to anything else

Long term fix for git-annex option 1: Do a chdir in the vbs file before calling git. This is not so good because where to?

Long term fix for git-annex option 2: Just place the vbs files in the parent directory (where \"Git Bash.vbs\" is). This looks like the cleanest solution to me.

"""]]


code:
Is there a way to restrict git-annex-shell to a specific directory?
Currently, if git-annex is paired to a remote repository, it adds this to the authorized_keys:

    $ cat ~/.ssh/authorized_keys
    command="~/.ssh/git-annex-shell",no-agent-forwarding,no-port-forwarding,no-X11-forwarding ssh-rsa AAAAB3...

    $ cat ~/.ssh/git-annex-shell
    #!/bin/sh
    set -e
    exec git-annex-shell -c "$SSH_ORIGINAL_COMMAND"

That gives whoever has the pubkey the right to access all repositories of one user.
It would be nice to have a manual way to limit the access to a specific repository like

    $ cat ~/.ssh/git-annex-shell
    #!/bin/sh
    set -e
    export GIT_ANNEX_SHELL_REPO=~/annex
    exec git-annex-shell -c "$SSH_ORIGINAL_COMMAND"

Or maybe some chroot hackery is the way to go?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.194"
 subject="comment 1"
 date="2012-11-05T16:40:39Z"
 content="""
I've added a `GIT_ANNEX_SHELL_DIRECTORY` environment variable, that, if set, prevents git-annex-shell from operating on any other directory.

I've made the assistant include that setting in `authorized_keys` that it sets up. For example:

    command=\"GIT_ANNEX_SHELL_DIRECTORY=/home/me/annex ~/.ssh/git-annex-shell\" ...
"""]]


code:
[[!comment format=mdwn
 username="http://olg.myid.net/"
 ip="93.218.131.119"
 subject="comment 2"
 date="2012-11-05T23:34:20Z"
 content="""
Wow, thank you for implementing this so quickly!

Just one question: As far as I understood git-annex-shell relays all unknown commands to git-shell. In this case are there the same restrictions active, too?

Thanks again...
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.27"
 subject="comment 3"
 date="2012-11-05T23:42:10Z"
 content="""
All commands usually run by `git-shell` have the repository directory as their last parameter, and git-annex-shell depends on that being the case and checks them. It's possible to add some commands to git-shell by putting them in a special directory, and if those commands don't take the directory last, they wouldn't work.
"""]]


code:
I use the assistant with a local directory in direct mode. I have read in different places that direct mode does not ensure that past versions of files are preserved.

What precisely happens to past versions? Under which conditions are past versions kept?

How would I go about retrieving the past version of a file in direct mode?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="64.134.31.139"
 subject="comment 1"
 date="2013-10-21T22:16:16Z"
 content="""
If you change a direct mode file and that file has not yet been transferred to some other repository, you've changed the only copy of the file, and so you cannot get back the old version.

Other than that, git-annex always preserves old versions of direct mode files, the same as it preserves old versions of indirect mode files.

To get back an old version of a file, use `git annex indirect` to get out of direct mode, use regular git commands to check out the version of the repository that has the file you need, and use `git annex get` to retrieve the file if it's not available in the local repository.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmNu4V5fvpLlBhaCUfXXOB0MI5NXwh8SkU"
 nickname="Adam"
 subject="Disk space?"
 date="2013-11-02T01:33:10Z"
 content="""
Sorry, I'm confused: does using regular client repositories in direct mode still store old versions of files?  If so, for how long?  I don't have unlimited disk space, and if it stored in git every copy of every file forever, especially large ones, I'd run out of space...and then what?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 3"
 date="2013-11-02T17:36:23Z"
 content="""
You can use `git annex unused` and `git annex dropunused` to remove old versions of files.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmNu4V5fvpLlBhaCUfXXOB0MI5NXwh8SkU"
 nickname="Adam"
 subject="comment 4"
 date="2013-11-02T22:30:07Z"
 content="""
Thank you.  Just curious, are there plans to put that in the webapp?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkAUMhKOSkh9JaBA6xst3XxQIIsDEq5Zd4"
 nickname="Ovidiu"
 subject="How to do this on Mac OSX"
 date="2014-05-08T22:46:49Z"
 content="""
So I have a local repository on my MBP and a backup repository on S3.
How do I now restore OLD or deleted file versions on my MBP with git-annex assistant?
"""]]


code:
I've recently found the following problem:

I really really want to get back my original folder structure - which includes the real files, not the symlinks. I've searched for quite a while, but I simply could not find an acceptable solution...

So I thought I would like to ask you guys here, if anybody experienced similar problems (or at least knows a solution for my problem)?

Greetings

Pethor


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 1"
 date="2013-08-15T09:02:38Z"
 content="""
Sounds like you want to switch to [[direct mode]].

(Or possibly to run `git annex unannex`, if you don't want to use git-annex for these files.)
"""]]


code:
I have made some mistakes while using `git annex import` in direct mode. Now I see that some files have been erroneously added and there are other problems. I have not yet used `git annex sync`.

How can I tell git-annex in direct mode (or bare git) to forget about all these changes and revert back to the last known good (pre-import) state? This means also removing the few imported files and recreate their links.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 1"
 date="2014-06-04T19:03:05Z"
 content="""
I wish that I had a good answer for you, but since you're using direct mode, I really do not!

You can run this to see the status:

    git -c core.bare=false status

That's safe. Everything below has a good chance of losing data if not used carefully.

You can run a command like this to *delete* an unwanted file (that is shown as a new file in the above).

    git -c core.bare=false rm -f unwanted_file

If the status shows a file as being modified, you can run this to throw away the modified version. This does not put back the old version! It will leave you with a broken symlink.

    git -c core.bare=false reset modified_file
    git -c core.bare=false checkout modified_file
    git annex fsck

If you're lucky, you can then `git annex get` to get the old version from some other repository that still has it.
"""]]


code:
How can I run the assistant on a server and access the webinterface over the network from my client pc?
The use case I have in mind is that the server can automatically synchronize files, pushed over XMPP.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn7gQ1zZDdWhXy9H51W2krZYShNmKL3qfM"
 nickname="Karsten"
 subject="comment 1"
 date="2013-01-17T08:30:54Z"
 content="""
You don't need the webapp to run the assistant, but if you want, you can forward it to your local system. You will find the URL in .git/annex/url and have to extract the port from it (http://127.0.0.1:<someport>/.../), afterwards forward that port either via ssh port forwarding or via a webserver acting as a reverse proxy.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.238"
 subject="comment 2"
 date="2013-01-17T16:29:39Z"
 content="""
If this is a true server, with a static IP (or some dynamic DNS), there is no need to run the assistant on it; clients using the assistant can ssh in and sync data to it directly.

Anyway, you can git config web.browser to run some script with the url it will be passed to the webapp when the webapp is started. I sometimes configure it to just echo the url out; it'd also be possible to automate
setting up a redirection for a remote browser, I suppose.
"""]]


code:
Unfortunately one of the machines that I am running git-annex on is too old to handle the watcher thread.

The error message is:

    Watcher crashed: Need at least OSX 10.7.0 for file-level FSEvents

I am however interested in duplicating the assistant steps manually.  I'm happy to accept that I won't have the great, auto-magic, coverage, but, how I run the steps that the assistant runs?

I'm guessing there is a:

    git annex add .
    git commit -m "[blah]"
    git annex sync

but how do I get the pushing to backup/transfer/whatever remotes to work?  And what order should I do these?

What I would *like* to do is get a script that runs all of the needed annex steps and then kick it off in a cron job 2-5 times a day.

Any thoughts?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-05-19T20:18:11Z"
 content="""
Essentially, you want to run this:

<pre>
git annex add
git annex sync
for remote in $(git remote); do
    git annex copy --auto --to $remote
    git annex get --auto --from $remote
done
git annex drop --auto
</pre>

Of course this cannot sync with XMPP remotes. Otherwise it will get you to essentially the same place as the assistant.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnRai_qFYPVvEgC6i1nlM1bh-C__jbhqS0"
 nickname="Matthew"
 subject="Looks great"
 date="2013-08-29T12:45:10Z"
 content="""
This looks great as I have:

 * A preference for multiple small repositories.
 * Old versions for `git-annex` due to being on Ubuntu LTS for my server.
 * A Samsung Galaxy Nexus which somehow seems too slow to run the assistant.

So these steps combined with some locking and maybe `inotify` seem perfect
"""]]


code:
When syncing with a huge git annex repository on usb disk, my small laptop partition runs out of inodes.

Any workaround for this?

- Use bare repository?

    Some git annex command are not supported.
    This makes managing (particularly adding) files difficult.

- Use a loop file partition with tiny block size and large inode numbers?

    Operations on a huge git repository are slow.
    On a loop file partition will be slower.

- Maybe shrink the partition and make room for a specific partition for git annex repository?

Any opinions?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.51"
 subject="comment 1"
 date="2013-09-12T20:37:05Z"
 content="""
Define \"huge\"? Checking out a git repository necessarily requires one inode per file in the repository, plus a smaller quantity for the things in .git. A git-annex repository  is much the same as any other git repository.

Even when I make a really tiny 100 mb ext4 filesystem, it defaults to 25000 inodes, which would be enough to contain a checkout of my second largest git-annex repository.

Anyway, using git branches seems like a reasonable workaround, to the extent I understand your problem. Make a branch with the files in it you want to have available on the small drive, or check out an empty branch on the small drive and `git annex add` files in there. You can merge the branch back into your master branch on the large drive.
"""]]


code:
I have two machines A & B.  Each have their our repository.

On machine A I have created a remotehost (S3) and have synced successfully.

I am trying to link machine B to the same remote host.  I keep getting an error on the initremote command on machine B.

I tried using the same command as I did on A and it is not working.   Is there a different command to link machine B to an existing remote repository?

Any help is appreciated.

Rob


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/FHnTlSBo1eCGJRwueeKeB6.RCaPbGMPr5jxx8A--#ce0d8"
 nickname="Hamza"
 subject="comment 1"
 date="2013-12-13T08:37:24Z"
 content="""
On machine B you don't initremote you enableremote,

    git annex enableremote mys3
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnKhw0-Tu5oHjeoz_gAP4Nu5V9uo8ta3hM"
 nickname="Robert"
 subject="unknown special remote"
 date="2013-12-13T15:12:37Z"
 content="""
How does machine B know about the remote?  Machines A & B are separate and don't share references. Am I missing a step?
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/FHnTlSBo1eCGJRwueeKeB6.RCaPbGMPr5jxx8A--#ce0d8"
 nickname="Hamza"
 subject="comment 3"
 date="2013-12-14T09:05:26Z"
 content="""
remote info is recorded in git once you clone the git repo it knows about machine a and s3 repo.
"""]]


code: Is it possible to use the same jabber account for different annexes? I think I've mixed up several repositories by using the same jabber account for synchronizing...


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-04-11T15:02:40Z"
 content="""
This is a limitation of how jabber is currently used. You need different accounts to sync different repositories over jabber. Supporting using 1 account is the main open todo item on the [[design/assistant/xmpp]] page.
"""]]


code:
[[!comment format=mdwn
 username="paulvt"
 ip="2001:1af8:ff19:0:a288:b4ff:fe2c:f600"
 subject="Resources"
 date="2013-07-13T08:14:35Z"
 content="""
XMPP resources (carrying annex.uuid) cannot be of any help here?
"""]]


code:
Hi, I'd like to share files with friends, to do so we need a shared server to exchange the files. I could setup an ssh access on one of my servers but I don't want others to have a full access. I assume I can restrict this using a `command='wrapper-around-git'` prefix in `.ssh/authorized_keys`, where `wrapper-around-git` is simply a script that checks if `SSH_ORIGINAL_COMMAND` is an authorised command and runs it. This is exactly the approach used with Mercurial (hg-ssh) and presumably git also relies on this mechanism.

But to do so, I need to know what commands can be executed over ssh when using git-annex assistant. Could you document about this? Thanks!


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-05-24T15:35:16Z"
 content="""
The git-annex assistant automatically sets up a ssh key that is locked down this way when you select \"ssh server\" in the webapp.

The command you need to allow to run is git-annex-shell. This has been designed to be secure.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkwjBDXkP9HAQKhjTgThGOxUa1B99y_WRA"
 nickname="Franck"
 subject="comment 2"
 date="2013-05-24T15:47:36Z"
 content="""
Thanks for your quick answer! But this is true only for servers where git-annex is installed. On a server that just does SSH, things must be different. So, I've started to use the SSH `command=` mechanism to log the commands and discovered so far four of them:

 * `sh -c 'echo git-annex-probe...` and `sh -c 'mkdir -p ...` when the server is first connected by a client
 * `rsync --server -vre.iLsf --partial-dir .rsync-partial . DIR/` and `rsync --server --sender -e31.14 --inplace . DIR//bd1/469/TEXT` when files are transfered between clients

I want to derive patterns from this but if you could give them to me (ie, tell me which parts are fixed and which are variable) this will be safer. Moreover, I'm quite sure there are somme commands missing from my logs... By the way, parameter `-e31.14` to `rsync` surprises me because `-e` is supposed to set the remote shell (like `--rsh`).

Cheers, Franck
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 3"
 date="2013-05-27T19:14:37Z"
 content="""
I am confident that it will be *easier* to install git-annex-shell on your ssh server than it will be to lock down rsync.

All you need to do is go get the standalone linux tarball of git-annex, untar it, and add its directory to PATH.

You can google for perl scripts that lock down rsync, but I have never been happy with any of the ones I found.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkwjBDXkP9HAQKhjTgThGOxUa1B99y_WRA"
 nickname="Franck"
 subject="comment 4"
 date="2013-05-28T06:34:04Z"
 content="""
Thanks, but my server is a synology nas and as you know from another thread of comments, having git-annex work on it is not that simple. ;-)
Moreover, I'd like to be able to use ssh accounts where I don't have a root access and not necessarily git. So, a general method to restrict ssh would interest me.

But your answer seems to suggest that almost arbitrary rsync commands may be given. If so, I agree that there are few hopes to build a secured jail around this... But if really a limited subset of commands is used, I think it should be possible to check them securely.

Now on I'm focused on having git-annex work because this looks like the most promising way. But I'll have another question regarding it: I noticed that we can restrict access to a specific repository using an appropriate environnement variable. But it's it possible to provide a list of repositories instead of just one? My collaborators will typically have access to several shares but not to all of them.

Thanks for your responsiveness, after trying tens of candidates git-annex appears to be the only serious solution to replace Dropbox and I'm really glad that you actively help your users!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 5"
 date="2013-05-29T16:56:37Z"
 content="""
I can't say what options rsync will choose to pass to the server, when the rsync special remote uses rsync. It's up to that command.

`GIT_ANNEX_SHELL_DIRECTORY` currently only supports specifying one directory.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkwjBDXkP9HAQKhjTgThGOxUa1B99y_WRA"
 nickname="Franck"
 subject="comment 6"
 date="2013-05-29T17:41:25Z"
 content="""
OK, I understand now why it is complicated to secure rsync, the best I can hope is to chroot it on a restricted account. Thanks! And the \"currently\" regarding ˋGIT_ANNEX_SHELL_DIRECTORYˋ gives me hope about a future change.  :-)
Cheers, Franck
"""]]


code:
Hi,

Is it possible to have git-annex send requests across the repository network? Say I have a network topology like this:
Home (annex) <-> Cloud (S3) <-> Laptop (annex)

Home has all files, cloud has zero, and laptop has subset of files. Let's also assume Laptop can't talk to Home directly (maybe it's behind a NAT), but both Home and Laptop are connected to the internet.
If I'm away on my laptop, can I retrieve a file from Home "through" Cloud?

That is, Laptop checking its remotes and none of them have the file I want, so it checks remotes of remotes, etc. I'm not sure if git-annex knows the topology (seems likely considering it can generate the graphviz image). It also seems there is a communication medium of Jabber/XMPP where it could communicate between internet-connected git-annex servers (which might not otherwise be able to talk to each other directly)? So the fact that Cloud isn't a git-annex server, just a dumb key/value store would be okay?

I realize that I could set it up so Home mirrors everything to Cloud and then that eventually mirrors over to Laptop, but let's assume both Cloud and Laptop have small storage capacities, so on-demand fetching would be needed.

This is basically the same usecase as the USB transfer drive to sync two annexes not on the same network, but automated.

Thanks!


code:
[[!comment format=mdwn
 username="http://ypid.wordpress.com/"
 ip="213.153.84.215"
 subject="preferred content …"
 date="2014-07-02T07:26:59Z"
 content="""
Hi

This can be accomplished with the use of [[/preferred_content/]].
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 2"
 date="2014-07-02T17:25:51Z"
 content="""
The network topology shown has only a S3 special remote connecting the two repositories. This only provides a place to store file contents, it does not allow syncing the contents of git repositories.

You need either XMPP(Jabber) for that, or a git remote on some other server. Once there is a way for the git repository to sync, the Home repository will see when Laptop is missing a file it wants (configured via the preferred content mentioned above), and upload it to S3.
"""]]


code:
I want to setup a rsync special remote on my server with a non-standard ssh port.

I tried the following steps:

    git annex initremote rsync-encrypted type=rsync rsyncurl=1.2.3.4:/encrypted-annex encryption=AAAAAAAAA
    git config remote.rsync-encrypted.annex-rsync-options "-e \'ssh -p 443\'"

But I just get this error:

    [2012-11-22 21:04:30 CET] read: rsync ["-e","'ssh","-p","443'","--progress","--recursive","--partial","-- partial-dir=.rsync-partial","/home/marco/annex/.git/annex/tmp/rsynctmp/15309/","1.2.3.4:/encrypted-annex"]
    Missing trailing-' in remote-shell command.

I tried some ways to escape the config but I don't have a clue. Anybody?


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmLB39PC89rfGaA8SwrsnB6tbumezj-aC0"
 nickname="Tobias"
 subject="comment 1"
 date="2012-11-22T22:44:11Z"
 content="""
Add the host to your .ssh/config like this

Host myotherpc
     HostName 10.0.0.10
     Port 2222

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnldTTAP8PAifJUmqhRar6RAWNWlRcencw"
 nickname="Marco"
 subject="~/.ssh/config"
 date="2012-11-23T05:08:47Z"
 content="""
Thank you very much. Works like a charm.
"""]]


code:
Hi,

When using git-annex assistant, is there a possibility to share only certain files or directories from a repository (similar to what Dropbox can do), or do we have to share whole repositories so far?
Could this be a nice wishlist item, or is there a strong reason against adding this feature to git-annex?

Thanks a lot!


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.255.110"
 subject="comment 1"
 date="2013-09-09T19:44:57Z"
 content="""
Everyone can see the names of every file that is put into the git repository. Whether the contents of those files is transferred from the machine where they were created to other clones of the repository is configurable using [[preferred_content]], but it's not really intended as an access control mechanism.
"""]]


code:
I just setup git-annex across three macs, using git-annex assistant.  I have one repository working fine, which successfully syncs between two of my own accounts on two different machines, via S3.

Now I'm trying to get the "share with a friend" feature to work.  The problem is that only Mac Alias's / sym links are created, for example:

Machine 1:
- Copy file into ~/annex/

Machine 2:
- The file is synced as a sym-link, with a link into a file in .git that doesn't exist.

The config is:
- Version: 4.20130909-ga29f960 - Build flags: Assistant Webapp Pairing Testsuite S3 WebDAV FsEvents XMPP DNS Feeds Quvi
- Two macs
- Share with friend over Jabber
- Use Box as the cloud sync service with encryption enabled.  Box is setup as a transfer repository.

I've looked through the logs in .git/annex/daemon.log* and there doesn't look to be any problems.

This is what the log looks like:

    [2013-09-14 19:55:47 PDT] Committer: Adding 20120101-..084-1.jpg
    failed
    add /Users/nick/Shared/20120101-GOPR0084-1.jpg (checksum...) ok
    add /Users/nick/Shared/20120101-GOPR0084-1.jpg (checksum...) ok
    add /Users/nick/Shared/20120101-GOPR0084-1.jpg (checksum...) ok
    add /Users/nick/Shared/20120101-GOPR0084-1.jpg (checksum...) ok
    add /Users/nick/Shared/20120101-GOPR0084-1.jpg (checksum...) [2013-09-14 19:55:47 PDT] Committer: Committing changes to git
    [2013-09-14 19:55:48 PDT] XMPPSendPack: Syncing with USER2
    Already up-to-date.
    To xmpp::USER2@gmail.com
       7b65b15..183b91c  git-annex -> refs/synced/6163ef8e-c36a-4b09-919b-9c18ade55234/bmlja2JsYWNrMUBnbWFpbC5jb20=/git-annex
       3104275..96afa09  master -> refs/synced/6163ef8e-c36a-4b09-919b-9c18ade55234/bmlja2JsYWNrMUBnbWFpbC5jb20=/master
    [2013-09-14 19:55:49 PDT] XMPPSendPack: Syncing with USER2
    Everything up-to-date
    recv: resource vanished (Connection reset by peer)

Any ideas what I should look at next?  Thanks in advance :-)


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.14.105"
 subject="comment 1"
 date="2013-09-19T17:41:47Z"
 content="""
git-annex can only share the contents of files with a friend when you both have access to a transfer repository. Your friend's git-annex webapp should detect that it does not have such a repository, and should be displaying an alert like this:

[[assistant/cloudnudge.png]]
(But in this case it should say \"Unable to download files from YourName\")

So, you need to give your friend access to the S3 repository that you're already using to keep your 2 computers in sync. Or, set up some other transfer repository that you both can access.

There is a way in S3 to create another key that can read and write to a S3 bucket. You can then give that key to your friend.

Or, you could set up a shared git repository on a ssh server you both have access to. Or, set up a box.com remote -- the webapp will offer to store the box.com credentials and share them with your friend.
"""]]


code: Hi, is there any particular problem with symlinking one .git/annex to share between multiple repos?


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="don't do that"
 date="2012-03-19T18:23:13Z"
 content="""
Suppose you do that to repos A and B. Now, in A, you `git annex drop` a file that is only present in those repositories. A checks B to make sure it still has a copy of the file. It sees the (same) file there, so assumes it's safe to drop. The file is removed from A, also removing it from B, and losing data.

It is possible to configure A and B to mutually distrust one-another and avoid this problem, but there will be other problems too.

Instead, git-annex supports using `cp --reflink=auto`, which on filesystems supporting Copy On Write (eg, btrfs), avoids duplicating contents when A and B are on the same filesystem.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnhsaESlYphzLTzpJy5IxxGFxxctIhWYfo"
 nickname="Bryon"
 subject="comment 2"
 date="2012-03-19T18:46:13Z"
 content="""
Ah, OK. Is there a configuration step to set this up, or is this included magic in a new enough git-annex client?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnhsaESlYphzLTzpJy5IxxGFxxctIhWYfo"
 nickname="Bryon"
 subject="comment 3"
 date="2012-03-19T18:55:03Z"
 content="""
Nevermind, found it. (git-annex 0.08)
"""]]


code: I want to use the annex assistant instead of the command line. Which is the recommended method to check out a file that I want to edit?


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmaXxj2aPZrtRkWzwswTNep8hFif1cXykw"
 nickname="Erik"
 subject="Follow-up: check-in"
 date="2012-11-08T08:59:14Z"
 content="""
And the obvious follow-up question: How to I check in and sync the file that I just edited? (Note: I want to avoid the command line)
"""]]


code:
I have previously been confused by how to have a bit more manual control over where files are (as in [[bugs/git-annex_immediately_re-gets_dropped_files/]]).

I thought perhaps something like an archive directory would work, but as I might want different computers to have different content, perhaps a per-computer archive repository.

To that end, I have

walter@dionysus:~/annex$ git annex content .
(exclude=archive/dionysus/* or (not (copies=archive:1 or copies=smallarchive:1))) or (not copies=semitrusted+:1)
ok

which is a modified version of the archive [[preferred_content]], but with the intention that I change the part after `exclude` to be different for each client.
I also set the group to client for this client (dionysus).

However, it does not seem to drop files when I move them into `archive/dionysus/`, and I cannot see anything in the logs to suggest why.

What am I doing wrong here? Or, is this the wrong approach?

--Walter


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 10"
 date="2013-11-08T18:32:12Z"
 content="""
If I could comprehensively fix that bug I would.

But, it's supposed to already be fixed when using direct mode. Were you using indirect mode when you saw the problem?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnSenxKyE_2Z6Wb-EBMO8FciyRywjx1ZiQ"
 nickname="Walter"
 subject="comment 11"
 date="2013-11-09T06:02:36Z"
 content="""
No, this was using direct mode.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.14.105"
 subject="comment 1"
 date="2013-09-19T17:57:22Z"
 content="""
That sound like it should work. I suggest you play around with `git annex drop --auto` at the command line. It will probably tell you why it is unable to drop a file if the problem is something like not enough copies located elsewhere. Or, if it doesn't try to drop the file at all, you'll know your preferred content expression makes it want to keep the file.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnSenxKyE_2Z6Wb-EBMO8FciyRywjx1ZiQ"
 nickname="Walter"
 subject="comment 2"
 date="2013-09-19T22:22:03Z"
 content="""
So, it sort of works...

I realised that I don't have `archive` repository but instead `backup` ones, so adding that makes `git annex drop --auto` want to drop files in archive/dionysus.

However, the webapp then gets them. This I don't understand.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.14.105"
 subject="comment 3"
 date="2013-09-20T00:15:07Z"
 content="""
does `git annex get --auto` also get them?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnSenxKyE_2Z6Wb-EBMO8FciyRywjx1ZiQ"
 nickname="Walter"
 subject="comment 4"
 date="2013-09-20T02:41:22Z"
 content="""
I am testing on my laptop, and the content expression there is
`(exclude=archive/kronos/* ) or (not copies=semitrusted+:1)`
to simplify things; I interpret this to mean it wants everything not in archive/kronos, and it wants things there where that's the only copy.

The problem is that `git annex drop --auto` does indeed drop things, but `git annex get --auto` gets them; surely it should not be possible for that to happen?

I also tried the preferred content expression of `(exclude=archive/kronos/* )`, and the same thing happens.

The version of git-annex that I am using is

git-annex version: 4.20130919-g9be7762
build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP DNS Feeds Quvi
local repository version: 4
default repository version: 3
supported repository versions: 3 4
upgrade supported from repository versions: 0 1 2

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4830:1600:187::2"
 subject="comment 5"
 date="2013-09-20T15:31:18Z"
 content="""
Ok. This seems to be a bug in git-annex then. While it's surprisingly difficult to do so, it tries to interpet preferred content expressions in a stable way -- that is, they should not want to get a file when it's missing and then want to drop it when it's present.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4830:1600:187::2"
 subject="comment 6"
 date="2013-09-20T15:36:25Z"
 content="""
I tried to reproduce this behavior, but failed. My configuration was 2 repos, A and B, with B the origin of A. A was configured with \"exclude=archive/kronos/*\"
(also tried \"(exclude=archive/kronos/* ) or (not copies=semitrusted+:1)\")

<pre>
joey@darkstar:~/tmp/A>git annex get --auto
joey@darkstar:~/tmp/A>git annex get
get archive/kronos/foo (from origin...) ok
(Recording state in git...)
joey@darkstar:~/tmp/A>git annex drop --auto
drop archive/kronos/foo ok
(Recording state in git...)
joey@darkstar:~/tmp/A>git annex get --auto
joey@darkstar:~/tmp/A>
</pre>

... Which is how you want it to behave.

So I wonder if it's something else in your configuration. The best thing to do would be if you can come up with a series of commands I can follow to build repositories that exhibit the problem.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4830:1600:187::2"
 subject="comment 7"
 date="2013-09-20T15:37:48Z"
 content="""
Also, please check if you're running into [[bugs/Handling_of_files_inside_and_outside_archive_directory_at_the_same_time]]
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnSenxKyE_2Z6Wb-EBMO8FciyRywjx1ZiQ"
 nickname="Walter"
 subject="comment 8"
 date="2013-09-20T20:30:52Z"
 content="""
Hmm, that could be it; the same content is both inside and outside the archive directory.

What I really want to do (because maybe I'm not going about this the best way) is to use the assistant not in manual mode, but allow for some repos not having some files.
I thought by placing them in an archive directory, then they would be dropped, and to get them again, I can just delete from the archive dir, and they will be got (as the file is also outside the archive dir.

But, if that is not going to work, is there a better way to manage that?

By the way, I really appreciate all the work you put into this awesome project.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnSenxKyE_2Z6Wb-EBMO8FciyRywjx1ZiQ"
 nickname="Walter"
 subject="comment 9"
 date="2013-09-23T21:58:42Z"
 content="""
Thinking about this further, I'm not entirely sure what I want. I think the confusion arises from git-annex sometimes (mostly) caring about file *contents* (ie dependent on the hash of the file), but sometimes (preferred content, not sure of anywhere else) caring about file *location*.

What I think I actually want is a way of specifying locations that are not synced, such that if the file is changed somewhere (on another computer), the new version should not be downloaded. But, if the same content is in another location as well, the behaviour should be stable, I don't know how that should work though.

So, perhaps the best strategy is to have an explicit list of locations that I don't want, in the preferred content expression, if it could cope with files being in and out of some location at the same time. I think this would be easiest if I could avoid manually editing the expression all the time, maybe make a file with a list of file locations, if it would be possible for git-annex to handle that? I think it isn't at the moment, and my haskell is non-existent. That way, I could write some helper to add and remove files from this list. For example `dont-want file1 file2 dir1` would add these locations to a file, and `want file1 file2 dir1` would remove them from this list. Actually, I suppose I could make it just create an appropriate preferred-content expression, and then it doesn't need to support some file of locations.

So, after that ramble, I guess I'm envisaging a preferred content expression like `content=(exclude=path/to/file1 and exclude=path/to/file1 and exclude=path/to/dir1/*) or (some statement about numcopies)`, which I imagine updating whenever I decide I do/don't want some file. The only obstacle to this working is [[bugs/Handling of files inside and outside archive directory at the same time]] (as I understand that bug, could be wrong on the implications of it), meaning (of course) if there are two files with the same content, and I exclude one of them, and not the other, then it both wants and doesn't want the file, and it (and I) get really confused.

I suppose a short-term (well, slow) solution is to find duplicates of files I don't want, and if that exists either add the duplicate to my content expression (to say I don't want it), or remove the one I don't want from the expression (to say I do). This doesn't work well for when the content of one of the files changes (and so they are no longer duplicates), but I think I would search for them each time I generate the expression, so at that time it would no longer find the duplicate.

So, @joey, I guess my question is, what are the chances of that bug being resolved somehow? Or if that is not likely to happen soon, I might try to implement my solution outline from the previous two paragraphs.
"""]]


code:
Automounted USB devices are not detected by the git-annex webapp on my Debian testing/squeeze installation, only drives with entries in /etc/fstab show up in the device list. Is there any way to tweak/get around this?

I'm running version 4.20131106 (couldn't manage to build the package for sid).