code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkAUMhKOSkh9JaBA6xst3XxQIIsDEq5Zd4"
 nickname="Ovidiu"
 subject="comment 2"
 date="2014-05-16T17:15:59Z"
 content="""
I don't recall doing anything besides using the git annex assistant...

That link is just a screen shot...

anyway here is more info:

du -h -s git-annex/*
 38M	git-annex/Finance
8.0K	git-annex/archive

BUT if I check the folder git-annex's properties it says:

45 980 806 bytes (80,4 MB on disk) for 8 420 items

Is that normal?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 3"
 date="2014-05-16T19:07:10Z"
 content="""
Sounds to me like some GUI thing that is confused in some way. Which is, in my experience not unusual when it comes to GUI things and the actual size of files on disk. I'd trust `du`.
"""]]

repo_name: kakafarm/git

license: unknown


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkAUMhKOSkh9JaBA6xst3XxQIIsDEq5Zd4"
 nickname="Ovidiu"
 subject="comment 4"
 date="2014-05-17T10:56:22Z"
 content="""
I thinks its all the small files.

    du -hs /Users/ovi/Sharing/git-annex/
     77M	/Users/ovi/Sharing/git-annex/

    du -hs /Users/ovi/Sharing/git-annex/Finance/
     38M	/Users/ovi/Sharing/git-annex/Finance/

    du -hs /Users/ovi/Sharing/git-annex/.git/
     39M	/Users/ovi/Sharing/git-annex/.git/

The GUI shows:
    46 136 859 bytes (80,6 MB on disk) for 8 002 items

And the hidden .git folder shows:
    7 373 616 bytes (41 MB on disk) for 7 625 items

So apparently there are a lot of very small files, smaller than the blocksize hence the info is skewed? Anyway, I think its alright?
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2012-02-14T16:58:33Z"
 content="""
Well, it should only move files to `.git/annex/bad/` if their filesize is wrong, or their checksum is wrong.

You can try moving a file out of `.git/annex/bad/` and re-run fsck and see if it fails it again. (And if it does, paste in a log!)

To do that --
Suppose you have a file `.git/annex/bad/SHA256-s33--5dc45521382f1c7974d9dbfcff1246370404b952` and you know that file `foobar` was supposed to have that content (you can check that `foobar` is a symlink to that SHA value). Then reinject it:

`git annex reinject .git/annex/bad/SHA256-s33--5dc45521382f1c7974d9dbfcff1246370404b952 foobar`
"""]]


code:
[[!comment format=mdwn
 username="antymat"
 ip="77.190.74.127"
 subject="comment 2"
 date="2012-02-14T22:48:37Z"
 content="""
Thanks, joey, but I still do not know, why the file that has been (and *is*) OK according to separate sha1 and sha256 checks, has been marked 'bad' by `fsck` and moved to `.git/annex/bad`. What could be a reason for that? Could have `rsync` caused it? I know too little about internal workings of `git-annex` to answer this question.

But one thing I know for certain - the false positives should *not* happen, unless something *is* wrong with the file. Otherwise, if it is unreliable, if I have to check twice, it is useless. I might as well just keep checksums of all the files and do all checks by hand...
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 3"
 date="2012-02-14T22:57:29Z"
 content="""
All that git annex fsck does is checksum the file and move it away if the checksum fails.

If bad data was somehow read from the disk that one time, what you describe could occur. I cannot think of any other way it could happen.
"""]]


code:
[[!comment format=mdwn
 username="antymat"
 ip="77.190.74.127"
 subject="comment 4"
 date="2012-02-15T07:13:12Z"
 content="""
OK, thanks. I was just wondering - since there are links in git(-annex), and a hard links too, that maybe the issue has been caused by `rsync`.

I will keep my eye on that and run checks with my own checksum and `fsck` from time to time, and see what happens. I will post my results here, but the whole run (`fsck` or checksum) takes almost 2 days, so I will not do it too often... ;)
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 5"
 date="2012-02-15T15:22:56Z"
 content="""
The symlinks are in the git repository. So if the rsync damanged one, git would see the change. And nothing that happens to the symlinks can affect fsck.

git-annex does not use hard links at all.

fsck corrects mangled file permissions. It is possible to screw up the permissions so badly that it cannot see the files at all (ie, chmod 000 on a file under .git/annex/objects), but then fsck will complain and give up, not move the files to bad.
So I don't see how a botched rsync could result in fsck moving a file with correct content to bad.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmWQTrnPloMWiPFg8Y2Y5g-2IYe26D0KKw"
 nickname="Jim"
 subject="comment 6"
 date="2012-07-07T16:18:06Z"
 content="""
It's also possible you got a one-time DRAM corruption.  You have to expect those to happen every so often unless you're using ECC memory.
"""]]


code:
Based on the thread over at <http://git-annex.branchable.com/forum/__34__du__34___equivalent_on_an_annex__63__/> I decided to finally write a du like utility for git-annex.  A 0.01 version is up over at <http://git-annex.mysteryvortex.com/git-annex-utils.html>.  It works, but I intend to make it smarter about handling git repos and annexed files, as well as adding more of the options available in the standard du utility.

Currently it will tally up the sizes of links that look like they are annexed files.  I plan to make it actually interact with git and git-annex to verify the files are annexed and enable options like tallying files only from specific remotes, only missing files, not double counting files which are annexed multiple times but stored only once, etc...

I'll have time to work on this on the weekends, and plan to get my git repo up soon.  After gadu is mostly complete I might work on some other tools.

Releases are signed with a PGP key with fingerprint 5E1A 65D7 D5E9 56F1 C239  43DF C6C8 9A0B 6003 8953 (available on the website)


code:
[[!comment format=mdwn
 username="Steve"
 ip="92.104.175.136"
 subject="comment 10"
 date="2013-01-14T01:54:12Z"
 content="""
No problem, glad to see it is useful.  I'm not exactly a web guy, but I want to get some sort of comment/discussion system up there soon so we aren't filling up Joey's web site with semi-offtopic discussion.  (also a little beautification is in order)

Yes, contributions are welcome.  GPG/PGP encrypted email is the preferred mode of communication.

Currently I ask for copyright assignment in case I want to change licenses in the future.  I pledge not to go to a non-free license, but the GPL3 license choice was fairly arbitrary.  I might want to add the \"or any later version\" clause, for example.  There is also potential for a library to be split off which might benefit from something like LGPL licensing or similar.  I haven't really studied the licensing situation since GPL3 came around, so I need to take some time to look into it.

I don't want to have a licensing discussion here though as it would be offtopic.  Feel free to email me and we can discuss.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 11"
 date="2013-03-11T05:31:04Z"
 content="""
I don't want to steal gadu's thunder, and I really quite like having an ecosystem of tools develop around git-annex.

With that said, \"git annex status .\" now shows the disk used for all files in the current directory and below. It also shows the number of keys, and the total amount of disk those keys would use.

Additionally, you can use all the standard git-annex file limiting options. For example, here I'm finding out how much disk space is used by files located on a *remote* system:

<pre>
git annex status . --in turtle
directory: .
local annex keys: 0
local annex size: 0 bytes
known annex keys: 10
known annex size: 3 gigabytes
</pre>
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 12"
 date="2013-03-11T05:33:09Z"
 content="""
BTW, I think gadu still has its own uses, due to having a du like output, that can list space used by subdirectories. You can do that with git annex status *, but it's much more verbose, and doesn't show a break down by deeper subdirectories.
"""]]


code:
[[!comment format=mdwn
 username="Steve"
 ip="92.104.175.136"
 subject="0.02 is up"
 date="2012-12-08T14:20:16Z"
 content="""
I fixed some bugs that gave the wrong answer occasionally, and made gadu much smarter now.

It now searches for the .git dir an makes sure the git-annex links are well formed before counting them.
I also added a few more du like options.
"""]]


code:
[[!comment format=mdwn
 username="http://sunny256.sunbase.org/"
 nickname="sunny256"
 subject="comment 2"
 date="2012-12-08T15:35:16Z"
 content="""
Have downloaded v0.02 and experimented a bit, and it seems to work nicely. A couple of things, though:

- It displays the sizes in 512 byte blocks as default. I find that very confusing, and the standard `du`(1) from GNU coreutils uses 1024kB as default. AFAIK 512 byte blocks is an old way of measuring sizes from the really ancient UNIX days. Traditionally correct, maybe, but not very useful these days.
- When not specifying a path, can it use \"`.`\" as default?
- A human-readable format would've been nice, like 234M or 13G. The `du`(1) from GNU coreutils uses `-h` for this, but that option is already used for `--help`. And that's OK, I think `-h` should be reserved for that purpose. IMHO using `-h` as a synonym for `--human-readable` was a bad choice by coreutils, but it's too late to change that now.

Is there any Git repository available for git-annex-utils somewhere? That's my preferred way of getting updates and follow the development.

Anyway, thanks. :)

"""]]


code:
[[!comment format=mdwn
 username="Steve"
 ip="92.104.175.136"
 subject="gadu 0.03 is up"
 date="2012-12-09T13:05:10Z"
 content="""
* 1K blocksize is now the default
* \".\" is now the default path
* implemented -B/--block-size option
* --help is no longer -h, it only has a long option like du
* implemented -h/--human-readable option

du will take up to yottabytes for the --block-size option.  I had been fudging the sizes with a size_t thinking 16 exabytes was plenty big enough for now, but since I was implementing --block-size I went ahead and converted everything to use the GNU MP.  So libgmp is now a dependency.

--human-readable probably doesn't have exactly the same output, but I think it is good enough.  I tried to make the options work mostly the same as du from core-utils.  Let me know if you find other discrepancies.

I'll see about making the git tree available soon, but it may have to wait until next weekend.  I may also look into a forum for the website, or a mailing list.
"""]]


code:
[[!comment format=mdwn
 username="http://sunny256.sunbase.org/"
 nickname="sunny256"
 subject="comment 4"
 date="2012-12-09T20:13:47Z"
 content="""
Thanks a lot, Steve. Awesome, got everything on my wishlist. :) A very useful utility, and works perfectly. Will be using this a lot. git-annex-utils is a good name for this, I'm sure if you place it on GitHub or somewhere else you'll get lots of contributions and this could grow to be a project containing many useful utilities for git-annex.
"""]]


code:
[[!comment format=mdwn
 username="Steve"
 ip="92.104.175.136"
 subject="comment 5"
 date="2012-12-10T04:07:53Z"
 content="""
I pay attention to feedback ;)

I'm not done with it yet, I want to add in some options to limit what gets counted.

For example:  If you have two annexed files that contain the same content using the same backend, they will be stored only once in the .git/annex/objects directory but be counted twice by gadu.

I want to fix that, but I'll leave an option to keep that behavior if you want.  I also want to add options to count or not count files that exist in a certain repo.  It will be very easy to add options to only count files that you have or don't have locally as well.

Making it pay attention to environment variables that git and git-annex do would also be a good idea. (like GIT_DIR, etc...)

I'm open to good ideas that anybody has, unfortunately I can only work on it on the weekends for now.
"""]]


code:
[[!comment format=mdwn
 username="markus"
 ip="79.243.252.165"
 subject="Great util!"
 date="2012-12-28T17:45:27Z"
 content="""
Hi

gadu is a great util! The speed increase compared to \"du -smL\" will make it my fav. util for size calc!

ciao markus
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/2grhJvAC049fJnvALDXek.6MRZMTlg--#eec89"
 nickname="John"
 subject="sizes has git-annex support"
 date="2012-12-30T22:47:42Z"
 content="""
Just to note that I already added git-annex support to my \"sizes\" utility on Hackage several months back.  With -A, it shows you storage totals with annex symlinks computed fully resolved.
"""]]


code:
[[!comment format=mdwn
 username="Steve"
 ip="92.104.175.136"
 subject="git repo is now up"
 date="2013-01-01T23:39:33Z"
 content="""
sunny256, the git repo is now accessible at <http://git.mysteryvortex.com>

Markus, never used the -m option myself.  I added it in git it'll be in the next tarball.  (I plan to go through the du man page and add all appropriate options soon)

John, I wasn't aware of your sizes utility.  I'll look into it.

"""]]


code:
[[!comment format=mdwn
 username="http://sunny256.sunbase.org/"
 nickname="sunny256"
 subject="The git-annex-utils repo"
 date="2013-01-09T09:39:16Z"
 content="""
Thanks for setting up the git-annex-utils repo, Steve. Will install the newest gadu(1) on my computers now. I'll probably contribute some patches for some utility scripts or programs, if that's OK. What's your preferred way to receive patches? GPG-encrypted mail to your address in the commit log?
"""]]


code:
is `git annex get` and `git annex copy --to somewhere` expected to work with bare repos? the [[bare repositories]] page doesn't indicate otherwise, but a `git annex get` does plain nothing in my setup.

if it's supposed not to work, there should be an error message saying that and an indication on the [[bare repositories]], otherwise, how can i trace it down?

in case it is just unimplemented for lack of use cases: my setup consists of several laptops using parts of a 200gb+ photo collection, a central trusted server that should host everything, and an external encrypted remote backup. clients *should* copy everything they add to both central locations, but i'd prefer the trusted server to sync the two of them too.

`get` and `copy` usually operate on the current directory, which in case of a bare repo does not contain any relevant files, but i tried explicitly specifying files too. `git annex` should either look them up in master, or always operate on all files (as indexed in the `git-annex` branch) unconditionally.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.252.11.120"
 subject="comment 1"
 date="2012-11-13T17:30:31Z"
 content="""
git-annex commands only operate on files in the currently checked out working copy. You can send files to/from bare repos from non-bare repos. But you cannot do anything with them inside the bare repo, just as regular git doesn't let you commit inside a bare repo.

There is talk about [[todo/add_-all_option]], which would be usable in bare repos.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.193"
 subject="comment 2"
 date="2013-07-03T18:13:25Z"
 content="""
`git annex get` and `git annex copy`, as well as `git annex move` can now be used in a bare repository. In this mode, they behave as if --all were specified, and so operate on every single version of every single file.
"""]]


code:
I'm not sure if this is my stupidity or if it's a bug, but

    git annex copy --force --to REMOTE .

just zip's through really quickly and doesn't actually force a copy to a
remote location. This is just following up on the
[[bugs/git-annex_directory_hashing_problems_on_osx]]. I want to just do a force copy of all my data to my portable disk to really make sure that the data is really there. I would similarly would want to make sure I can force a

    git annex copy --force --from REMOTE .

to pull down files from a remote.


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2011-04-03T16:49:01Z"
 content="""
How remote is REMOTE? If it's a directory on the same computer, then git-annex copy --to is actually quickly checking that each file is present on the remote, and when it is, skipping copying it again.

If the remote is ssh, git-annex copy talks to the remote to see if it has the file. This makes copy --to slow, as Rich [[complained_before|forum/batch_check_on_remote_when_using_copy]]. :)

So, copy --to does not trust location tracking information (unless --fast is specified), which means that it should be doing exactly what you want it to do in your situation -- transferring every file that is really not present in the destination repository already.

Neither does copy --from, by the way. It always checks if each file is present in the current repository's annex before trying to download it.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 2"
 date="2011-04-03T16:59:47Z"
 content="""
Remote as in \"another physical machine\". I assumed that

    git annex copy --force --to REMOTE .

would have not trusted the contents in the current directory (or the remote that is being copied to) and then just go off and re-download/upload all the files and overwrite what is already there. I expected the combination of *--force* and copy *--to* that it would not bother to check if the files are there or not and just copy it regardless of the outcome.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 3"
 date="2011-04-03T17:12:35Z"
 content="""
On second thought maybe the current behaviour is better than what I am suggesting that the force command should do. I guess it's better to be safe than sorry.
"""]]


code:
I had one repo on HDD and another in an USB drive. After mount point changed, I went to the HDD repo folder and tried to use git remote-set url to point it to the new location, however I think I ended in a weird state. git-annex info shows this:

    git-annex info
    repository mode: indirect
    trusted repositories: 0
    semitrusted repositories: 5
        00000000-0000-0000-0000-000000000001 -- web
        26c0c4ba-6489-4416-a054-670d373f09bd -- juan@invasor.local:/Volumes/sapo_hfs/live/papers
        61158820-db14-45b9-b9f9-8619d956388e -- [usb_papers]
        d4456c86-fa2b-43a7-a132-027915390cf6 -- usb_papers
        fed56814-08c6-11e3-bf3c-af5da9f7f388 -- sapo [here]
    untrusted repositories: 0
    transfers in progress: none
    available local disk space: 255.46 gigabytes (+1 megabyte reserved)
    local annex keys: 1252
    local annex size: 2.55 gigabytes
    annexed files in working tree: 1297
    size of annexed files in working tree: 2.7 gigabytes
    bloom filter size: 16 mebibytes (0.3% full)
    backend usage:
        SHA256E: 2549

usb_papers was the original usb repo, and now a new one [usb_papers] shows. webapp shows only sapo[here] and [usb_papers], the other 2 repos are unknown. I was digging around but couldn't find a way to remove them.
Seems that the webapp managed to use the new URL, but said something about "fixing repo".
Any suggestions?
Thanks in advance.
Best regards,
              Juan


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 1"
 date="2014-07-11T19:40:32Z"
 content="""
    61158820-db14-45b9-b9f9-8619d956388e -- [usb_papers]

This is the git remote you currently have set up.

    d4456c86-fa2b-43a7-a132-027915390cf6 -- usb_papers

This is the old remote from before.

Take a look at the repository in the USB drive, and see what its annex.uuid is set to. git-annex doesn't change the UUID after a repository is set up, so the fact that you have two different UUIDs is puzzling.

If you really wanted to, you could remove the old repository from the list by running: `git annex dead d4456c86-fa2b-43a7-a132-027915390cf6`
But I would want to get to the bottom of what happened before doing that.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmTNrhkVQ26GBLaLD5-zNuEiR8syTj4mI8"
 nickname="Juan"
 subject="web remote"
 date="2014-07-15T13:19:42Z"
 content="""
Ok. Got it. And what's up with the web remote? Why is it there, I didn't add that one.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 3"
 date="2014-07-15T18:56:31Z"
 content="""
It's there to represent the world wide web when using eg, `git annex addurl`
"""]]


code:
After spending the whole last night playing with it, I think I have it figured out:

it runs on my Macbook and I have added a free 50GB box.com account as archive and an S3 bucket as full-backup and set the min. number of file occurrences to keep to 2 so IF I delete anything on my Macbook, this should make sure I definitely have 2 more occurrences of each file on two separate storages.

BUT here come the problems/questions

- say I add a folder called: Documents, it has now apparently successfully synced to box.com and S3. I say apparently because there is no tool in the assistant to check
- I can't seem to figure out how to delete the log of the assistant, its getting quite longish
- I can't seem to figure out how to upgrade from 5.20140420 to 5.20140421
- I move a file from a local repo to the archive folder but nothing happens except git-annex recording the move: (Recording state in git...)  /Users/ovi/Sharing/git-annex/Finance/1-564 Erkl. z. Gebiets- u. Steuereigenschaft 08_04.pdf still has writers, not adding [2014-05-09 17:22:43 SAST] Committer: Committing changes to git
- the assistant doesn'T have any kind of gui for retrieving/deleting/finding stuff - I'm no stranger to the command line but I can't figure out how to add the /Applications/gitannex.app.5.20140421/Contents/MacOS to my path and if I go there and run runshell, most git-annex commands tell me: not in a git repository so all I can do right now, is look at things :-/
- reading the description on the kickstarter page I must say I expected more. This is in no way comparable to Dropbox.

Don't get me wrong, this is the most promising project of its kind I came across if I can be pointed in the right direction :-)

P.S. How does it work on the "inside"? i.e. I edit a file and git-annex picks it up and "syncs" - what is synced? The whole file? A diff? How can one restore older versions or even look to see what versions are available?


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkAUMhKOSkh9JaBA6xst3XxQIIsDEq5Zd4"
 nickname="Ovidiu"
 subject="partially solved"
 date="2014-05-11T09:32:32Z"
 content="""
I managed to add the path and now I can use git annex command line.
All my other questions and remarks are still open though :-)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkYmMFDdf3GJ9Oba6NCVkzGc4JyB9WavMs"
 nickname="Xinruo"
 subject="comment 2"
 date="2014-05-11T15:15:50Z"
 content="""
> say I add a folder called: Documents, it has now apparently successfully synced to box.com and S3. I say apparently because there is no tool in the assistant to check

From the command line, git annex whereis FILE will tell you which remotes have the file, or git annex list will list out this for a directory

> I can't seem to figure out how to delete the log of the assistant, its getting quite longish

They live in .git/annex/daemon.log*

> I can't seem to figure out how to upgrade from 5.20140420 to 5.20140421

> I move a file from a local repo to the archive folder but nothing happens except git-annex recording the move: (Recording state in git...) /Users/ovi/Sharing/git-annex/Finance/1-564 Erkl. z. Gebiets- u. Steuereigenschaft 08_04.pdf still has writers, not adding [2014-05-09 17:22:43 SAST] Committer: Committing changes to git

It says the pdf still has writers.. Maybe the file is locked by another program? In this case the file is not going to be annexed, I believe, until the lock is revoked.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkAUMhKOSkh9JaBA6xst3XxQIIsDEq5Zd4"
 nickname="Ovidiu"
 subject="comment 3"
 date="2014-05-12T11:03:59Z"
 content="""
So basically, I need to check the log for a specific commit and then restore that one?
No way to list i.e. all commits for a folder and restore the whole folder?
Or list all commits for a fiel and then restore that one?

Still learning all of this, any hints welcome.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkAUMhKOSkh9JaBA6xst3XxQIIsDEq5Zd4"
 nickname="Ovidiu"
 subject="comment 4"
 date="2014-06-21T13:03:11Z"
 content="""
Any feedback on my last questions?
Is there a road map somewhere? Need to figure out if this is going somewhere soon or if I should look for another solution?
Any other kickstarter campaigns you're planning? Wouldn't mind chipping in if it means this goes somewhere fast(er) ;-)
"""]]


code:
What exactly does that mean?
I got that while exploring the git annex command:

shiny-2:git-annex ovi$ pwd
/Users/ovi/Sharing/git-annex
shiny-2:git-annex ovi$ git annex map
map /Users/ovi/Sharing/git-annex ok

  running: dot -Tx11 /Users/ovi/Sharing/git-annex/.git/annex/map.dot

failed
git-annex: map: 1 failed


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 1"
 date="2014-05-16T16:29:54Z"
 content="""
`dot` is a command from the graphviz package. Perhaps you do not have it installed.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkAUMhKOSkh9JaBA6xst3XxQIIsDEq5Zd4"
 nickname="Ovidiu"
 subject="comment 2"
 date="2014-05-16T17:21:09Z"
 content="""
indeed I didn't have it... was it stated somewhere that I need it?

Busy installing it now... going to kill my 3g Connection :-/

sudo port install graphviz
--->  Computing dependencies for graphviz
--->  Dependencies to be installed: autoconf m4 automake fontconfig freetype gd2 jpeg libvpx tiff xpm xorg-libXext xorg-libX11 xorg-kbproto xorg-libXau xorg-xproto xorg-libXdmcp xorg-libxcb xorg-libpthread-stubs xorg-xcb-proto xorg-xextproto xorg-libXt xorg-libsm xorg-libice gts netpbm jasper jbigkit libLASi pango Xft2 xrender xorg-renderproto cairo libpixman xorg-xcb-util gobject-introspection libtool harfbuzz graphite2 poppler curl lcms2 openjpeg15 poppler-data urw-fonts webp xorg-libXaw xorg-libXmu
"""]]


code:
Hi,

I've been experimenting with combining [ikiwiki](http://ikiwiki.info) with git-annex and it seems to work. Thought I'd post my process. I've [commented](http://ikiwiki.info/todo/git-annex_support/discussion/) on the ikiwiki website as well but perhaps it'd be of interest to git-annex folks.

I have very little understanding of any of the tools involved and have just attempted to make it work using my limited knowledge. I don't use the web interface for ikiwiki which simplifies things.

The [website in question](http://stockholm.kalleswork.net) just went online and is currently an archive of architectural photographs and the site relies heavily on the ikiwiki osm and album plugins.

### Setting things up

To start with I set up the wiki on the server and git clone to into `$wrkdir` on my laptop. I then initialize a git-annex repo in the `$srcdir` on the server. Leaving the `$gitdir` untouched. The `$scrdir` git-annex repo has to be in `direct` mode. Before doing any syncing I add `annex-ignore = true` and `annex-sync = false` to `.git/config` in the origin repo (`$gitdir`): this is to prevent polluting `$gitdir` with git-annex data. The same process is repeated in the `$wrkdir` on the laptop.

### Pushing and syncing

With this setup I can then `git add remote $srcdir`, `git add $file` and `git-push` mdwn files and other lightweight data from the laptop. While `git annex-add`, `git-annex sync` and `git-annex copy --to $srcdir` jpg's and other heavy files. All pure git commands work as expected with ikiwiki and the website recompiles etc.

### Snags

I'm frequently left with (non-dangling) symlinks in the `$srcdir` despite the annex repo being in direct mode. When this happens `git-annex fsck` sorts things out.

Uploading image files does require a bit of manual work. But as this is done less frequently it's not much of an issue for me. I'm guessing that by doing things it the correct order (whatever that might be) I could avoid some of the manual work.

The thing to keep in mind is to never `git-add` the typechanged annexed files in the $srcdir. In general I never use git commands in the $srcdir.

The main problem is the symlinks though as they demand a manual `git-annex fsck`. I have no idea what causes the symlinks in a direct mode repo.

Any comments?


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="72.0.72.144"
 subject="how about the web interface?"
 date="2013-09-25T11:09:58Z"
 content="""
I understand you do not use the web interface - but what if you did? would it commit all those files into git?

Could we add the git-annex files to a .gitignore file?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawk3HGoDpnOPob5jOjvIootmkve1-nCpRiI"
 nickname="Kalle"
 subject="comment 2"
 date="2013-09-25T11:58:42Z"
 content="""
I have very poor understanding of what ikiwiki actually does behind the scenes including how it uses the $srcdir and $gitdir.

The only way I could see the web interface working would be to use the git-annex content expressions and having the assistant running on the server. That still doesn't prevent large files from being checked into git though? That all depends on which order ikiwiki can be made to do things.

I might be able to test on a local wiki but that would have to wait a while.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.220"
 subject="comment 3"
 date="2013-09-25T17:03:29Z"
 content="""
Interesting experiment.

I don't know why you don't want to let git-annex sync its data to $gitdir.

The symlinks could be occuring because of a bug in direct mode. (I have fixed many past bugs that caused that.) But just as likely it's because ikiwiki will run `git pull` in the srcdir.

I think it would make more sense to use the underlay plugin and keep your annexed repository in a separate underlay. This would guarantee ikiwiki doesn't run git commands in there, and would ensure that nothing done with the web interface could mess with the annex.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawk3HGoDpnOPob5jOjvIootmkve1-nCpRiI"
 nickname="Kalle"
 subject="comment 4"
 date="2013-09-25T18:28:40Z"
 content="""
@Joey

    > I don't know why you don't want to let git-annex sync its data to $gitdir.

Well neither do I! :) It seemed to be the way to avoid duplicating data while still having the images picked up by the ikiwiki album plugin. Wouldn't the files in the $gitdir end up duplicated in $srcdir?

    > The symlinks could be occuring because of a bug in direct mode.
    > (I have fixed many past bugs that caused that.) But just as likely
    > it's because ikiwiki will run git pull in the srcdir.

When you mention it I've had similar problems with my vfat usb annex repos. Using the post-receive merge hook to make files visible for non git-annex devices about town. Nothing I can reliably recreate but I will keep my eye out for bugs.

    > I think it would make more sense to use the underlay plugin and keep
    > your annexed repository in a separate underlay.

Yep that would be ideal. For my usecase the album plugin is vital and I can't understand how album would pick up and deal with images in an underlay dir. This is a bit OT for this site though most of my questions are ikiwiki related.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawk3HGoDpnOPob5jOjvIootmkve1-nCpRiI"
 nickname="Kalle"
 subject="comment 5"
 date="2013-09-26T06:56:49Z"
 content="""
Turns out using the underlay was a piece of cake! Just mirror the folder structure of the repo in your underlaydir put your jpegs there and off you go. Images are picked up by the album plugin and it all just works. No need to coax git-annex into doing odd stuff.

Thought I'd post this for other ikiwikers.
"""]]


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="72.0.72.144"
 subject="step by step?"
 date="2013-09-26T09:40:57Z"
 content="""
So how does this actually work in practice? Is the underlay directory a completely disconnected repository?

Can we have step by step instructions on how to set this up?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawk3HGoDpnOPob5jOjvIootmkve1-nCpRiI"
 nickname="Kalle"
 subject="comment 7"
 date="2013-09-26T13:05:53Z"
 content="""
@anarcat

    > Can we have step by step instructions on how to set this up?

[I described the ikiwiki album setup](http://ikiwiki.info/forum/ikiwiki_with_album___38___underlay_plugins/) on the ikiwiki website as there are no git-annex tricks anymore. Just standard behaviour.

    > So how does this actually work in practice? Is the underlay directory a
    > completely disconnected repository?

Yes they are completely separate repos. Really the underlay dir could be managed in any way you like svn, rsync, ftp etc. I didn't expect the album plugin to pick everything up but apparently it does! Perhaps underlays are completely integrated and appear as 'normal' to all of ikiwiki?
"""]]


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="72.0.72.144"
 subject="comment 8"
 date="2013-09-26T13:10:25Z"
 content="""
Yes, I believe that is the way the underlay works - the files are just picked up.

As for the ikiwiki forum post - maybe you should move it to the \"tips\" section? :)

Thanks for the documentation!!
"""]]


code:
Hi everyone,

I need some suggestions on how to operate git-annex best in my setup.

I need git-annex mainly for its ability to have directories of all my data on all my nodes but not for the data redundancy it can provide.
I have one node that contains 2 filesystems that I want to merge in one git-annex repository. One filesystem (lets call it SAFE) is on top of a RAID1 between two 1TB hds. The other (BIG) is on top of a 3TB hd. SAFE holds data I do not want to loose (like digital pictures). BIG holds data that I can loose.

I do not have enough disk space on other nodes to get rid of the RAID1.

This is how I mount my filesystems:

SAFE at ~/AllData/

BIG at ~/AllData/bigfiles/

The root of the git repository is at ~/AllData/ however when I do:

git-annex add ~/AllData/bigfiles/file1
It says:
add bigfiles/file1 failed

I assume that is because of file1 being on a different filesystem.

Do I have to create two repositories: one for each filesystem or do you have any ideas on how to use git-annex best in this scenario?
Having two repositories also has the disadvantage that I need two repositories on all other nodes am I right?

Thanks for you suggestions


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-04-22T19:48:55Z"
 content="""
git-annex stores the contents of files inside `.git/annex/objects`. The `git annex add` is failing because it cannot `rename()` the file into that directory, because it is on a different filesystem. Even if it did a more expensive move of the file, it would not do what you want, because all the files would be moved to the `.git/annex/objects` directory, which is stored on your smaller drive.

The way git-annex is intended to be used with multiple drives is this:

* Make a separate git repository on each drive.
* Set up git remotes connecting these repositories together. You don't have to connect them all up, but at least make
  the git repository on your main filesystem have a remote for each git repository on other drives.
* Use `git annex sync` to keep the git repositories in sync. (Or do it manually with `git pull`)
* When you want a file to be available in the local repository, use `git annex get $file` to get it.
* When your local repository is getting too full, use `git annex drop` or `git annex move` to flush files
  out to the other drive(s).

The [[walkthrough]] goes through an example of adding a removable USB drive this way, but you can do the same thing for
non-removable drives.

> Having two repositories also has the disadvantage that I need two repositories on all other nodes am I right?

No -- you combine the two repositories, so any clone of either one contains all the files in both. Other notes then only need one
repository. However, for another node to be able to get files from both repositories on this node, it will need to have two git remotes configured, one for each repository.
"""]]


code:
[[!comment format=mdwn
 username="https://pradermecker.myopenid.com/"
 ip="81.244.215.244"
 subject="comment 2"
 date="2013-04-27T17:14:43Z"
 content="""
It works fine. After it is set up with the client as described, sync is automatic from the Assistant.

What I find cumbersome is the fact that I need to manually call \"git annex sync\" on the remote (usb or local ssh) to view (generate) the link. Is there a way to avoid this extra step ?

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmp1ThsNNAbSn46ju-gwFELfStlhl8usJo"
 nickname="Marek"
 subject="I should have thought of this"
 date="2013-04-30T19:55:19Z"
 content="""
Thanks joey, I thought of the object storage just yesterday and that the drive where it resides needs to be the big one.

I think that I will do it as you described with one addition:

I set up the git-annex repository on my big drive. Lets call that BIG.
Then I set up another on the smaller drive (or raid) and set both to sync. Lets call that RAID.

BUT: Can I use num copies to tell BIG to only sync files in certain directories to RAID. Or will syncing sync everything regardless?

Thanks for your help

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmp1ThsNNAbSn46ju-gwFELfStlhl8usJo"
 nickname="Marek"
 subject="Another correction"
 date="2013-04-30T20:11:57Z"
 content="""
sync just syncs the meta data but not the objects itself right? I have to do an explicit git-annex copy -to or copy -from?

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 5"
 date="2013-04-30T21:57:13Z"
 content="""
Yes, `git annex sync` only syncs the metadata.

If you have numcopies set to to, and run `git annex copy --to RAID --auto`, it will only copy files that have less than 2 copies. (Same with `git annex copy --from BIG --auto` or `git annex get --auto`)
"""]]


code:
Hi,

Thanks for git-annex, really a great project.
Another related feature which would be useful to have is described by tagfs [3][] [4][], experimental implementations such as [tagfs over fuse][1], and [tagsistant][2] exist, but having a solution within the power of git-annex would really be attractive.

How hard would this be to implement within the existing infrastructure?
Thanks.

related post: [[multiple_sym_links___40__for_tagging_photos__41____63__]]

[1]: http://code.google.com/p/tagfs/
[2]: www.tagsistant.net/
[3]: http://site.xam.de/2006/01-tagfs.pdf
[4]: http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=6&ved=0CE4QFjAF&url=http%3A%2F%2Fweb.mit.edu%2F6.033%2F2011%2Fwwwdocs%2Fwriting-samples%2Fsbezek_dp1.pdf&ei=ILQkUdqxMueR0QWgx4CwDw&usg=AFQjCNE1eWeFxmaxzOLZYVsb0tomqWNQaw&bvm=bv.42661473,d.d2k


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.210"
 subject="comment 1"
 date="2013-02-24T21:52:41Z"
 content="""
I welcome people doing this kind of thing with git-annex. Since you can have multiple files linking to the same content easily, it should be a reasonable platform with which to build something like this.
"""]]


code:
Hello,

I am looking for a solution to replicate lots of files between 2 servers (with dedicated IP address) over WAN. (I already looked at GlusterFS, DRBD, Bittorent-Sync and ownCloud, but comments are welcome). Now I am evaluating git-annex but it is (kind of) hard to get the concepts as I am not familiar with git.

I could already connect the servers by following the steps of "remote sharing walkthrough" but I feel that using a 3rd special node (as "transfer repository" / rsync or ssh) is an overhead and should not be needed. But no matter how hard I tried, I couldn't make the 2 servers do a sync without a special node.

Could you please give me a hint how to do it? Or even better, some detailed steps. :)

Thanks,

David


code:
[[!comment format=mdwn
 username="David"
 ip="178.21.48.117"
 subject="comment 10"
 date="2013-11-16T23:15:49Z"
 content="""
Great. Understood. :) Thanks.

Although there is one more thing that I don't really understand. Why do we need 2 repositories. In my mind (without knowing git) I thout it would work with one repository, where both server1 and server2 can upload/download files and they got synced through this.

    server1 <-> repository (somewhere, eother server1 or server2) <-> server2

Now they use different repositories and I don't get why. Also I don't get how conflict-handling can be done if there are 2 repositories for the 2 transfer-ways.

Sorry to bother you with this.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 11"
 date="2013-11-16T23:25:26Z"
 content="""
When using git, each place you access your files is a separate repository. Thus, you have:

server1 (repository) <--> server2 (repository)
"""]]


code:
[[!comment format=mdwn
 username="David"
 ip="178.21.48.117"
 subject="comment 12"
 date="2013-11-18T11:25:06Z"
 content="""
Hello,

Thanks for your help and explanation. Now I had some time to test git-annex and I had a sad experience. I tried to sync the files from server1 to server2 (as described before). We are talking about 87000 small files (these are maildir folders and each file is an e-mail) with the total size of 19 Gbytes. After a few hours git-annex assistant consumed all memory on both servers ... and then all swap ... and then kernel killed git-annex. The servers have 512 MB RAM and 1 GB swap.

According to the Scalability page (http://git-annex.branchable.com/scalability/) the \"memory usage should be constant\" and I am sure I didn't run \"git-annex unused\".

Is this memory consumption normal or it might be a bug? What shall we do?

Thanks,

David
"""]]


code:
[[!comment format=mdwn
 username="David"
 ip="178.21.48.117"
 subject="comment 13"
 date="2013-11-19T09:17:26Z"
 content="""
Please tell me if/how I can help you. For example if you need access to servers to test...
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 14"
 date="2013-12-16T20:36:14Z"
 content="""
I see the assistant spike up to around 150 mb when adding a lot of files, then drop back down once it finishes one batch (limited to around 5000 files) and moves on to the next.

You might need to file a bug report with some more details so I can reproduce the problem.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.246"
 subject="comment 1"
 date="2013-11-16T17:42:48Z"
 content="""
The [[assistant/remote sharing walkthrough]] assumes that the computers are not servers, so they might not be turned on at the same time, and cannot directly contect each other.

With 2 servers, it's much simpler. Just add a git remote on each pointing at the git repository on the other server, and run the git-annex assistant on both to keep them in sync.

You can add the git remote with `git remote add otherserver ssh://otherserver/path/to/repo`, or you can use the webapp to do it (Add another repository -> Remote server)

(If your WAN puts the servers on the same virtual subnet, ie a VPN, you can also probably use local pairing over the WAN to get to the same setup by a different route.)
"""]]


code:
[[!comment format=mdwn
 username="David"
 ip="178.21.48.117"
 subject="still need some clarification"
 date="2013-11-16T22:02:46Z"
 content="""
Hello,

Thanks for your reply. I guessed that there is an easier, direct way to do it. But unfortunately I still need some clarification. This is how I tried:

1. Cleared everything to start from scratch.
2. I started git-annex on server1: screen git annex webapp --listen=server1_ip, opened the webapp and created the local repository /media/mail
3. I configured the XMPP account on server1.
4. Started git-annex on server2: screen git annex webapp --listen=server2_ip, opened the webapp and created the local repository /media/mail
5. I configured the XMPP account on server2.

And here comes the confusing part:

6. On server1 I clicked Add another repository / Remote server: and configured ssh server2 to the same folder: /media/mail. I chose client repository group instead of transfer. (I am not sure if I did it right) ... but server1 started sync-ing.
7. On server2 I clicked Add another repository / Remote server: and configured ssh server1 to the same folder: /media/mail. Then I received an error message:

    Failed to make repository
    Something went wrong setting up the repository on the remote server.

    Transcript:

    init  error: could not lock config file /media/mail-test/mail/.git/config: Permission denied
    git-annex: git [Param \"config\",Param \"annex.version\",Param \"3\"] failed

Cold you please help me where I did wrong?

Thanks,

David
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.246"
 subject="comment 3"
 date="2013-11-16T22:18:20Z"
 content="""
It seems to me that you probably told git-annex to log into server2 as a user that does not have write access to sever2:/media/mail. Check the username field in the remote repository form.
"""]]


code:
[[!comment format=mdwn
 username="David"
 ip="178.21.48.117"
 subject="still need some clarification"
 date="2013-11-16T22:31:07Z"
 content="""
Hey,

Yes, you were right. The ssh user did not have privileges to read/write the folder. Now I have corrected this issue and things started sync-ing.

But I am afraid, this is not exactly the thing I wanted to achieve. My goal was to sync from folder to folder ... and not from folder to repository both ways.

So, what I would like to do is that if I create a file at server1 (e.g. /media/mail/test01.txt) then it should appear at server2 at /media/mail/test01.txt.

How can I achieve this?

Thanks a lot.

David
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 5"
 date="2013-11-16T22:55:34Z"
 content="""
Well, that's what should happen. What behavior are you seeing? What version of git-annex are you using?
"""]]


code:
[[!comment format=mdwn
 username="David"
 ip="178.21.48.117"
 subject="comment 6"
 date="2013-11-16T22:58:15Z"
 content="""
Hello,

Yes, meanwhile files started appearing. :) Maybe I just did not wait enough time.

So, I have just one more question. Shall I set the remote ssh repositories to client or transfer? I read the docs and understand the difference, but in my case I simply cannot decide.

Thanks a lot for your hard work and great support.

David
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 7"
 date="2013-11-16T23:04:29Z"
 content="""
In your situation, you do not have or need a repository whose only job is to transfer files between two other client repositories. So the right choice for your repositories is client -- or something else possibly -- but almost certainly not transfer.
"""]]


code:
[[!comment format=mdwn
 username="David"
 ip="178.21.48.117"
 subject="comment 8"
 date="2013-11-16T23:09:32Z"
 content="""
Okay. One last question. Could you please help me understand how this sync works now?

So for example if I save a file on server1, it gets pushed to server2's repository. But how is it translated back to a normal file from the repository?

And in this case will this file be on server2's repository as well (so are there 2 copies on server2, one in the repository and one human-readabl)?

Thanks!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 9"
 date="2013-11-16T23:11:58Z"
 content="""
Since you created these repositories with the webapp, they default to using [[direct_mode]]. git-annex handles putting the files in place when they are transferred to the repository. Only 1 copy is made, there is no separate copy in the repository.
"""]]


code:
Hi,

Is there a chance to get a build of git-annex for the Nokia N9 smartphone with Meego Harmattan and later for Sailfish OS (Jolla phone)?
Git is already available for Meego Harmattan: http://www.who.is.free.fr/wiki/doku.php?id=harmattan

Cheers,
Tobias


code:
[[!comment format=mdwn
 username="paulvt"
 ip="2001:1af8:ff19:0:a288:b4ff:fe2c:f600"
 subject="comment 1"
 date="2013-07-13T08:21:29Z"
 content="""
I've just checked and almost all of the depends that Debian Sid on my laptop has, the N9 has.  But in the case of build-depends, they are nowhere near since the entire Haskell stack is not available...
"""]]


code:
Thought I'd ask how y'all are finding the current communication by this forum/website/git repo only.

Would there be a benefit to having an irc channel for git-annex?

Maybe a mailing list? (Any persuasive reason why it would be better than this forum?)

Are the existing RSS feeds on this site, for eg, new [[comments]] and posts to this forum, sufficient to keep up with
things?

--[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 1"
 date="2011-03-28T15:48:08Z"
 content="""
No matter what you end up doing, I would appreciate a git-annex-announce@ list.

I really like the persistence of ikiwiki, but it's not ideal for quick communication. I would be fine with IRC and/or ML. The advantage of a ML over ikiwiki is that it doesn't seem to be as \"wasteful\" to mix normal chat with actual problem-solving.  But maybe that's merely my own perception.

Speaking of RSS: I thought I had added a wishlist item to ikiwiki about providing per-subsite RSS feeds. For example there is no (obvious) way to subscribe to changes in http://git-annex.branchable.com/forum/git-annex_communication_channels/ .

FWIW, I resorted to tagging my local clone of git-annex to keep track of what I've read, already.

-- RichiH
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 2"
 date="2011-03-28T18:35:50Z"
 content="""
I think the forums/website currently is sufficient, I do at times wish there was a mailing list or anonymous git push to the wiki as I find editing posts through the web browser is some times tedious (the lack of !fmt or alt-q bugs me at times ;) ). The main advantage of keeping stuff on the site/forum is that everything gets saved and passed on to anyone who checks out the git repo of the code base.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 3"
 date="2011-03-28T20:47:23Z"
 content="""
Push access to the non-code bits of git-annex' ikiwiki would be very welcome indeed. Given the choice, I would rather edit everything in Vim than in a browser. -- RichiH
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnx8kHW66N3BqmkVpgtXDlYMvr8TJ5VvfY"
 nickname="Yaroslav"
 subject="comment 4"
 date="2011-04-13T17:53:26Z"
 content="""
.1 cents: Having IRC would be really nice for seeking quick help.   E.g. like I was trying to do now, google lead me to this page.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkptNW1PzrVjYlJWP_9e499uH0mjnBV6GQ"
 nickname="Christian"
 subject="comment 5"
 date="2011-04-14T11:24:59Z"
 content="""
I would also like an git-annex channel. Would be #git-annex@OFTC ok?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 6"
 date="2011-04-15T19:32:08Z"
 content="""
We seem to be using #vcs-home @ OFTC for now. madduck is fine with it and joeyh pokes his head in there, as well. I just added a CIA bot to #vcs-home and this comment is a test if pushing works. -- RichiH
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="anonymous git push"
 date="2011-05-19T19:21:51Z"
 content="""
@Jimmy mentioned anonymous git push -- that is now enabled for this wiki. Enjoy!

I may try to spend more time on #vcs-home -- or I can be summoned there from my other lurking places on irc, I guess.
"""]]


code:
I want to put my photos into a git-annex repository, syncing it with my girlfriends computer and our NAS.
If possible, I'd like to be able to add files to the NAS and have them synced, so I guess a special remote doesn't work here.

Unfortunately, git-annex doesn't run on my NAS directly (yet), so I thought of mounting the NAS with CIFS, and creating an annex there that syncs with the local ones.
While this seems to work fine with one computer, I wonder what will happen if I mount the Samba share on my and my girlfriend's computer at the same time.

In theory, the NAS supports Samba Unix extensions which includes POSIX locking, but a weird bug that prevents you from removing a named pipe from Samba, making git-annex init fail (sent a bug report to Synology). When I disable Unix extensions it works. It then detects a crippled file system though.

Any thoughts?


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmqz6wCn-Q1vzrsHGvEJHOt_T5ZESilxhc"
 nickname="Sören"
 subject="Update"
 date="2013-06-26T17:26:05Z"
 content="""
After reading trough this again, I'd like to rephrase my question.

Is it safe if two independent assistants access the same repository at the same time (assuming locking works)?

But, much more important, a big thanks for creating git-annex! I'm using it successfully to keep my files in sync at home and at work and I think it is a really amazing tool. So thanks for the hard work!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.193"
 subject="comment 2"
 date="2013-06-26T17:40:21Z"
 content="""
If you're asking if you can run two git-annex assistant programs inside the same repository at the same time, the answer is no. Assuming locking works (grin), the second program will detect the first is running and refuse to start.

If your question is, can two or more git-annex assistant or other programs, running in separate repositories, access files in the same remote safely at the same time, the answer is yes! You can, for that matter, run multiple git-annex commands at the same time in the same repository, and they're also entirely safe, thanks to some locking it does.

You should post your story to [[design/assistant/blog/day_288__success_stories]]!
"""]]


code:
I am trying to get git-annex working on Ubuntu, specifically 13.04 and now 13.10 (I thought upgrading to a new distro. would fix the issue).

I have tried:

  * installing via apt-get install git-annex
  * installing via cabal (full-build)

Both times git-annex install successfully with no errors. I then start the webapp and create a repository (which git-annex creates successfully), I then add the folders and files I want it to sync. Git-annex finds the files and says it has began syncing them, but it never moves past the first batch of files it 'says' it had started syncing. I have waited 5+ hours at one point and nothing has changed. Also, the webapp is incredibly slow, it takes 15+ seconds to perform ANY action (such as saving options or viewing the logs). The issue definitely isn't my computer as I am using an high-end SSD, core i7, DDR3 RAM, etc...

Also the logs produce no errors of any kind and actually show git-annex adding files, most of the log entries say add 'filedirectory/filename.txt' as an example.

I have gotten this to successfully work on Fedora, but Ubuntu (13.04/13.10) is having serious problems.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="64.134.31.139"
 subject="comment 1"
 date="2013-10-22T17:57:27Z"
 content="""
I don't really understand the problem you're describing. Is the problem with it uploading the files it's added to another remote? Does it sit there saying \"syncing with $remote\" forever? Or what?

As to the webapp speed problem, it seems likely to me this is not related to CPU speed or load. It sounds rather like a DNS timeout problem, or an ipv6 to ipv4 fallback problem. Although since the URL to the webapp is http://127.0.0.1, it can't really be either of those..
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlEeLtIxQCU0qe4q5lbrZp8AcUSUUcRrkg"
 nickname="Zach"
 subject="comment 3"
 date="2013-10-23T02:01:09Z"
 content="""
The problem is not with a remote annex. The issue I am having is git-annex never finishes syncing.

Maybe a screenshot will help. http://i.imgur.com/9HSr8bl.png

It never gets passed the Startup scan.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="64.134.31.139"
 subject="comment 3"
 date="2013-10-23T15:27:40Z"
 content="""
Your screenshot shows git-annex is in the process of adding 6244 files. Depending on the size of those files, it could take some time to read their contents in order to calculate a checksum and add them to the repository. This would not normally take hours, unless your disk is very slow or the files are extremely big.

You might get some useful information if you enable debugging (Configuration -> Preferences -> Enable debug logging) and look at the log.

(This is not \"syncing\" btw. You have not set up any other repository for it to sync data with.)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlEeLtIxQCU0qe4q5lbrZp8AcUSUUcRrkg"
 nickname="Zach"
 subject="comment 4"
 date="2013-10-25T20:40:26Z"
 content="""
Everything works, it was just taking a while on one the folders; let it run overnight, and it was done.

Thanks for the help!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 5"
 date="2013-10-26T15:53:28Z"
 content="""
Can you confirm that the files in the directory it was adding were so large that it would normally take hours to checksum them all?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlEeLtIxQCU0qe4q5lbrZp8AcUSUUcRrkg"
 nickname="Zach"
 subject="comment 6"
 date="2013-10-26T21:05:07Z"
 content="""
The folder that took a while is actually only 291.5 MB total, containing 25,977 items in total. I know that is the folder that was having some issues because I committed each folder separately.
"""]]


code:
Hi!

I can't get git-annex to run on my archlinuxARM sytem. It fails with

    $ git annex test
    error: git-annex died of signal 11

 Its a RaspberryPi (model B I think) so its an ARM11 (armv6h if I'm not mistaken). I use archlinuxarm because I am familiar with the distribution and you can have a very lean system. There is a very crude package for git-annex arm and armv7 that is build like this: https://github.com/archlinuxarm/PKGBUILDs/blob/master/alarm/git-annex/PKGBUILD . Crude, because it just takes the debian package, changes the version of libraries where apropriate and packs it as archlinux package. Two problems: 1) It's not been tested on armv6. 2) The 4.20130417 version does not exist anymore on the debian mirrors.

1. The debian package is not marked as armv5 or armv7 specific and is runs on the RaspberryPi with Raspbian (but in an pretty old version if I remember correctly). So I would have imagined it runs on armv6h in general.\\
2. I just did the same magic as in the linked PKGBUILD with a more current version to build the package but it still crashes. Can I get the old version anywhere to check if it works with that?

Cheers,
  Hannes

Background for whoever's interested :)
I tried to set up a RaspberryPi to manage my files, contacts and calendars. That shouts out loud for owncloud because of its convenience (caldav, carddav, webdav and a nice interface bundled together). However, compared to git-annex owncloud is unbearably slow (on an RaspberryPi) and lacks git-annex's easy way to keep your files sorted even if you don't have them on your local disk.

For these reasons I tried to install a hybrid git-annex/owncloud system. There, basically, the files are served by owncloud but at the same time also managed (and served) by git-annex (+ssh). As far as I can see that should not have any noticeable side effects in my use case (mostly single user). I have some external hard drives connected to the RaspberryPi of which I will just symlink the appropriate folders to the owncloud 'files' folder. This way I can also just take the drives with me and still have the same (UUID) git annex repository on them. Any comments?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.51"
 subject="comment 1"
 date="2013-09-12T16:56:50Z"
 content="""
You seem to be trying to take a binary built for one distribution with one CPU architecture and use it on another distribution on another CPU architecture. It's not very susprising that this does not work.

Compiling git-annex from source for your particular architecture and distribution should not be difficult.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmNvX8PQVP5sLzQ78sKpB6VeH3Fu8HvZ5g"
 nickname="Jeff"
 subject="comment 2"
 date="2014-01-26T04:29:09Z"
 content="""
I've actually been having a pretty tough time with it. Could you point me in the right direction? There's not any Haskell infrastructure on Arch ARM yet so I'd either be compiling the whole Haskell Platform or cross-compiling on x86_64 right?
Thanks
Jeff
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.46"
 subject="comment 3"
 date="2014-01-26T17:59:52Z"
 content="""
Well, yes, the first step will be getting a ghc for the architecture, and then it's just a matter of following [[install/cabal]]
"""]]


code:
on a Mac OSX 10.9.1 client I would like to try out git-annex.

I download the latest Mavericks build, extract from the .dmg and it just does not run.  I open the program, one "bounce" on the dock and it is gone, nothing happens.

I sadly do not have another OSX client to try it on.  But nothing shows in the Console.app or anything.

Am I missing something?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.146"
 subject="comment 1"
 date="2014-03-05T22:30:00Z"
 content="""
The thing to do is to try to start git-annex from the command line in a terminal. This should get an error message that will help find out what's going wrong.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.146"
 subject="comment 2"
 date="2014-03-07T15:17:50Z"
 content="""
Probably this was because the OSX build was missing the webapp. Fixed now.
"""]]


code:
Hi all,

git-annex basically renders my repository unmanageble.  What is the
best and save (!) way to recover?

Here is my situation:

I have a fairly large repository with ~8000 managed files taking about
65GB of disk space.

git-annex worked well there.  But some programs choke on the
symlinks.  So, I converted the repository to direct mode.  The
transition worked well.

Now git status reports a type change for the ~8000 files.

But as soon as I run

    git commit -m "typechange" even-only-one-of-the-files

the process `git-annex pre-commit .` eats 3.5GB of ram, where I
usually kill it, as I only have 4GB of ram....

-- Andreas


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/FHnTlSBo1eCGJRwueeKeB6.RCaPbGMPr5jxx8A--#ce0d8"
 nickname="Hamza"
 subject="comment 1"
 date="2013-09-18T07:19:49Z"
 content="""
AFAIK in direct mode you are not supposed to commit you just run

    git annex sync

and it will commit if files are changed. you only add new files annexed files are handled by sync

"""]]


code:
[[!comment format=mdwn
 username="ringprince"
 ip="134.76.140.110"
 subject="comment 2"
 date="2013-09-18T12:27:38Z"
 content="""
Hi Hamza,

thanks for that comment.  I thought `git annex sync` is just a wrapper around `git commit -a` (among others).

Using `git annex sync` does not help, as it just means that now `git annex sync` eats all my memory until swapping starts.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.14.105"
 subject="comment 3"
 date="2013-09-19T18:49:05Z"
 content="""
git-annex sync is a wrapper around git commit. But not -a! git commit -a will stage every one of your large files directly into the git repository, wasting much memory and worse, disk space. It is ok to use `git commit` or `git commit --staged` in direct mode after eg `git annex add`. But not `git commit -a` or `git commit even-only-one-of-the-files`. It's best to just use `git annex sync` rather than `git commit`, as it avoids finger memory causing you to run the wrong type of commit command. Please see [[direct_mode]] for the details.

I was able to make pre-commit take a lot of memory by committing a 1 gb file directly to git. git-annex was buffering the whole file content in memory
due to not thinking to check first if it was a symlink. I have fixed that bug.

So I think you must have run that command you showed, and you now have a lot of data stored in your git repository that you had meant git-annex to handle. You might need to use git-filter-branch to remove it..

This kind of thing is why I need to write the [[todo/direct_mode_guard]].
"""]]


code:
[[!comment format=mdwn
 username="ringprince"
 ip="134.76.140.110"
 subject="comment 4"
 date="2013-09-19T20:06:39Z"
 content="""
Let me ask a simple question:  All files are regular files (no symlinks).  If I can live with loosing the history, is it save to just remove the .git directory and start over?  Or do I risk anything?

PS: .git is about 7GB
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.14.105"
 subject="comment 5"
 date="2013-09-19T20:57:18Z"
 content="""
You can safely delete the git repository in the situation you describe. Better yet: Tar it up, and check the tarball into git-annex with the other files when you start over. :) Then if I'm wrong, you still have that data.
"""]]


code:
[[!comment format=mdwn
 username="ringprince"
 ip="134.76.140.110"
 subject="comment 6"
 date="2013-09-19T22:27:38Z"
 content="""
Thanks a bunch!  That is what I did now.  Seems good.

May I additionally suggest a change in the man page of git annex?  There I read

    sync [remote ...]
              ...
              The sync process involves first committing all local changes (git commit -a), ...
              ...

which made me think, I could do `git commit -a` manually as well.
"""]]


code:
Hi,

I'm on a centos 6.4 64bits & installed git-annex
version : git-annex-3.20120522-2.1.el6.x86_64

From my understanding of the help page, I should be able to sync content with :

git annex sync --content

but it is not recognized. The help tells me I may be able to use "get" but I'm unclear how...

Thanks for any help


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 1"
 date="2014-06-05T17:32:33Z"
 content="""
This feature was added in 5.20140210, which can be easily seen by reading the changelog.
"""]]


code:
Hi,

Does git-annex (preferably assistant) have any management of users / groups or teams? Our use case is we have many users which should only have access to certain folders (repos). Is this a planned feature or any manual way to work around this for now?

Thanks!


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.193"
 subject="comment 1"
 date="2013-06-25T17:10:17Z"
 content="""
Anyone with access to a git repository can access all the files stored in it. git-annex does not try to change this, so you need to find a way to have one repository for each group.
"""]]


code:
I've been organizing my music collection deleted/replaces some files but git-annex unused does not show any unused files even though they are deleted and all repos are in sync.

    git log --stat --all -S'SHA256E-s8034842--5c3475d7fef6f0c3545721f34e7cbfb6727a00708bdde192f0f1d53af251b444'
    commit a0fecdc02f7564f8bce9726f6b934fefc11de58b
    Date:   Thu Sep 12 17:04:53 2013 +0300

        Deleted

     .../Dido - Sitting On the Roof of the World.mp3    |    1 -
     1 file changed, 1 deletion(-)

    commit 7f216228fc0e6298f0290ee1d8646bc9b16eca10
    Date:   Mon Aug 5 03:49:55 2013 +0200

        Initial Import

     .../Dido - Sitting On the Roof of the World.mp3    |    1 +
     1 file changed, 1 insertion(+)`

even though the file is deleted its still present in annex objects,

    find .git -name '*5c3475d7fef6f0c3545721f34e7cbfb6727a00708bdde192f0f1d53af251b444*' -exec ls -al '{}' \;
    total 7856
    dr-xr-xr-x 2 user user    4096 Sep  9 01:24 .
    drwxrwxr-x 3 user user    4096 Sep  9 01:24 ..
    -r--r--r-- 1 user user 8034842 Aug  5 04:52 SHA256E-s8034842--5c3475d7fef6f0c3545721f34e7cbfb6727a00708bdde192f0f1d53af251b444.mp3
    -r--r--r-- 1 user user 8034842 Aug  5 04:52 .git/annex/objects/Jz/74/SHA256E-s8034842--5c3475d7fef6f0c3545721f34e7cbfb6727a00708bdde192f0f1d53af251b444.mp3/SHA256E-s8034842--5c3475d7fef6f0c3545721f34e7cbfb6727a00708bdde192f0f1d53af251b444.mp3

I got around 200 files that should be deleted but not showing up in unused. I though maybe one of the dead repos is causing the problem so i did a,

    git annex forget --drop-dead

and synced all repos. Still I can not get them to drop.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.2.134"
 subject="comment 1"
 date="2013-09-13T15:34:03Z"
 content="""
The most likely explanation is that you have a branch in git that still contains the deleted file, which will prevent unused from removing it. Try `git branch -a` to see all branches (a tag could also do it).
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/FHnTlSBo1eCGJRwueeKeB6.RCaPbGMPr5jxx8A--#ce0d8"
 nickname="Hamza"
 subject="comment 2"
 date="2013-09-13T15:59:29Z"
 content="""
I did not create tags or branches, running returns,

    git branch -a

      git-annex
    * master
      synced/git-annex
      synced/master
      remotes/origin/HEAD -> origin/master
      remotes/origin/git-annex
      remotes/origin/master
      remotes/origin/synced/master

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.2.134"
 subject="comment 3"
 date="2013-09-13T16:17:39Z"
 content="""
Depending on your git configuration, `git annex sync` might not be updating certian of these branches, such as remotes/origin/HEAD. You can check the branches out and see which still contains your deleted files.
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/FHnTlSBo1eCGJRwueeKeB6.RCaPbGMPr5jxx8A--#ce0d8"
 nickname="Hamza"
 subject="comment 4"
 date="2013-09-13T17:57:38Z"
 content="""
Which one of those branches I can safely delete? this is a central repo all other clients drop their stuff here it does not have any remotes (I did clones it from a client then removed origin section from config.)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.2.134"
 subject="comment 5"
 date="2013-09-13T18:23:02Z"
 content="""
So this is a bare repository?

It seems to have had an origin at one point; see the remotes/origin/* branches. If it no longer has that remote, then of course those branches will not be getting updated, and so will contain the old files and that's why unused won't find them. In that case, you could safely remove the remotes/origin/* branches.
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/FHnTlSBo1eCGJRwueeKeB6.RCaPbGMPr5jxx8A--#ce0d8"
 nickname="Hamza"
 subject="comment 6"
 date="2013-09-13T18:41:30Z"
 content="""
It is a non bare repo, I am using the following setup,

http://git-annex.branchable.com/forum/Annex_dropping_files/
"""]]


code:
I've noticed that if I'm using git-assistant, it wants to pull down all my files from other repos onto my laptop, even after I've dropped them.  (My laptop is set up as "client," my usb drive and an ssh server as "backup".)

I want to use git annex to save space on my laptop, but of course when I'm running the assistant, it pulls everything down there, even things I've manually dropped.

Is my "I want to save space, with a partial archive on my laptop" use case simply out of scope for the assistant?  So I should just be using the command line for my needs?  That's fine if that's the case.

Or maybe something like this is what I should be doing?  http://git-annex.branchable.com/assistant/archival_walkthrough/  ?  so instead of manually "git annex drop"-ping files in place, I should set up a directory called "archive" on my machine, from which files will magically disappear and get backed up elsewhere?

If it's the case that a directory named "archive" in your checkout has the magical property of having the assistant drop and archive its contents, that's awesome, maybe just what I need, but if that behavior is spelled out in so many words anywhere I managed to miss it.

Apologies for all these questions, just enjoying the software immensely and wanting to get to know it.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn7gQ1zZDdWhXy9H51W2krZYShNmKL3qfM"
 nickname="Karsten"
 subject="comment 1"
 date="2012-11-30T08:26:24Z"
 content="""
I'm second this request. Also, I'm posting this to enable the rss comment subscription button for this post.

Maybe these design documents are interesting:

<http://git-annex.branchable.com/design/assistant/transfer_control/>

<http://git-annex.branchable.com/design/assistant/partial_content/>
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.6.49"
 subject="comment 2"
 date="2012-11-30T16:31:21Z"
 content="""
The archive directory is a new feature, but yes, it does work. Just use the webapp to edit the configuration of a remote, put it into the \"small archive\" group, and the contents of archive directories will be sent to it.

This is a particular application of [[preferred_content]] settings, which give you a large amount of control over which data end up where when using the assistant.
"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 3"
 date="2012-11-30T18:49:37Z"
 content="""
Very cool!

So not only is archive/ a magic subdirectory due to the \"preferred content settings\" for \"client,\" but *every* subdirectory named \"archive\" similarly drops content when possible.  That's awesome to know.

Also if I really wanted to manually manage all my content and still have git-annex assistant running, I could simply use \"present\" for my preferred content settings on my laptop, and it would respect my manual gets and drops.

Thank you!

"""]]


code: Back in January Joey [mentioned](http://git-annex.branchable.com/design/assistant/blog/day_179__brief_updates/) the [git-remote-gcrypt](https://github.com/blake2-ppc/git-remote-gcrypt) and possibly adding support for it in the Assistant. I think this would be a great addition. Now that the first big Android push is complete, is there a schedule for this feature?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-03-07T18:42:37Z"
 content="""
I have not put it on the schedule (and don't think I'll get to it this month), but I have added it to the [[roadmap|design/assistant]].

The main use cases it adds are encrypting git remotes on ssh servers, and on USB drives. Both are useful for sure..

It also opens the possibility of using github for encrypted git remotes, but with xmpp git push that is not a pressing use case. With significantly more work, encrypted git remotes could be stored on S3 etc, but again xmpp git push covers the same use cases.
"""]]


code:
[[!comment format=mdwn
 username="spwhitton"
 ip="82.36.235.9"
 subject="comment 2"
 date="2013-03-12T09:33:17Z"
 content="""
The use case this is really good for, that XMPP push can't help with, is for when two paired machines are almost never both turned on at once.
"""]]


code:
[[!comment format=mdwn
 username="annexuser"
 ip="50.125.45.235"
 subject="comment 3"
 date="2013-05-25T22:58:16Z"
 content="""
I'm still hoping to see this. The feature received some attention on the [April poll](http://git-annex.branchable.com/design/assistant/polls/goals_for_April/).
"""]]


code:
Hi all,

how can I get rid of all those 'typechange' messages in `git status`
when in direct mode?

Here is an example session:

    > git init
    Initialized empty Git repository in /some/path/.git/
    > git config user.name dtrn
    > git config user.email drn@drn.com
    > git annex init
    init  ok
    (Recording state in git...)
    > git annex direct
    commit
    # On branch master
    #
    # Initial commit
    #
    nothing to commit (create/copy files and use "git add" to track)
    ok
    direct  ok
    > dd if=/dev/zero of=testfile.bin count=1000
    1000+0 records in
    1000+0 records out
    512000 bytes (512 kB) copied, 0.00317424 s, 161 MB/s
    > git annex add testfile.bin
    add testfile.bin (checksum...) ok
    (Recording state in git...)
    > git commit -m "annexed testfile.bin"
    ok
    [master (root-commit) 281e740] annexed testfile.bin
     1 file changed, 1 insertion(+)
     create mode 120000 testfile.bin
    >  git status
    # On branch master
    # Changes not staged for commit:
    #   (use "git add <file>..." to update what will be committed)
    #   (use "git checkout -- <file>..." to discard changes in working directory)
    #
    #       typechange: testfile.bin
    #
    no changes added to commit (use "git add" and/or "git commit -a")

Regards,
Andreas


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4830:1600:187::2"
 subject="comment 1"
 date="2013-09-20T15:24:25Z"
 content="""
These messages are normal in direct mode.

They happen because git status does not know about git-annex's direct mode. So it sees a file that has a symlink checked into git, but a normal file in place in the working tree. Thus, its type has changed. Short of hacking or wrapping git, or switching to indirect mode ;), there's not much that can be done about this.

You might want to read [[direct_mode]] for more about this, and some of the problems it can cause when running certian git commands.
"""]]


code:
[[!comment format=mdwn
 username="ringprince"
 ip="134.76.140.110"
 subject="comment 2"
 date="2013-09-20T20:16:47Z"
 content="""
Thanks for the info and the quick reply.
"""]]


code:
Hi,

I am a happy user of [git-subtree](http://github.com/apenwarr/git-subtree), and I wonder whether it integrates nicely with git-annex?

My use-case looks like this: I have two annex repositories -- one at home and one at work. The annex at work is a strict subset of the one at home, i.e. all files that I have at work ought to be part of the annex at home, but not the other way round. Now, I realize that I could have one annex and selectively copy files to the checked out copy at work, but I don't want to do that because I don't want to have (broken) symlinks for all kinds of stuff visible on my machine at work that is not supposed to be there (such as MP3 files, etc.). Instead, I would like to use git-subtree to import the work annex into a sub-directory of the one at home, so that both annex are logically separate, but still the one at home always contains everything that the one at work contains.

Is that possible?

And if not, is there maybe another way to accomplish this kind of thing?


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2012-01-03T17:00:53Z"
 content="""
I have no experience using git-subtree, but as long as the home repository has the work one as a git remote, it will automatically merge work's git-annex branch with its own git-annex branch, and so will know what files are present at work, and will be able to get them.

Probably you won't want to make work have home as a remote, so work's git-annex will not know which files home has, nor will it be able to copy files to home (but home will be able to copy files to work).
"""]]


code:
[[!comment format=mdwn
 username="http://peter-simons.myopenid.com/"
 ip="77.188.44.113"
 subject="comment 2"
 date="2012-01-03T18:11:37Z"
 content="""
The point of git-subtree is that I can import another repository into sub-directory, i.e. I can have a directory called \"work\" that contains all files from the annex I have at work. If I make the other annex a remote and merge its contents, then all contents is going to be merged at the top-level, which is somewhat undesirable in my particular case.
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 3"
 date="2012-01-03T18:42:08Z"
 content="""
You can make it a remote without merging its contents. Git will not merge its contents by default unless it's named \"origin\". git-annex will be prefectly happy with that.
"""]]


code:
[[!comment format=mdwn
 username="http://peter-simons.myopenid.com/"
 ip="77.188.44.113"
 subject="comment 4"
 date="2012-01-03T19:17:36Z"
 content="""
Okay, I see, but is `git annex get --auto .` going to import all those files from the work remote into my home if the `master` branch of that remote isn't merged?
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 5"
 date="2012-01-03T19:31:45Z"
 content="""
Yes; git-annex uses the git-annex branch independently of the branch you have checked out. You may find [[internals]] interesting reading, but the short answer is it will work.
"""]]


code:
[[!comment format=mdwn
 username="http://peter-simons.myopenid.com/"
 ip="77.188.44.113"
 subject="comment 6"
 date="2012-01-03T19:47:11Z"
 content="""
Very cool! Thank you for the explanation.
"""]]


code:
Perhaps stupidly I added some very large bare git repos into a git-annex.

This took a very long time, used lot's of memory, and then crashed.  I didn't catch the error (which is annoying) - sorry about that.  IIRC it is the same error if one Ctrl-c's the addition.

I ran `git annex add .` a second time and eventually killed it (I perhaps should have waited - I now think it was working).

A `git annex unannex` fixed up some files but somehow I managed to end up with tonnes of files all sym-linked into the git annex object directory but not somehow recognised as annexed files.  I'm assuming that they somehow didn't make it into git annex's meta-data layer (or equivalent).

Commands such as `git annex {fsck,whereis,unannex} weirdfile` immediately returned without error.

I've now spent a lot of manual time copying the files back. Doing the following, not the cleverest but I was a little panicky about my data...

    find . -type l -exec mv \{} \{}.link \; #Move link names out of the way
    find . -type l -exec cp \{} \{}.cp \; #Copy follows links so we can copy target back to link location
    find . -type f -name "*.link.cp" | xargs -n 1 rename 's/\.link\.cp//' #Change to original name
    find . -type l -exec rm \{} \; #Ditch the links
    git annex unused
    git annex dropunused `seq 9228`

9228 files were found to be unused, this gives an idea of the scale of the number of "lost" files for want of a better term.

A pretty poor bug report as these things go.  Anyone any idea what might have happened (it didn't seem space or memory related)?  Or how I might have fixed it a little more cleverly?

For reference I am using stable Debian, git annex version 3.20111011.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnXybLxkPMYpP3yw4b_I6IdC3cKTD-xEdU"
 nickname="Matt"
 subject="comment 1"
 date="2011-12-06T12:50:27Z"
 content="""
Ah HA!  Looks like I found the cause of this.

    [matt@rss01:~/files/matt_ford]0> git annex add mhs
    add mhs/Accessing_Web_Manager_V10.pdf ok
    ....
    add mhs/MAHSC Costing Request Form Dual
    Organisations - FINAL v20 Oct 2010.xls git-annex: unknown response from git cat-file refs/heads/git-annex:8d5/ed4/WORM-s568832-m1323164214--MAHSC Costing Request Form Dual missing

Spot the file name with a newline character in it!  This causes the error message above.  It seems that the files proceeding this badly named file are sym-linked but not registered.

Perhaps a bug?
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 2"
 date="2011-12-06T17:08:37Z"
 content="""
The bug with newlines is now fixed.

Thought I'd mention how to clean up from interrupting `git annex add`.
When you do that, it doesn't get a chance to `git add` the files it's
added (this is normally done at the end, or sometimes at points in the middle when you're adding a *lot* of files).
Which is also why fsck, whereis, and unannex wouldn't operate on them, since they only deal with files in git.

So the first step is to manually use `git add` on any symlinks.

Then, `git commit` as usual.

At that point, `git annex unannex` would get you back to your starting state.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnXybLxkPMYpP3yw4b_I6IdC3cKTD-xEdU"
 nickname="Matt"
 subject="comment 3"
 date="2011-12-07T07:39:15Z"
 content="""
Ah - very good to know that recovery is easier than the method I used.

I wonder if it could be made a feature to automatically and safely recover/resume from an interrupted `git add`?
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 4"
 date="2011-12-07T20:54:51Z"
 content="""
Good idea! I've made `git annex add` recover when ran a second time.
"""]]


code:
I really like git-annex and am thankfull to its creator!

I especially like the high level of abstraction it provides when handling backups or syncing.

But git and my ext4 filesystem and HDD just hate the amount of files/folder/symlinks it throws at them and I am asking, what the alternatives are. I am especially looking for the numcopies and "--auto" functionallity.

Most alternatives I checked out (eps. distributed redundant filesystems) come up with other burdens or restrictions.

Do you have any experience with different systems?


code: Here's one thing I don't fully understand yet. If I add a remote repository, like an archive repository on Box—or if I want to change a transfer repository to an archive repository—do I need to add it or change it separately on each of my computers? Or just one?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.191"
 subject="comment 1"
 date="2014-04-17T20:59:43Z"
 content="""
This information is stored on the git-annex branch, and so it is synced between repositories with the rest of git-annex's data. You only need to change it in one place.. But it's also fine to change it in multiple places if necessary.
"""]]


code:
I am trying to share files between my PC at home at that at work using the walkthrough here: http://git-annex.branchable.com/assistant/remote_sharing_walkthrough/. However, I don't have the option on my machine to "Share with other devices". Any ideas why this would be missing? I am using Ubuntu 13.04 if that helps.

Update: it now works after a software update, I guess I just had an older version of git-annex. Now, I have version 4.20130723.


code:
I keep a repo synced between machines over ssh. Assuming all the files are in sync, so no actual file transfer needs to takes place, when I do

```
git annex copy --fast --quiet --to blah
```

is quite slow, about 10 seconds, using 100% CPU on one core, just to decide nothing needs to be done. On the other hand, doing

```
  git annex copy --fast --quiet --from blah
```

takes about 1 second.

I'm confused, as it seems to me that, since I'm using --fast, both transactions should use only locally available data, and both should need about the same amount of computing. Am I missing something? Can this be fixed?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.90"
 subject="comment 1"
 date="2013-07-17T19:11:38Z"
 content="""
How many files are in the directory tree you're copying?

`copy --fast --to` does indeed avoid the check to see if the remote already has the file before copying it.

However, it still needs to look in the location log to see which files are already present on the remote. Whereas `copy --from` can do a single stat of the file on disk to see if it's present in the local repo. Location log lookups are about as fast as I can make them, but they still require requesting info from out of the git repository. If you have a lot of files this otherwise minor difference in speed can stack up..
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkkyBDsfOB7JZvPZ4a8F3rwv0wk6Nb9n48"
 nickname="Abdó"
 subject="comment 2"
 date="2013-07-17T20:45:17Z"
 content="""
That example I gave: 10 sec vs 1 sec is on a repository of pictures with about 6200 files on a SSD.

Oh, I think I understand the source of the asymmetry, now! So, `git annex copy --to` queries the location log file by file? I've tested a `git grep` on the git-annex branch as follows

```
git grep -e <repo uuid> git-annex
```

and seems to be quite fast, less than a second on my test repo. Could git annex make use of this to speed up bulk queries to the location log?

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.140"
 subject="comment 3"
 date="2013-07-18T17:39:37Z"
 content="""
Using git grep like that would only be an optimisation when it's acting on every file in the repository. If it's only copying a subdirectory, or a single file, git grep is likely to come out much slower.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkkyBDsfOB7JZvPZ4a8F3rwv0wk6Nb9n48"
 nickname="Abdó"
 subject="comment 4"
 date="2013-07-18T22:58:24Z"
 content="""
I see. Thanks!
"""]]


code:
I just spend an hour to understand why the following command did not do anything:

    $ git annex copy --to someremote SomeDirectory

With SomeDirectory containing a lot of file that are not in the someremote repository.
I just realized that the annex from which I run the command did not contain those files either.

I then did

    $ git annex get SomeDirectory

So that now the git annex copy command correctly copies the files to the someremote remote.

I hope I won't fall into that trap again, but I think it is very annoying not to know why nothing happened.

Wouldn't it be great if the git annex copy command would indicate why some file are not copied, like:

    $ git annex copy --to someremote SomeDirectory
    Cannot copy SomeFile1 since it is not present in here, (hint: run git annex get first)
    Cannot copy SomeFile2 since it is not present in here, (hint: run git annex get first)
    Cannot copy SomeFile3 since it is not present in here, (hint: run git annex get first)
    ...

May be this kind of information could be provided if the verbose flag is set.

Another awesome (totally subjectively speaking) way of doing it could be to get the file before copying it

    $ git annex copy --to someremote SomeDirectory
    Cannot copy SomeFile1 since it is not present in here, attempting getting it first
    get SomeFile1 (from someotherremote...)
    SHA256E-s109353088--71734pq1p4qo6qs1p156r48s2290q7p61p1658029p103591nrs1rr708s064p59
         12,615,808 100%  161.27kB/s    0:01:16 (xfr#1, to-chk=0/1)
    ok
    copy SomeFile1 (checking someremote...) (to someremote...)
    SHA256E-s109353088--71734pq1p4qo6qs1p156r48s2290q7p61p1658029p103591nrs1rr708s064p59
         12,615,808 100%  161.27kB/s    0:01:16 (xfr#1, to-chk=0/1)
    ok
    Cannot copy SomeFile2 since it is not present in here, attempting getting it first
    get SomeFile2 (from someotherremote...)
    SHA256E-s109353088--71734pq1p4qo6qs1p156r48s2290q7p61p1658029p103591nrs1rr708s064p59
         12,615,808 100%  161.27kB/s    0:01:16 (xfr#1, to-chk=0/1)
    ok
    copy SomeFile2 (checking someremote...) (to someremote...)
    SHA256E-s109353088--71734pq1p4qo6qs1p156r48s2290q7p61p1658029p103591nrs1rr708s064p59
         12,615,808 100%  161.27kB/s    0:01:16 (xfr#1, to-chk=0/1)
    ok
    Cannot copy SomeFile3 since it is not present in here, attempting getting it first
    get SomeFile3 (from someotherremote...)
    SHA256E-s109353088--71734pq1p4qo6qs1p156r48s2290q7p61p1658029p103591nrs1rr708s064p59
         12,615,808 100%  161.27kB/s    0:01:16 (xfr#1, to-chk=0/1)
    ok
    copy SomeFile3 (checking someremote...) (to someremote...)
    SHA256E-s109353088--71734pq1p4qo6qs1p156r48s2290q7p61p1658029p103591nrs1rr708s064p59
         12,615,808 100%  161.27kB/s    0:01:16 (xfr#1, to-chk=0/1)
    ok

We could also specify from which remote to get the file with

    $ git annex copy --to someremote --from someotherremote SomeDirectory


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="206.74.132.139"
 subject="comment 1"
 date="2014-02-02T21:13:21Z"
 content="""
I think that this behavior of git-annex is quite useful once you get used to it. It can sometimes trip up new users, but new users would not know about some --explain switch that made it say why it skipped each file. So, I consider this a documentation issue, and I've added a section to the walkthrough to help users learn about it:

<http://git-annex.branchable.com/walkthrough/quiet_please:_When_git-annex_seems_to_skip_files/>

As far as copy --from --to, it has been suggested before; I think there is even a todo about it somewhere, but such remote-to-remote transfers are expensive and I would hope it would not normally be used.
"""]]


code:
Whenever I try to get a file with a non-English character, git annex gives me trouble that I don't have with English filenames from the same repo.  I'm hesitant to file a bug report here since the problem seems to stem from rsync not being able to handle the characters.  Here's the output (everything enclosed in pointy brackets are redacted):

    get <artist>/<album>/<track> (from origin...)
    git-annex: /home/<username>/Music/.git/annex/transfer/upload/53e11324-041f-11e2-93ca-27250d76416f/SHA256-s7941079--c5a31f15302d57563d8cd35e43ba34669b5485a99b617a83c89fba02fb2ca981: commitBuffer: invalid argument (invalid character)
    protocol version mismatch -- is your shell clean?
    (see the rsync man page for an explanation)
    rsync error: protocol incompatibility (code 2) at compat.c(174) [Receiver=3.0.9]

    git-annex-shell: sendkey: 1 failed
      rsync failed -- run git annex again to resume file transfer

      Unable to access these remotes: origin

      Try making some of these repositories available:
            604f093f-a980-4740-a9df-d21b580a2ba4 -- origin (<remote>)
    failed

It spits out similar errors for all files with Japanese/German/French/special characters, and ends with this on the last line:

    git-annex: get: 138 failed


code:
[[!comment format=mdwn
 username="sauluskahn"
 ip="67.171.218.222"
 subject="comment 1"
 date="2012-09-21T21:40:30Z"
 content="""
Also, git annex is 3.20120825 on both machines, and rsync is 3.0.9

Both have no problem handling the characters on their own, but when I try to rsync them I get trouble.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.14.141"
 subject="comment 2"
 date="2012-09-21T23:23:04Z"
 content="""
git-annex 3.20120825 is not an actual release, just some version number used in git head.

This bug, TTBOMK, was introduced in version 3.20120721, and fixed by git commit 0b12db64d834979d49ed378235b0c19b34e4a4d6 last Sunday.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.14.141"
 subject="comment 3"
 date="2012-09-21T23:23:52Z"
 content="""
I take that back about 3.20120825, it was so a real release! :) Anyway, bug is fixed in git head, and there will be a release along soon with it.
"""]]


code:
[[!comment format=mdwn
 username="sauluskahn"
 ip="67.171.218.222"
 subject="comment 4"
 date="2012-09-22T03:01:02Z"
 content="""
Sounds good, I'll get the next version through cabal and let you know if it's fixed.  Thanks for your time!
"""]]


code:
[[!comment format=mdwn
 username="sauluskahn"
 ip="67.171.218.222"
 subject="comment 5"
 date="2012-09-22T03:02:15Z"
 content="""
>I'll let you know if it's *not* fixed.

Typo.
"""]]