code:
Built a `git-recover-repository` command today. So far it only does the
detection and deletion of corrupt objects, and retrieves them from remotes
when possible. No handling yet of missing objects that cannot be recovered
from remotes.

Here's a couple of sample runs where I do bad things to the git
repository and it fixes them:

joey@darkstar:~/tmp/git-annex>chmod 644 .git/objects/pack/*
joey@darkstar:~/tmp/git-annex>echo > .git/objects/pack/pack-a1a770c1569ac6e2746f85573adc59477b96ebc5.pack
joey@darkstar:~/tmp/git-annex>~/src/git-annex/git-recover-repository
Running git fsck ...
git fsck found a problem but no specific broken objects. Perhaps a corrupt pack file? Unpacking all pack files.
fatal: early EOF
Unpacking objects: 100% (148/148), done.
Unpacking objects: 100% (354/354), done.
Re-running git fsck to see if it finds more problems.
Re-running git fsck to see if it finds more problems.
Initialized empty Git repository in /home/joey/tmp/tmprepo.0/.git/
Trying to recover missing objects from remote origin
Successfully recovered repository!
You should run "git fsck" to make sure, but it looks like
everything was recovered ok.

----

joey@darkstar:~/tmp/git-annex>chmod 644 .git/objects/00/0800742987b9f9c34caea512b413e627dd718e
joey@darkstar:~/tmp/git-annex>echo > .git/objects/00/0800742987b9f9c34caea512b413e627dd718e
joey@darkstar:~/tmp/git-annex>~/src/git-annex/git-recover-repository
Running git fsck ...
error: unable to unpack 000800742987b9f9c34caea512b413e627dd718e header
error: inflateEnd: stream consistency error (no message)
error: unable to unpack 000800742987b9f9c34caea512b413e627dd718e header
error: inflateEnd: stream consistency error (no message)
git fsck found 1 broken objects. Unpacking all pack files.
removing 1 corrupt loose objects
Re-running git fsck to see if it finds more problems.
Re-running git fsck to see if it finds more problems.
Initialized empty Git repository in /home/joey/tmp/tmprepo.0/.git/
Trying to recover missing objects from remote origin
Successfully recovered repository!
You should run "git fsck" to make sure, but it looks like
everything was recovered ok.

Works great! I need to move this and `git-union-merge` out of the git-annex
source tree sometime.

----

Today's work was sponsored by Francois Marier.


code:
Started work on [gcrypt](https://github.com/blake2-ppc/git-remote-gcrypt)
support.

The first question is, should git-annex leave it up to gcrypt to transport
the data to the encrypted repository on a push/pull? gcrypt hooks into git
nicely to make that just work. However, if I go this route, it limits
the places the encrypted git repositores can be stored to regular git
remotes (and rsync). The alternative is to somehow use gcrypt to
generate/consume the data, but use the git-annex special remotes to store
individual files. Which would allow for a git repo stored on S3, etc.
For now, I am going with the simple option, but I have not ruled out
trying to make the latter work. It seems it would need changes to gcrypt
though.

Next question: Given a remote that uses gcrypt, how do I determine the
annex.uuid of that repository. I found a nice solutuon to this. gcrypt has
its own gcrypt-id, and I convert it to a UUID in a
[[reproducible, and even standards-compliant way|design/gcrypt]]. So
the same encrypted remote will automatically get the same annex.uuid
wherever it's used. Nice. Does mean that git-annex cannot find a uuid
until `git pull` or `git push` has been used, to let gcrypt get the
gcrypt-id. Implemented that.

The next step is actually making git-annex store data on gcrypt remotes.
And it needs to store it encrypted of course. It seems best to avoid
needing a `git annex initremote` for these gcrypt remotes, and just have
git-annex automatically encrypt data stored on them. But I don't
know. Without initializing them like a special remote is, I'm limited to
using the gpg keys that gcrypt is configured to encrypt to, and cannot use
the regular git-annex hybrid encryption scheme. Also, I need to generate
and store a nonce anyway to HMAC ecrypt keys. (Or modify gcrypt
to put enough entropy in gcrypt-id that I can use it?)

Another concern I have is that gcrypt's own encryption scheme is simply
to use a list of public keys to encrypt to. It would be nicer if the
full set of git-annex encryption schemes could be used. Then the webapp
could use shared encryption to avoid needing to make the user set up a gpg
key, or hybrid encryption could be used to add keys later, etc.

But I see why gcrypt works the way it does. Otherwise, you can't make an
encrypted repo with a friend set as one of the particpants and have them be
able to git clone it. Both hybrid and shared encryption store a secret
inside the repo, which is not accessible if it's encrypted using that
secret. There are use cases where not being able to blindly clone a gcrypt
repo would be ok. For example, you use the assistant to pair with a friend
and then set up an encrypted repo in the cloud for both of you to use.

Anyway, for now, I will need to deal with
setting up gpg keys etc in the assistant. I don't want to tackle
full [[design/assistant/gpgkeys]] yet. Instead, I think I will start by
adding some simple stuff to the assistant:

* When adding a USB drive, offer to encrypt the repository on the drive
  so that only you can see it.
* When adding a ssh remote make a similar offer.
* Add a UI to add an arbitrary git remote with encryption.
  Let the user paste in the url to an empty remote they have,
  which could be to eg github. (In most cases this won't be used for
  annexed content..)
* When the user has no gpg key, prompt to set one up. (Securely!)
* Maybe have an interface to add another gpg key that can access the gcrypt
  repo. Note that this will need to re-encrypt and re-push the whole
  git history.

repo_name: kakafarm/git

license: unknown


code:
Solid day of working on repository recovery. Got `git recover-repository
--force` working, which involves fixing up branches that refer to missing
objects. Mostly straightforward traversal of git commits, trees, blobs, to
find when a branch has a problem, and identify an old version of it that
predates the missing object. (Can also find them in the reflog.)

The main complication turned out to be that `git branch -D` and `git
show-ref` don't behave very well when the commit objects pointed to by refs
are themselves missing. And git has no low-level plumbing that avoids
falling over these problems, so I had to write it myself.

Testing has turned up one unexpected problem: Git's index can itself refer
to missing objects, and that will break future commits, etc. So I need to
find a way to validate the index, and when it's got problems,
either throw it out, or possibly recover some of the staged data from it.


code:
I think that git-recover-repository is ready now. Made it deal with the
index file referencing corrupt objects. The best approach I could think of
for that is to just remove those objects from the index, so the user can
re-add files from their work tree after recovery.

Now to integrate this git repository repair capability into the git-annex
assistant. I decided to run `git fsck` as part of a scheduled
repository consistency check. It may also make sense for the assistant to
notice when things are going wrong, and suggest an immediate check. I've
started on the webapp UI to run a repository repair when fsck detects
problems.

[[!img /assistant/brokenrepositoryalert.png]]

[[!img /assistant/repairrepository.png]]

[[!meta title="the user interface I hope noone ever sees"]]


code:
[[!comment format=mdwn
 username="aarone"
 ip="2607:f470:8:d008:290:f5ff:fec3:a9e6"
 subject="Email alert"
 date="2013-10-22T21:23:24Z"
 content="""
I have at least one git-annex repository where I seldom open the web UI, but rather have the assistant humming along in the background.  I think for my use case it would be best to receive an email notification that the assistant had detected a problem, which would prompt me to open the web UI and perform the recovery.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="64.134.31.139"
 subject="excellent idea"
 date="2013-10-22T22:13:58Z"
 content="""
Added to my todo. I think it would require the system have an MTA though, I really don't want git-annex to grow to the point it can send email. ;)
"""]]


code:
[[!comment format=mdwn
 username="http://jasonwoof.com/"
 nickname="JasonWoof"
 subject="Please check if it needs help from the user first"
 date="2013-10-23T23:30:10Z"
 content="""
Awesome that you're working on recovery, and recovery automation!

Please only bother the user if there is a serious problem _and_ it can't be fixed without their help. Otherwise I fear people will learn to ignore your dialog boxes, like they do most dialog boxes. If you want to notify your user that some hard drive corruption happened (and it's been fixed already) then put a little yellow/orange line across somewhere with a warning message.

When I read to the part about how it can (probably) be fixed automatically, I got a flash of annoyance and thought \"well, then fix it automatically, why are you bothering me?\"

Please (if you aren't already) check if it can be automatically fixed without help from the user before telling the user about it. Then you can say \"The data to fix this could not be reached, please plug in another repo or something.\"
"""]]


code:
The webapp now fully handles repairing damage to the repository.

Along with all the git repository repair stuff already built, I added
additional repairs of the git-annex branch and git-annex's index file.
That was pretty easy actually, since git-annex already handles merging
git-annex branches that can sometimes be quite out of date. So when git repo
repair has to throw away recent changes to the git-annex branch, it just
effectively becomes out of date. Added a `git annex fsck --fast` run to
ensure that the git-annex branch reflects the current state of the
repository.

When the webapp runs a repair, it first stops the assistant from committing
new files. Once the repair is done, that's started back up, and it runs a
startup scan, which is just what is needed in this sitation; it will add
any new files, as well as any old files that the git repository damange
caused to be removed from the index.

Also made `git annex repair` run the git repository repair code,
for those with a more command-line bent. It can be used in non-git-annex
repos too!

----

So, I'm nearly ready to wrap up working on disaster recovery. Lots has been
accomplished this month. And I have put off making a release for entirely
too long!

The big missing piece is repair of git remotes located on removable drive.
I may make a release before adding that, but removable drives are probably
where git repository corruption is most likely to occur, so I certainly
need to add that.

----

Today's work was sponsored by Scott Robinson.


code:
Got well caught up on bug fixes and traffic. Backlog is down to 40.

Made the assistant wait for a few seconds before doing the startup
scan when it's autostarted, since the desktop is often busy starting
up at that same time.

Fixed an ugly bug with chunked webdav and directory special remotes
that caused it to not write a "chunkcount" file when storing data,
so it didn't think the data was present later. I was able to make it
recover nicely from that mistake, by probing for what chunks are actually
present.

Several people turn out to have had problems with `git annex sync` not
working because receive.denyNonFastForwards is enabled. I made the webapp
not enable it when setting up a ssh repository, and I made `git annex sync`
print out a hint about this when it's failed to push. (I don't think this
problem affects the assistant's own syncing.)

Made the assistant try to repair a damaged git repository without
prompting. It will only prompt when it fails to fetch all the lost
objects from remotes.

Glad to see that others have managed to
[get git-annex to build on Max OS X 10.9](http://git-annex.branchable.com/bugs/git_annex_doesn__39__t_work_in_Max_OS_X_10.9/#comment-8e8ee5e50506a6fde029d236f4809df8).
Now I just need someone to offer up a ssh account on that OS, and I could
set up an autobuilder for it.


code:
Finally got the assistant to repair git repositories on removable drives,
or other local repos. Mostly this happens entirely automatically, whatever
data in the git repo on the drive has been corrupted can just be copied
to it from `~/annex/.git`.

And, the assistant will launch a git fsck of such a repo whenever it fails
to sync with it, so the user does not even need to schedule periodic fscks.
Although it's still a good idea, since some git repository problems don't
prevent syncing from happening.

Watching git annex heal problems like this is quite cool!

One thing I had to defer till later is repairing corrupted gcrypt
repositories. I don't see a way to do it without deleting all the objects
in the gcrypt repository, and re-pushing everything. And even doing that
is tricky, since the `gcrypt-id` needs to stay the same.


code:
All command line stuff today..

Added --want-get and --want-drop, which can be used to test preferred content settings
of a repository. For example `git annex find --in . --want-drop` will list the same
files that `git annex drop --auto` would try to drop. (Also renamed `git annex content`
to `git annex wanted`.)

Finally laid to rest problems with `git annex unannex` when multiple files point to the
same key. It's a lot slower, but I'll stop getting bug reports about that.


code:
Spent today reviewing my [[plans_for_the_month|assistant/disaster_recovery]]
and filling in a couple of missing peices.

Noticed that I had forgotten to make repository repair clean up any stale
git locks, despite writing that code at the beginning of the month, and
added that in.

Made the webapp notice when a repository that is being used does not have
any consistency checks configured, and encourage the user to set up checks.
This happens when the assistant is started (for the local repository),
and when removable drives containing repositories are plugged in. If the
reminders are annoying, they can be disabled with a couple clicks.

And I think that just about wraps up the month. (If I get a chance, I would
still like to add recovery of git-remote-gcrypt encrypted git repositories.)

My [[design/roadmap]] has next month dedicated to user-driven features
and polishing and bugfixing.


code:
Low activity the past couple of days. Released a new version of git-annex
yesterday. Today fixed three bugs (including a local pairing one that was
pretty compicated) and worked on getting caught up with traffic.


code:
I've been investigating ways to implement a [[/todo/direct_mode_guard]].
Preventing a stray `git commit -a` or `git add` doing bad things in a
direct mode repository seems increasingly important.

First, considered moving `.git`, so git won't know it's a git repository.
This doesn't seem *too* hard to do, but there will certainly be unexpected
places that assume `.git` is the directory name.

I dislike it more and more as I think about it though, because it moves
direct mode git-annex toward being entirely separate from git, and I don't
want to write my own version control system. Nor do I want to complicate
the git ecosystem with tools needing to know about git-annex to work in
such a repository.

So, I'm happy that one of the other ideas I tried today seems quite
promising. Just set core.bare=true in a direct mode repository. This nicely
blocks all git commands that operate on the working tree from doing
anything, which is just what's needed in direct mode, since they don't know
how to handle the direct mode files. But it lets all git commands and other
tools that don't touch the working tree continue to be used. You can even
run `git log file` in such a repository (surprisingly!)

It also gives an easy out for anyone who really wants to use git commands
that operate on the work tree of their direct mode repository, by just
passing `-c core.bare=false`. And it's really easy to implement in
git-annex too -- it can just notice if a repo has core.bare and
annex.direct both set, and pass that parameter to every git command it
runs. I should be able to get by with only modifying 2 functions to
implement this.


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="70.82.37.38"
 subject="awesome!"
 date="2013-11-05T15:50:15Z"
 content="""
`core.bare=true` seems like an awesome idea!!! it probably means this can be factored into current installs without any change too right?
"""]]


code:
Long, long day coding up the direct mode guard today. About 90% of the fun
is dealing with `receive.denyCurrentBranch` not preventing pushes that
change the current branch, now that core.bare is set in direct mode.
My current solution to this involves using a special branch when using
direct mode, which nothing will ever push to (hopefully). A much nicer
solution would be to use a `update` hook to deny pushes of the current
branch -- but there are filesystems where repos cannot have git hooks.

The test suite is falling over, but the `directguard` branch otherwise
seems usable.

----

Today's work was sponsored by Carlo Matteo Capocasa.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnZEanlyzay_QlEAL0CWpyZcRTyN7vay8U"
 nickname="Carlo"
 subject="comment 1"
 date="2013-11-06T11:50:00Z"
 content="""
Thanks, Joey! I am very proud!
"""]]


code:
Woke up with a pretty solid plan for gcrypt. It will be structured as a
separate special remote, so `initremote` will be needed, with a gitrepo=
parameter (unless the remote already exists). git-annex will then set up
the git remote, including pushing to it (needed to get a gcrypt-id).

Didn't feel up to implementing that today. Instead I expectedly spent
the day doing mostly Windows work, including setting up a VM on my new
laptop for development. Including a ssh server in Windows, so I can
script local builds and tests on Windows without ever having to
touch the desktop. Much better!


code:
Started by tracking down a strange bug that was apparently
ubuntu-specific and caused git-annex branch changes to get committed to
master. Root cause turned out to failing to recover from an
exception. I'm kicking myself about that, because I remember looking at the
code where the bug was at least twice before and thinking "hmm, should add
exception handling here? nah..". Exceptions are horrible.

Made a release with a fix for that and a few minor other accumulated
changes since last Friday's release. The pain point of this release is to
fix building without the webapp (so it will propigate to Debian testing,
etc). This release does not include the direct mode guard, so I'll have a
few weeks until the next release to get that tested.

Fixed the test suite in `directguard`. This branch is now nearly ready to
merge to master, but one command that is badly needed in guarded direct
mode is "git status". So I am planning to rename "git annex status" to
"git annex info", and make "git annex status" display something similar
to "git status".

Also took half an hour and added optional [[EKG]] support to git-annex.
This is a Haskell library that can add a terrific monitoring console web
UI to any program in 2 lines of code. Here we can see the git-annex
webapp using resources at startup, followed in a few seconds by the
assistant's startup scan of the repository.

[[!img ekg/ekg.png]]

BTW, Kevin tells me that the machine used to build git-annex for OSX is
going to be upgraded to 10.9 soon. So, hopefully I'll be making autobuilds
of that. I may have to stop the 10.8.2 autobuilds though.

----

Today's work was sponsored by [Protonet](http://protonet.info/).


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/2grhJvAC049fJnvALDXek.6MRZMTlg--#eec89"
 nickname="John"
 subject="OS X builds"
 date="2013-11-07T05:12:13Z"
 content="""
Joey, were you not interested in my offer of an OS X build server that I've posted elsewhere on this list, and also in e-mail to you?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 2"
 date="2013-11-07T16:01:00Z"
 content="""
John, must have missed that; can't see to find it anywhere..
"""]]


code:
Finished the direct mode guard, including the new `git annex status`
command.

Spent the rest of the day working on various bug fixes. One of them turned
into rather a lot of work to make the webapp's UI better for git
remotes that do not have an annex.uuid.


code:
Been chipping away at my backlog of messages, and it's down to 23 items.

Finally managed to get ghc to build with a newer version of the NDK.
This *might* mean a solution to git-annex on Android 4.2. I
[need help with testing](http://git-annex.branchable.com/bugs/git-annex_broken_on_Android_4.3/#comment-90b82735cd6090a7765f423b743fffd3).


code:
Finally found the [root cause](http://git-annex.branchable.com/bugs/git-annex_broken_on_Android_4.3/#comment-452bee7d0a816300ccb4a34f9758134e)
of the Android 4.3/4.4 trouble, and a fix is now in place!

As a bonus, it looks like I've fixed a problem accessing the
environment on Android that had been worked around in an ugly way before.

Big thanks to my remote hands Michael Alan, Sören, and subito. All
told they ran 19 separate tests to help me narrow down this tricky
problem, often repeating long command lines on software keyboards.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmW0kg4uiMIhSHeVuvJFyo2VYMl7Qoej0s"
 nickname="Chris"
 subject="Woot!"
 date="2013-11-12T15:52:48Z"
 content="""
Thanks for getting this fixed.  I'm about to go on a long plane ride, and this will make it a lot easier for me to put movies on my tablet.
"""]]


code:
Annoyingly, the Android 4.3 fix breaks git-annex on Android 4.0 (probably
through 4.2), so I now have two separate builds of the Android app.

---

Worked on Windows porting today. I've managed to get the assistant
and watcher (but not yet webapp) to build on Windows.
The `git annex transferrer` interface needs POSIX stuff, and seems to be
the main thing that will need porting for Windows for the assistant to
work, besides of course file change detection. For that, I've hooked up
[Win32-notify](http://hackage.haskell.org/package/Win32-notify).

So the watcher might work on Windows.
At least in theory. Problem is, while all the code builds ok,
it fails to link:

	ghc.exe: could not execute: C:\Program Files (x86)\Haskell Platform\2012.4.0.0\lib/../mingw/bin/gcc.exe

I wonder if this is case of too many parameters being passed?

This happens both on the autobuilder and on my laptop, so I'm stuck here.
Oh well, I was not planning to work on this anyway until February...


code:
[[!comment format=mdwn
 username="http://nullroute.eu.org/~grawity/"
 nickname="Mantas"
 subject="comment 1"
 date="2013-11-13T06:22:49Z"
 content="""
[ProcExp](http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx) may be useful -- it can show process-related syscalls and their return codes _(among other things like file or registry access)_ so it should tell exactly why the exec failed.
"""]]


code:
One of my goals for this month is to get a better sense of how git-annex is
being used, how it's working out for people, and what areas need to be
concentrated on. To start on that, I am doing the
[2013 git-annex user survey](http://git-annex-survey.branchable.com/polls/2013/), similar to the git user
surveys. I will be adding some less general polls later (suggestions for
topics appreciated!), but you can go vote in any or all of 10 polls now.

----

Found a workaround for yesterday's Windows build problem. Seems that only
cabal runs gcc in a way that fails, so `ghc --make` builds is successfully.
However, the watcher doesn't quite work on Windows. It does get events when
files are created, but it seems to then hang before it can add the file to
git, or indeed finish printing out a debug log message about the event.
This looks like it could be a problem with the threaded ghc runtime on
Windows, or something like that.

Main work today was improving the git repository repair to handle corrupt
index files. The assistant can now start up, detect that the index file is
corrupt, and regenerate it all automatically.


code:
The user survey is producing some interesting and useful results!
Added two more polls: [using with](http://git-annex-survey.branchable.com/polls/2013/using_with/) and [blocking problems](http://git-annex-survey.branchable.com/polls/2013/blocking_problems/)
(There were some load issues so if you were unable to vote yesterday, try
again..)

Worked on getting the autobuilder for OS X Mavericks set up. Eventually
succeeded, after patching a few packages to work around a cpp that thinks
it should parse haskell files as if they're C code.
Also, Jimmy has resuscitated the OS X Lion autobuilder.

A not too bad bug in automatic merge conflict resolution has been reported,
so I will need to dig into that tomorrow. Didn't feel up to it today, so
instead have been spending the remaining time finishing up a branch that
switches the test suite to use the tasty test framework.


code:
Fixed two difficult bugs with direct mode. One happened (sometimes) when a
file was deleted and replaced with a directory by the same name and then
those changes were merged into a direct mode repository.

The other problem was that direct mode did not prevent writes to
.git/annex/objects the way that indirect mode does, so when a file in the
repository was not currently present, writing to the dangling symlink would
follow it and write into the object directory.

Hmm, I was going to say that it's a pity that direct mode still has so many
bugs being found and fixed, but the last real bug fix to direct mode was
made last May! Instead, I probably have to thank Tim for being a very
thorough tester.

Finished switching the test suite to use the tasty framework, and prepared
tasty packages for Debian.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkx5V3MTbzCXS3J7Mn9FEq8M9bPPYMkAHY"
 nickname="Pedro"
 subject="Is direct mode still using symlinks then?"
 date="2013-11-16T16:28:39Z"
 content="""
Is direct mode still using symlinks? I see the bug I opened is still unclosed:

http://git-annex.branchable.com/bugs/Direct_mode_repositories_still_use_symlinks_sometimes/

Any opinion on this? This is still my main problem with replacing dropbox with git-annex.
"""]]


code:
Release today, right on bi-weekly schedule. Rather startled
at the size of the changelog for this one; along with the direct mode
guard, it adds support for OS X Mavericks, Android 4.3/4.4, and fixes
numerous bugs.

Posted another question in the survey,
<http://git-annex-survey.branchable.com/polls/2013/roadmap/>.

Spun off git-repair as an independant package from git-annex. Of course,
most of the source code is shared with git-annex. I need to do something
with libraries eventually..


code:
About half way done with a gcrypt special remote. I can initremote it (the
hard part to get working), and can send files to it. Can't yet get files
back, or remove files, and only local repositories work so far, but this is
enough to know it's going to be pretty nice!

Did find one issue in gcrypt that I may need to develop a patch for:
<https://github.com/blake2-ppc/git-remote-gcrypt/issues/3>


code:
Wrote some evil code you don't want to run today. Git.Destroyer randomly
generates Damage, and applies it to a git repository, in a way that is
reproducible -- applying the same Damage to clones of the same git repo
will always yeild the same result.

This let me build a test harness for git-repair, which repeatedly clones,
damages, and repairs a repository. And when it fails, I can just ask it to
retry after fixing the bug and it'll re-run every attempt it's logged.

This is already yeilding improvements to the git-repair code.
The first randomly constructed Damage that it failed to recover
turned out to be a truncated index file that hid some other
corrupted object files from being repaired.

	[Damage Empty (FileSelector 1),
	 Damage Empty (FileSelector 2),
	 Damage Empty (FileSelector 3),
	 Damage Reverse (FileSelector 3),
	 Damage (ScrambleFileMode 3) (FileSelector 5),
	 Damage Delete (FileSelector 9),
	 Damage (PrependGarbage "¥SOH¥STX¥ENQ¥f¥a¥ACK¥b¥DLE¥n") (FileSelector 9),
	 Damage Empty (FileSelector 12),
	 Damage (CorruptByte 11 25) (FileSelector 6),
	 Damage Empty (FileSelector 5),
	 Damage (ScrambleFileMode 4294967281) (FileSelector 14)
	]

I need to improve the ranges of files that it damages -- currently QuickCheck
seems to only be selecting one of the first 20 or so files. Also, it's quite
common that it will damage `.git/config` so badly that git thinks it's not
a git repository anymore. I am not sure if that is something `git-repair`
should try to deal with.

---

Today's work was sponsored by the WikiMedia Foundation.


code:
Pushed out a minor release of git-annex today, mostly to fix build problems
on Debian. No strong reason to upgrade to it otherwise.

Continued where I left off with the Git.Destroyer. Fixed quite a lot of
edge cases where git repair failed due to things like a corrupted .git/HEAD
file (this makes git think it's not in a git repository), corrupt
git objects that have an unknown object type and so crash git hard, and
an interesting failure mode where git fsck wants to allocate 116 GB of
memory due to a corrupted object size header. Reported that last to the git
list, as well as working around it.

At the end of the day, I ran a test creating 10000 corrupt git
repositories, and **all** of them were recovered! Any improvements will
probably involve finding new ways to corrupt git repositories that my code
can't think of. ;)


code:
Still working on the git repair code. Improved the test suite, which found
some more bugs, and so I've been running tests all day and occasionally
going and fixing a bug in the repair code. The hardest part of repairing a
git repo has turned out to be reliably determining which objects in it are
broken. Bugs in git don't help (but the git devs are going to fix the one I
reported).

But the interesting new thing today is that I added some upgrade alert code
to the webapp. Ideally everyone would get git-annex and other software as
part of an OS distribution, which would include its own upgrade system --
But the  [survey](http://git-annex-survey.branchable.com/polls/2013/how_installed/)
tells me that a quarter of installs are from the prebuilt binaries I
distribute.

So, those builds are going to be built with knowledge of an upgrade url,
and will periodically download a small info file (over https) to see if a
newer version is available, and show an alert.

I think all that's working, though I have not yet put the info files in
place and tested it. The actual upgrade process will be a manual
download and reinstall, to start with, and then perhaps I'll automate it
further, depending on how hard that is on the different platforms.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawniayrgSdVLUc3c6bf93VbO-_HT4hzxmyo"
 nickname="Tobias"
 subject="gitannex-install"
 date="2013-11-22T07:13:02Z"
 content="""
To install and upgrade the prebuilt binary on Linux I use the fantastic [gitannex-install](https://github.com/zerodogg/scriptbucket/blob/master/gitannex-install) script, together with the upgrade alert this is just great.
"""]]


code:
[[!comment format=mdwn
 username="http://id.clacke.se/"
 nickname="clacke"
 subject="git-annex release annex"
 date="2013-11-25T06:15:23Z"
 content="""
Immediate thought: instead of an info file, why not sync a public git repo with releases annexed with URLs?
"""]]


code:
The difference picking the right type can make! Last night, I realized that
the where I had a `distributionSha256sum :: String`, I should instead use
`distributionKey :: Key`. This means that when git-annex is eventually
downloading an upgrade, it can treat it as just another Key being
downloaded from the web. So the webapp will show that transfer along with
all the rest, and I can leverage tons of code for a new purpose. For
example, it can simply fsck the key once it's downloaded to verify its
checksum.

Also, built a DistriutionUpdate program, which I'll run to generate the
info files for a new version. And since I keep git-annex releases in a
git-annex repo, this too leverages a lot of git-annex modules, and ended up
being just 60 easy lines of code. The upgrade notification code is tested
and working now.

And, I made the assistant detect when the git-annex program binary is
replaced or modified. Used my existing DirWatcher code for that. The plan
is to restart the assistant on upgrade, although I need to add some sanity
checks (eg, reuse the lsof code) first. And yes, this will work even for
`apt-get upgrade`!

----

Today's work was sponsored by Paul Tötterman


code:
Completely finished up with making the assistant detect when git-annex's
binary has changed and handling the restart.

It's a bit tricky because during an upgrade there can be two assistant
daemons running at the same time, in the same repository. Although I
disable the watcher of the old one first. Luckily, git-annex has long
supported running multiple concurrent git-annex processes in the same
repository.

The surprisingly annoying part turned out to be how to make the webapp
redirect the browser to the new url when it's upgraded. Particularly needed
when automatic upgrades are enabled, since the user will not then be taking
any action in the webapp that could result in a redirect. My solution to this
feels like overkill; the webapp does ajax long polling until it gets an
url, and then redirects to it. Had to write javascript code and ugh.

But, that turned out to also be useful when manually restarting the webapp
(removed some horrible old code that ran a shell script to do it before),
and also when shutting the webapp down.

[[!img assistant/downloadupgrade.png alt="assistant downloading an upgrade to itself"]]

Getting back to upgrades, I have the assistant downloading the upgrade, and
running a hook action once the key is transferred. Now all I need is some
platform-specific code to install it. Will probably be hairy, especially on
OSX where I need to somehow unmount the old git-annex dmg and mount the new
one, from within a program running on the old dmg.

----

Today's work was sponsored by Evan Deaubl.


code:
[[!comment format=mdwn
 username="abg"
 ip="184.75.210.234"
 subject="Upgrade installation in OSX"
 date="2013-11-24T00:28:03Z"
 content="""
Hey,

You don't have to implement any whacky DMG replacement logic for OSX upgrades. No sane user will be using the application from the DMG. They will drag the .app directory to /Applications or some non-standard location and unmount the DMG.  So your upgrade logic for OSX should end up looking very similar to the upgrade logic for Linux.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmuh_0-Mjz7QkVz7gT_7PRpCgmMcBFkj14"
 nickname="Wichert"
 subject="Running from DMG is not a sane use-case"
 date="2013-11-25T08:23:38Z"
 content="""
I don't think you should bother trying to support upgrades when people are running from a dmg. A DMG is just a disk image, and a format commonly used to distribute software. Running software from a DMG is not something anyone does beyond a quick first test of an application. A more reasonable thing to do, which many applications do now, is to offer to move the application from the DMG to /Applications if you detect someone is running it from a DMG.
"""]]


code:
[[!img assistant/upgradecomplete.png]]

Upgrades are fully working on Linux. OSX code is written but intested and I
thought of one bug it certainly has on my evening walk. Probably another
hour's work left later this evening to finish it off.


code:
Upgrades should be working on OSX Mavericks, Linux, and sort of on Android.
This needs more testing, so I have temporarily made the daily builds think
they are an older version than the last git-annex release. So when you
install a daily build, and start the webapp, it should try to upgrade
(really downgrade) to the last release. Tests appreciated.

Looking over the whole upgrade code base, it took 700 lines of code
to build the whole thing, of which 75 are platform specific (and mostly
come down to just 3 or 4 shell commands). Not bad..

----

Last night, added support for quvi 0.9, which has a completely changed
command line interface from the 0.4 version.

Plan to spend tomorrow catching up on bug reports etc and then low activity
for rest of the week.


code:
My last day before thanksgiving, getting caught up with some recent bug
reports and, quite a rush to get a lot of fixes in. Adding to the fun,
wintery weather means very limited power today.

It was a very productive day, especially for Android, which hopefully has
XMPP working again (at least it builds..), halved the size of the package,
etc.

Fixed a stupid bug in the automatic v5 upgrade code; annex.version was not
being set to 5, and so every git annex command was
actually re-running the upgrade.

Fixed another bug I introduced last Friday, which the test suite luckily
caught, that broke using some local remotes in direct mode.

Tracked down a behavior that makes `git annex sync` quite slow on
filesystems that don't support symlinks. I need to switch direct mode to
not using `git commit` at all, and use plumbing to make commits there.
Will probably work on this over the holiday.


code:
Made a release yesterday to fix a bug that made git-annex init in a bare
repository set core.bare=false. This bug only affected git-annex 5, it
was introduced when building the direct mode guard. Currently recovering
from it is a [manual (pretty easy) process](http://git-annex.branchable.com/bugs/assistant_creating_.git_directory_inside_bare_repo/#comment-73a8ce8aa100baa7c03861b769fdca29).
Perhas I should automate that, but I mostly wanted to get a fix out
before too many people encountered the bug.

Today, I made the assistant run batch jobs with ionice and nocache, when
those commands are available. Also, when the assistant transfers files,
that also runs as a batch job.

Changed how git-annex does commits, avoiding using `git commit` in direct
mode, since in some situations `git commit` (not with `-a`!) wants to
read the contents of files in the work tree, which can be very slow.


code:
Still working through thanksgiving backlog. Around 55 messages to go.

Wrote hairy code to automatically fix up bad bare repositories created by
recent versions of git-annex. Managed to do it with only 1 stat call
overhead (per local repository). Will probably keep that code in git-annex
for a year or so, despite the bug only being present for a few weeks,
because the repositories that need to be fixed might be on removable drives
that are rarely used.

Various other small bug fixes, including dealing with box.com having
changed their WebDAV endpoint url.

Spent a while evaluating various key/value storage possibilities.
[[bugs/incremental_fsck_should_not_use_sticky_bit]] has the details.


code:
gcrpyt is fully working now. *Most* of the examples in
[[tips/fully_encrypted_git_repositories_with_gcrypt]] should work.

A few known problems:

* `git annex sync` refuses to sync with gcrypt remotes. some url parsing issue.
* Swapping two drives with gcrypt repositories on the same mount point doesn't work yet.
* http urls are not supported


code:
[[!comment format=mdwn
 username="spwhitton"
 ip="82.36.235.9"
 subject="Converting an rsync special remote to a gcrypt special remote"
 date="2013-09-11T17:36:09Z"
 content="""
I'm guessing there is no way to convert an rsync special remote to a gcrypt special remote?  It would be cool not to have to upload 100GB across the Atlantic again!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.51"
 subject="comment 2"
 date="2013-09-12T16:53:02Z"
 content="""
Well, rsync and gcrypt use the same locations for annexed content, so it's theoretically possible.

However, it seems a lot easier to just add a gcrypt remote and let git-annex use the existing rsync remote for the content that is already stored in it.
"""]]


code:
The [2013 git-annex user survey](http://git-annex-survey.branchable.com/polls/2013/)
has been running for several weeks and around 375 people have answered
at least the first question. While I am going to leave it up through the
end of the year, I went over the data today to see what interesting
preliminary conclusions I can draw.

* 11% build git-annex from source. More than I would have guessed.

* 20% use the prebuilt versions from the git-annex website.

  This is a number to keep in mind later, when more people have upgraded to
  the last release, which checks for upgrades. I can run some stats on
  the number of upgrade checks I receive, and multiplying that by 5 would
  give a good approximation of the total number of computers running
  git-annex.

* I'm surprised to see so many more Linux (79%) than OSX (15%) users.
  Also surprising is there are more Windows (2%) than Android (1%) users.
  (Android numbers may be artificially low since many users will use it in
  addition to one of the other OSes.)

* Android and Windows unsurprisingly lead in ports requested, but the
  Synology NAS is a surprise runner up, with 5% (more than IOS).

  In theory it would not be too hard to make a standalone arm tarball,
  which could be used on such a device, although IIRC the Synology had
  problems with a too old linker and libc. It would help if I could make
  the standalone tarball not depend on the system linker at all.

  A susprising number (3%) want some kind of port the the Raspberry Pi, which
  is weird because I'd think they'd just be using Raspbian on it.. but a
  standalone arm tarball would also cover that use case.

* A minimum of 1664 (probably closer to 2000) git annex repositories are being
  used by the 248 people who answered that question. Around 7 repositories
  per person average, which is either one repository checked out on 7
  different machines or two repositories on 3 machines, etc.

* At least 143 terabytes of data are being stored in git-annex. This does
  not count redundant data. (It also excludes several hundred terabytes from
  one instituion that I don't think are fully online yet.)
  Average user has more than half a terabyte of data.

* 8% of users store scientific data in git-annex! :) A couple of users are
  using it for game development assets, and 5% of users are using it for
  some form of business data.

* Only 10% of users are sharing a git-annex repository with at least one
  other person. 27% use it by themselves, but want to get others using
  their repositories. This probably points to it needing to be easier for
  nontechnical users.

* 61% of git-annex users have good or very good knowledge of git.
  This question intentionally used the same wording as the
  [general git user survey](https://git.wiki.kernel.org/index.php/GitSurvey2012),
  so the results can be compared. The curves have somewhat different
  shapes, with git-annex's users being biased more toward the higher
  knowledge levels than git's users.

* The question about how happy users are also used the same wording.
  While 74% of users are happy with git-annex, 94% are similarly happy with
  git, and a while the median git-annex user is happy, the median git user
  is very happy.

  The 10% who wrote in "very enthusiastic, but still often
  bitten by quirks (so not very happy yet, but with lots of confidence in
  the potential" might have thrown off this comparison some, but they
  certianly made their point!

* 3% of respondants say that a bug is preventing them from using git-annex,
  but that they have not reported the bug yet. Frustrating! 1% say that a
  bug that's been reported already is blocking them.

* 18% wrote in that they need the webapp to support using github (etc)
  as a central server. I've been moving in that direction with the
  encryption and some other changes, so it's probably time to make a UI for
  that.

* 12% want more control over which files are stored locally when using the
  assistant.

* A really surprising thing happened when someone wrote in that I should
  work on "not needing twice disk space of repo in direct mode", and 5% of
  people then picked this choice. This is some kind of documentation
  problem, because of course git-annex never needs 2x disk space, whether
  using direct mode or not. That's one of its advantages over git!

* Somewhere between 59 and 161 of the survey respondants use Debian.
  I can compare this with [Debian popularity contest data](http://qa.debian.org/popcon-graph.php?packages=git-annex)
  which has 400 active installations and 1000 total installations,
  and make guesses about what fraction of all git-annex users have answered
  the survey. By making different assumptions I got guesses that varied by
  2 orders of magnitude, so not worth bothering with. Explicitly asking how
  many people use each Linux distribution would be a good idea in next
  year's survey.

----

Main work today was fixing Android DNS lookups, which was trying to use
/etc/resolv.conf to look up SRV records for XMPP, and had to be changed to
use a getprop command instead. Since I can't remember dealing with this
before (not impossible I made some quick fix to the dns library before and
lost it though), I'm wondering if XMPP was ever usable on Android before.
Cannot remember. May work now, anyway...


code:
Had planned to spend all day not working on git-annex and instead getting
caught up on conference videos. However, got a little bit multitasky while
watching those, and started investigating why, last time I worked on
Windows port, git-annex was failing to link.

A good thing to do while watching conference videos since it involved lots of
test builds with different flags. Eventially solved it.
Building w/o WebDAV avoids crashing the compiler anyhow.

Thought I'd try the resulting binary and see if perhaps I had forgotten to
use the threaded RTS when I was running ghc by hand to link it last time,
and perhaps that was why threads
[[seemed to have hung|day_56__git-annex_user_survey]] back then.

It was. This became clear when I saw a "deadlocked indefinitely in MVar"
error message, which tells me that it's at least using the threaded RTS.
So, I fixed that, and a few other minor things, and ran this command
in a DOS prompt box:

	git annex watch --force --foreground --debug

And I've been making changes to files in that repository, and amazingly,
the watcher is noticing them, and committing them!

So, I was almost entirely there to a windows port of the watcher a month
ago, and didn't know. It has some rough edges, including not doing anything
to check if a newly created file is open for write when adding it, and
getting the full assistant ported will be more work, and the full webapp
may be a whole other set of problems, but this is a quite nice milestone
for the Windows port.


code:
Got the entire webapp to build on Windows.

Compiling was easy. One line of code had to be #ifdefed out, and the whole
rest of the webapp UI just built!

Linking was epic. It seems that I really am runninginto a 32kb command line length
limit, which causes the link command to fail on Windows. git-annex with all
its bells and whistles enabled is just too big. Filed a
[ghc bug report](https://ghc.haskell.org/trac/ghc/ticket/8596), and got back a
helpful response about using <http://gcc.gnu.org/wiki/Response_Files> to
work around.

6 hours of slogging through compiling dependencies and fighting with
toolchain later, I have managed to link git-annex with the webapp!

The process is not automated yet. While I was able to automate
passing gcc a @file with its parameters, gcc then calls collect2, which
calls ld, and both are passed too many parameters. I have not found a way
to get gcc to generate a response file. So I did it manually. Urgh.

Also, it crashes on startup with `getAddrInfo` failure. But some more
porting is to be expected, now that the windows webapp links.. ;)


code:
Android has the EvilSplicer, now Windows gets the EvilLinker. Fully
automated, and truly horrible solution to the too long command line problem.

Now when I run `git annex webapp` on windows, it almost manages to open
the web browser.

At the same time, I worked with Yuri to upgrade the Windows autobuilder to a
newer Haskell platform, which can install Yesod. I have not quite achieved
a successful webapp build on the autobuilder, but it seems close.

----

Here's a nice Haskell exercise for someone. I wrote this quick and dirty
function in the EvilSplicer, but it's crying out for a generalized solution.

[[!format haskell """
{- Input contains something like
 - c:/program files/haskell platform/foo -LC:/Program Files/Haskell Platform/ -L...
 - and the *right* spaces must be escaped with \
 -
 - Argh.
 -}
escapeDosPaths :: String -> String
escapeDosPaths = replace "Program Files" "Program\\ Files"
        . replace "program files" "program\\ files"
        . replace "Haskell Platform" "Haskell\\ Platform"
        . replace "haskell platform" "haskell\\ platform"
"""]]


code:
Windows webapp now starts, opens a web browser, and ... crashes.

<img src="https://identi.ca/uploads/joeyh/2013/12/7/hDkkSA.png">

This  is [a bug in warp](https://github.com/yesodweb/wai/issues/202)
or a deep level of the stack. I know that yesod apps have run on Windows
before, so apparently something has changed and introduced this problem.

Also have a problem with the autobuilder; the EvilSplicer or something
it runs is locking up on that system for reasons not yet determined.

Looks like I will need to wait a bit longer for the windows webapp, but I
could keep working on porting the assistant in the meantime.

The most important thing that I need to port is how to check if a file
is being written to at the same time the assistant adds it to the
repository. No real `lsof` equivilant on Windows. I might be able to do
something with exclusive locking to detect if there's a writer (but this
would also block using the file while it was being added). Or I may be able
to avoid the need for this check, at least in direct mode.


code:
[[!comment format=mdwn
 username="http://nullroute.eu.org/~grawity/"
 nickname="Mantas"
 subject="comment 1"
 date="2013-12-08T12:52:26Z"
 content="""
Take a look at how Process Hacker implements the \"Search for open handles\" function; it is rather close to `lsof`. (Unlocker probably does it the same way, too.)

But locking (e.g. I think the \"share mode\" could prevent other programs from writing but not reading) may be more reliable, if a program starts writing to file *after* the lsof check happens...
"""]]


code:
I have seen the glory of the webapp running on Windows.

One of the warp developers pointed me in the right direction and I
developed a fix for the `recv` bug.

My Windows and MSIE are old and fall over on some of the
javascript, so it's not glorious enough for a screenshot. But large chunks
of it do seem to work.


code:
[[!comment format=mdwn
 username="https://launchpad.net/~maestro-alubia"
 nickname="maestro-alubia"
 subject="comment 1"
 date="2013-12-09T19:47:32Z"
 content="""
Yeah! Looking forward to finally introduce git-annex in my heterogeneous network. Thanks and keep up the good work!
"""]]


code:
Fixed up a few problems with the Windows webapp, and it's now completely
usable, using any browser other than MSIE. While there are missing
features in the windows port, all the UI for the features it does have
seems to just work in the webapp.

Fixed a ugly problem with Firefox, which turned out to have been introduced
a while ago by a workaround for an ugly problem in Chrome. Web browsers are
so wonderful, until they're crap.

Think I've fixed the bug in the EvilLinker that was causing it to hang on
the autobuilder, but still don't have a Windows autobuild with the webapp
just yet.

Also improved `git annex import` some more, and worked on a bug in git
repository repair, which I will need to spend some more time on tomorrow.


code:
Got the Windows autobuilder building the webapp. Have not tried that build
yet myself, but I have high hopes it will work.

Made other Windows improvements, including making the installer
write a start menu entry file, and adding free disk space checking.

Spent rest of the day improving git repair code on a real-world corrupted
repository.


code:
I've switched over to mostly working on Windows porting in the evenings
when bored, with days spent on other git-annex stuff. So, getting back to
the planned [[design/roadmap]] for this month..

Set up a [tip4commit for git-annex](http://tip4commit.com/projects/152).
Anyone who gets a commit merged in will receive a currently small amount of
bitcoin. This would almost be a good way to encourage more committers
other than me, by putting say, half the money I have earmarked for that into
the tip jar. The problem is, I make too many commits myself, so most of the
money would be quickly tipped back out to me! I have gotten in touch with the
tip4commit people, and hope they will give me a way to blacklist
myself from being tipped.

Designed a [[design/external_special_remote_protocol]] that seems pretty
good for first-class special remotes implemented outside git-annex.
It's moderately complicated on the git-annex side to make it simple and
flexible on the special remote side, but I estimate only a few days to build
it once I have the design finalized.

# windows

Tested the autobuilt windows webapp. It works! Sorted out some issues with
the bundled libraries.

Reworked how `git annex transferkeys` communicates, to make it easier to
port it to Windows. Narrowly managed to avoid needing to write Haskell
bindings to Windows's equivilant of `pipe(2)`. I think the Windows
assistant can transfer keys now. and the webapp UI may even be able to be
used to stop transfers. Needs testing.

Investigated what I'll need to get XMPP working on Windows. Most of the
libs are available in cygwin, but gsasl would need to be built from source.
Also some kind of space-in-path problem is preventing cabal installing some
of the necessary dependencies.


code:
Spent most of today catching up with a weeks's worth of traffic.

Fixed 2 bugs. Message backlog is 23 messages.


code:
Got git annex sync working with gcrypt. So went ahead and made a release
today. Lots of nice new features!

Unfortunately the linux 64 bit daily build is failing, because my build
host only has 2 gb of memory and it is no longer enough. I am looking for a
new build host, ideally one that doesn't cost me $40/month for 3 gb of ram
and 15 gb of disk. (Extra special ideally one that I can run multiple builds
per day on, rather than the current situation of only building overnight to
avoid loading the machine during the day.) Until this is sorted out, no
new 64 bit linux builds..


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/2grhJvAC049fJnvALDXek.6MRZMTlg--#eec89"
 nickname="John"
 subject="comment 1"
 date="2013-09-09T21:51:47Z"
 content="""
Joey, I'd be happy to host 64-bit builds on both Mac and Linux using my personal Jenkins server (both machines are local and have 16+ GB of RAM).  I might even be able to get you SSH jail access to the build tree for the 64-bit Linux build.  Just let me know at johnw@fpcomplete.com.
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/2grhJvAC049fJnvALDXek.6MRZMTlg--#eec89"
 nickname="John"
 subject="comment 2"
 date="2013-09-10T23:11:27Z"
 content="""
Everything is all setup now to build git-annex on both systems, at most once per hour (if there have been new Git commits), and only between 2am and 4pm.  Just contact me on IRC or e-mail and I'll setup you up with credentials so that you can manipulate the jobs and access the build artifacts.
"""]]


code:
Made some improvements to git-annex's plumbing level commands today. Added
new lookupkey and examinekey commands. Also expanded the things that
`git annex find` can report about files. Among other things, the elusive
hash directory locations can now be looked up, which IIRC a few people have
asked for a way to do.

Also did some work on the linux standalone tarball and OSX app. Both now
include man pages, and it's also now possible to just unpack it and symlink
git-annex into ~/bin or similar to add it to PATH.


code:
Made the Linux standalone builds more self-contained, now they include
their own linker and glibc, and ugly hacks to make them be used when
running the included programs. This should make them more portable
to older systems.

Set up an arm autobuilder.
This autobuilder runs in an Debian armel chroot, using
qemu-user-static (with a patch to make it support some syscalls ghc uses).
No webapp yet; waiting on feedback of how well it works. I *hope* this
build will be usable on eg, Synology NAS and Raspberry PI.

Also worked on improving the assistant's batching of commits during the
startup scan. And some other followups and bug triage.

Today's work was sponsored by Hamish Coleman.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnRai_qFYPVvEgC6i1nlM1bh-C__jbhqS0"
 nickname="Matthew"
 subject="Binaries"
 date="2013-12-17T12:49:26Z"
 content="""
Are there binaries for the RPi version anywhere, it would be great to try it.

Thanks
"""]]


code:
Fixed a few problems in the [[armel build|install/Linux_standalone]], and
it's been confirmed to work on Raspberry Pi and Synology NAS. Since none of
the fixes were specific to those platforms, it will probably work anywhere
the kernel is new enough. That covers 9+% of the
[missing ports in the user survey](http://git-annex-survey.branchable.com/polls/2013/missing_ports/)!

Thought through the possible issues with the assistant on Windows not being
able to use lsof. I've convinced myself it's probably safe. (In fact, it
might be safe to stop checking with lsof when using the assistant in direct
mode entirely.) Also did some testing of some specific interesting
circumstances (including 2 concurrent writers to a single file).

I've been working on adding the webapp to the armel build. This can mostly
reuse the patches and EvilSplicer developed for Android, but it's taking
some babysitting of the build to get yesod etc installer for various reasons.
Will be surprised if I don't get there tomorrow.

One other thing.. I notice that <http://git-annex.org/> is up and running.
This was set up by Subito, who offered me the domain, but I suggested he
keep it and set up a pretty start page that points new users at the
relevant parts of the wiki. I think he's done a good job with that!


code:
[[!comment format=mdwn
 username="Attila"
 ip="213.163.19.162"
 subject="git-annex.org"
 date="2013-12-27T14:14:07Z"
 content="""
Minor glitch on git-annex.org: the first link to http://git-annex.branchable.com is missing http:// prefix and so points to http://www.git-annex.org/git-annex.branchable.com
"""]]


code:
Got the arm webapp to build! (I have not tried to run it.) The build
process for this is quite elaborate; 2 chroots, one amd64 and one armel,
with the same versions of everything installed in each, and git-annex is
built in the first to get the info the EvilSplicer needs to build it in the
second.

Fixed a nasty bug in the assistant on OSX, where at startup it would follow
symlinks in the repository that pointed to directories outside the
repository, and add the files found there. Didn't cause data loss itself
(in direct mode the assistant doesn't touch the files), but certainly
confusingly breaks things and makes it easy to shoot your foot off. I will
be moving up the next scheduled release because of this bug, probably to
Saturday.

Looped the git developers in on a problem with git failing on some kernels
due to `RLIMIT_NOFILE` not working. Looks like git will get more robust and
this should make the armel build work on even more embedded devices.

Today's work was sponsored by Johan Herland.


code:
Resurfaced today to fix some problems with the Linux standalone builds
in the Solstice release. The worst of these prevented the amd64 build from
running on some systems, and that build has been updated. The other
problems all involved the binary shimming, and were less serious.

As part of that work, replaced the hacky shell script that handled the
linux library copying and binary shimming with a haskell program.

Also worked on some Windows bugs, and fixed a typo in the test suite.
Got my own little present: haskell-tasty finally got out of Incoming, so
the next Debian package build will once again include the test suite.


code:
Only did a few hours today, getting started on implementing the
[[design/external_special_remote_protocol]].

Mostly this involved writing down types for the various messages,
and code to parse them. I'm very happy with how the parsing turned out;
nearly all the work is handled by the data types and type classes,
and so only one line of very simple code is needed to parse each message:

[[!format haskell """
instance Receivable Response where
       parseCommand "PREPARE-SUCCESS" = parse0 PREPARE_SUCCESS
       parseCommand "TRANSFER-SUCCESS" = parse2 TRANSFER_SUCCESS
       parseCommand "TRANSFER-FAILURE" = parse3 TRANSFER_FAILURE
"""]]

An especially nice part of this implementation is that it knows exactly how
many parameters each message should have (and their types of course), and so
can both reject invalid messages, and avoid ambiguity in tokenizing the
parameters. For example, the 3rd parameter of TRANSFER-FAILURE is an error
message, and as it's the last parameter, it can contain multiple words.

	*Remote.External> parseMessage "TRANSFER-FAILURE STORE SHA1--foo doesn't work on Christmas" :: Maybe Response
	Just (TRANSFER_FAILURE Upload (Key {keyName = "foo", keyBackendName = "SHA1", keySize = Nothing, keyMtime = Nothing}) "doesn't work on Christmas")

That's the easy groundwork for external special remotes, done.


code:
Built most of the external special remote today. While I've written 600
lines of code for this, and think it's probably working, and complete
(except for a couple of features), all I know is that it compiles.

I've also written an [[example external special remote program in shell script|special_remotes/external/example.sh]],
so the next step is to put the two together and see how it works.
I also hope that some people who have built hook special remotes
in the past will update them to the new external special remote interface,
which is quite a lot better.

Today's work was sponsored by Justine Lam.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawleVyKk2kQsB_HgEdS7w1s0BmgRGy1aay0"
 nickname="Milan"
 subject="comment 1"
 date="2013-12-27T08:44:44Z"
 content="""
Will it support splitting of large files for remotes that limits size of stored file?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 2"
 date="2013-12-27T19:12:17Z"
 content="""
That's on the TODO list and the protocol should be easily flexible enough to let it be added. The existing code to handle chunks is a bit of a mess though.
"""]]


code:
The [[external_special_remote|special_remotes/external]] interface is now
done, and tested working great! Now we just need all the old hook special
remotes to be converted to use it..

I punted on per-special-remote, per-key state storage in the git-annex
branch for now. If I find an example of a remote that needs it (Tahoe-LAFS
may, but still TBD), I'll add it. Added suppport for using the same
credential storage that git-annex uses for S3 and WebDAV credentials.

The main improvement I'd like to make is to add an interface for transferring
files where the file is streamed to/from the external special remote,
rather than using temp files as it does now. This would be more efficient
(sometimes) and make the progress bars better. But it needs to either use a
named pipe, which is complicated and non-portable, or serialize the file's
contents over a currently line-based protocol, which would be a pain.
Anyway, this can be added later, the protocol is extensible.


code:
Fixed a bug that could leave a direct mode repository stuck at
annex.version 3. As part of that, v3 indirect mode repositories will be
automatically updated to v5. There's no actual change in that upgrade, it
just simplifies things to have only one supported annex.version.

Added youtube playlist support to git-annex. Seems I had almost all the
pieces needed, and didn't know it. Only about a dozen lines of code!

Added PREPARE-FAILURE support to the external special remote interface.

After I found the cable my kitten stole (her apport level is high), fixed
file transfers to/from Android. This broke because git-annex assistant
tries to use ionice, if it's in PATH, and Android's ionice is not suitable.
It could probably include ionice in the busybox build and use that one, but
I wanted a quick fix for this before the upcoming release.


code:
Worked on bug report and forum backlog (24 messages left), and made a few
bug fixes. The main one was a fix for a Windows-specific direct mode merge
bug.

This month didn't go entirely to plan. I had not expected to work on the
Windows assistant and webapp and get it so close to fully working. Nor had
I expected to spend time and make significant progress on porting git-annex
to *Linux* -- particularly to embedded NAS devices! I had hoped to
encourage some others to develop git-annex, but only had one bite from a
student and it didn't work out. Meanwhile, automatically rewarding
committers with bitcoin is an interesting alternative approach to possibly
motivating contributors, and I would like to set that up, but the software
is new and I haven't had time yet. The only thing that went exactly as
planned was the external special remote implementation.

A special surprise this month is that I have started hearing privately from
several institutions that are starting using git-annex in interesting ways.
Hope I can share details of some of that 2014!


code:
I've been out sick. However, some things kept happening. Mesar contributed
a build host, and the linux and android builds are now happening, hourly,
there. (Thanks as well to the two other people who also offered hostng.)
And I made a minor release to fix a bug in the test suite that I was pleased
three different people reported.

Today, my main work was getting git-annex to notice when a gcrypt remote
located on some removable drive mount point is not the same gcrypt remote
that was mounted there before. I was able to finesse this so it
re-configures things to use the new gcrypt remote, as long as it's a
special remote it knows about. (Otherwise it has to ignore the remote.)
So, encrypted repos on removable drives will work just as well as
non-encrypted repos!

Also spent a while with rsync.net tech support trying to work out why
someone's git-annex apparently opened a lot of concurrent ssh connections
to rsync.net. Have not been able to reproduce the problem though.

Also, a lot of catch-up to traffic. Still 63 messages backlogged however,
and still not entirely well..


code:
Various work on Debian, OSX, and Windows stuff. Mostly uninteresting, but
took most of the day.

Made `git annex mirror --all` work. I can see why I left it out; when the
mirroring wants to drop an object, in --all mode it doesn't have an
associated file in the tree, so it cannot look at the annex.numcopies in
gitattributes. Same reason why `git annex drop --all` is not implemented.
But decided to go ahead and only use other numcopies configuration for
mirroring.

Added GETWANTED and SETWANTED to the external special remote protocol, and
that is as far as I want to go on adding git-annex plumbing stuff to the
protocol. I expect Tobias will release a boatload of special remotes
updated to the new protocol soon, which seems to prove it has everything
that could reasonably be needed.

This is a nice public git-annex repository containing a growing collection
of tech conference videos. <https://github.com/RichiH/conference_proceedings>

Did some design work on [[todo/untracked_remotes]], which I think will turn
out to be read-only remotes. Being able to clone a repository and use
git-annex in the clone without anything leaking back upstream is often
desirable when using public repository, or a repository with many users.


code:
Implemented read-only remotes. This may not cover every use case around
wanting to clone a repository and use git-annex without leaking the
existence of your clone back to it, but I think it hits most of them in a
quite easy way, and allows for some potentially interesting stuff like
partitioned networks of git-annex repositories.

Zooko and I have been talking things over (for rather too long),
and I think have now agreed on a how a more advanced git-annex
Tahoe-LAFS special remote should work. This includes storing the
tahoe file-caps in the git-annex branch. So, I really need to add that
per-special-remote data storage feature I've been thinking about.


code:
Yesterday, added per-remote, per-key state storage. This is exported via
the external special remote protocol, and I expect to use it at least for
Tahoe Lafs.

Also, made the assistant write ssh config files with better permissions,
so ssh won't refuse to use them. (The only case I know of where that
happened was on Windows.)

Today, made `addurl` and `importfeed` honor annex.diskreserve. Found out about
this the hard way, when an importfeed cron job filled up my server with
youtube videos. I should probably also make `import` honor
annex.diskreserve.

---

I've been working, so far inconclusively, on making the assistant
deal with remotes that might open a long duration network connection.
Problem being that if the connection is lost, and the remote is not smart
enough to reconnect, all further use of it could fail.

In a `restarttransferrer` branch, I have made the assistant start separate
`transferkeys` processes for each remote. So if a remote starts to
fail, the assistant can stop its `transferkeys` process, and restart it,
solving the problem.

But, if a resource needed for a remote is not available, this degrades to
every transfer attempt to that remote restarting it. So I don't know if this is the
right approach.

Other approaches being considered include asking that implementors of
external special remotes deal with reconnection themselves (Tobias, do you
deal with this in your remotes?), or making the
assistant only restart failing remotes after it detects there's been
a network connection change.


code:
Spent ages tracking down a memory leak in the assistant that showed up when
a lot of files were added. Turned out to be a standard haskell laziness
induced problem, fixed by adding strictness annotations. Actually there
were several of them, that leaked at different rates. Eventually, I seem to
have gotten them all fixed:

Before: [[bugs/import_memleak_from_the_assistant/leakbefore.png]]
After: [[bugs/import_memleak_from_the_assistant/leakafter.png]]

Also fixed a bug in `git annex add` when the disk was completely full.
In that situation, it could sometimes move the file from the work tree to
.git/annex/objects and fail to put the symlink in place.


code:
Taught the assistant to stop reusing an existing `git annex transferkeys`
process after it detects a network connection change. I don't think this is
a complete solution to what to do about long-duration network connections
in remotes. For one thing a remote could take a long time to time out
when the network is disconnected, and block other transfers (eg to
local drives) in the meantime. But at least if a remote loses its network
connection and does not try to reconnect on its own, and so is continually
failing, this will get it back into a working state eventually.

Also, fixed a problem with the OSX Mavericks build, it seems that the
versions of wget and coreutils stuff that I was including in it were built
by homebrew with full optimisations turned on, so didn't work on some CPUs.
Replaced those with portable builds.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnX1msQxnLoSeu7q-i-c9BWghonsN7Qmns"
 nickname="Jan Ulrich"
 subject="comment 1"
 date="2014-01-07T16:02:01Z"
 content="""
Thanks for the Mavericks fix!
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnxp2XU8gIribhhGhGuYtU6eMMwHv5gUGI"
 nickname="Amitai"
 subject="obligatory "bah homebrew, yay pkgsrc" comment"
 date="2014-01-12T03:01:17Z"
 content="""
Bah homebrew, yay pkgsrc.

:-)
"""]]


code:
Been on reduced activity the past several days. I did spend a full day
somewhere in there building the Tahoe LAFS special remote. Also, Tobias has
finished updating his full suite of external special remotes to use the new
interface!

Worked on closing up the fundraising campaign today (long overdue).
This included adding a new wall-o-names to [[/thanks]].


code:
If you've been keeping an eye on the [[design/roadmap]], you'll have seen that
[[design/assistant/xmpp_security]] keeps being pushed back. This was because
it's a hard and annoying problem requiring custom crypto and with an
ugly key validation problem built into it too. I've now removed it from the
roadmap entirely, replacing it with a [[design/assistant/telehash]] design.

I'm excited by the possibilities of using telehash with git-annex. It seems
it would be quite easy to make it significantly more peer-to-peer and
flexible. The only issue is that telehash is still under heavy
development and the C implementation is not even usable yet..
(I'll probably end up writing Haskell bindings to that.)
So I've pushed it down the roadmap to at least March.

Spent the rest of the day making some minor improvements to external special
remote protocol and doing some other minor bug fixes and backlog catch up.
My backlog has exploded to nearly 50 messages remaining.

----

Today's work was sponsored by Chad Horohoe.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkGCmVc5qIJaQQgG82Hc5zzBdAVdhe2JEM"
 nickname="Bruno"
 subject="is telehash udp only?"
 date="2014-01-14T14:21:36Z"
 content="""
Is telehash UDP only?

I like that XMPP works with HTTPS so it also works at work with proxies that only allows http traffic.
"""]]


code:
Fixed a bug that one or two people had mentioned years ago, but I was never
able to reproduce myself or get anyone to reproduce in a useful way. It
caused log files that were supposed to be committed to the git-annex branch
to end up in master. Turned out to involve weird stuff when the environment
contains two different settings for a single variable. So was easily fixed
at last. (I'm pretty sure the code would have never had this bug if
Data.AssocList was not buried inside an xml library, which rather
discourages using it when dealing with the environment.)

Also worked on, and hopefully fixed, another OSX cpu optimisations problem.
This one involving shared libraries that git-annex uses for XMPP.

Also made the assistant detect corrupt .git/annex/index files on startup
and remove them. It was already able to recover from corrupt .git/index
files.

Today's work was sponsored by David Wagner.


code:
Activity has been a bit low again this week. It seems to make sense to do
weekly releases currently (rather than bi-monthly), and Thursday's
release had only one new feature (Tahoe LAFS) and a bunch of bug fixes.

Looks like git-annex will get back into Debian testing soon, after various
fixes to make it build on all architectures again, and then the
backport can be updated again too.

I have been struggling with a problem with the OSX builds, which fail with
a SIGKILL on some machines. It seems that homebrew likes to agressively
optimise things it builds, and while I have had some success with its
`--build-bottle` option, something in the gnutls stack used for XMPP is
still over-optimised. Waiting to hear back from Kevin on cleaning up some
optimised system libraries on the OSX host I use. (Is there some way to make
a clean chrooot on OSX that can be accessed by a non-root user?)

Today I did some minor work involving the --json switch, and also
a small change (well, under 300 line diff) allowing
--all to be mixed with options like --copies and --in.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkq2cjugiSvKWiWmcah3CPBqviQV_cin9I"
 nickname="Yury"
 subject="The world of Mac"
 date="2014-01-19T14:13:55Z"
 content="""
I think that there is nothing fundamental about Mac OS X, which would prevent one from making a clean chroot that would work for non-root users. For instance, I believe that jailkit has been reported to work just fine. The problem is that you'll have to rebuild most of the stuff you need from scratch, which is very tedious. Might be easier to find instances where Homebrew has something like -march=native added to the $CFLAGS (not that I particularly fancy Homebrew of all 'missing' package managers on OS X).

Anyways, it sounds like I'm going to get a Mac Mini donated sometime soon and I'm curious as to what would be the best way to make use of it for the CI system...
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnE6kFAbud1LWrQuyX76yMYnUjHt9tR-A8"
 nickname="Leonardo"
 subject="Chroots"
 date="2014-01-21T13:52:29Z"
 content="""
How about simply having sshd inside the chroot?
"""]]


code:
Worked to get git-remote-gcrypt included in every git-annex autobuild
bundle.
(Except Windows; running a shell script there may need some work later..)

Next I want to work on making the assistant easily able to create encrypted
git repositories on removable drives. Which will involve a UI to select
which gpg key to use, or creating (and backing up!) a gpg key.

But, I got distracted chasing down some bugs on Windows. These were
quite ugly; more direct mode mapping breakage which resulted in
files not being accessible. Also fsck on Windows failed to detect and fix
the problem. All fixed now. (If you use git-annex on Windows, you should
certainly upgrade and run `git annex fsck`.)

As with most bugs in the Windows port, the underlying cause turned out to
be stupid: `isSymlink` always returned False on Windows. Which makes sense
from the perspective of Windows not quite having anything entirely like
symlinks. But failed when that was being used to detect when files in the
git tree being merged into the repository had the symlink bit set..

Did bug triage. Backlog down to 32 (mostly messages from August).


code:
[[!comment format=mdwn
 username="http://nullroute.eu.org/~grawity/"
 nickname="Mantas"
 subject="comment 1"
 date="2013-09-13T22:05:37Z"
 content="""
Windows *does* have something very much like symlinks – they're called *symlinks* (\"symbolic links\") and they are meant to [\"function just like UNIX links\"][1] as the official docs say.

But on the other hand, yes – `isSymlink` makes less sense on Windows, because symlinks are just a type of reparse points, and there may be several other types (e.g. directory junctions should also be treated like symlinks, but mount points shouldn't...)

[1]: http://msdn.microsoft.com/en-us/library/windows/desktop/aa365680(v=vs.85).aspx
"""]]


code:
Yesterday I learned of a nasty bug in handling of merges in direct mode. It
turns out that if the remote repository has added a file, and there is a
conflicting file in the local work tree, which has not been added to git, the
local file was overwritten when git-annex did a merge. That's really bad, I'm
very unhappy this bug lurked undetected for so long.

Understanding the bug was easy. Fixing it turned out to be hard, because
the automatic merge conflict resolution code was quite a mess. In
particular, it wrote files to the work tree, which made it difficult for a
later stage to detect and handle the abovementioned case. Also, the
automatic merge resolution code had weird asymmetric structure that I never
fully understood, and generally needed to be stared at for an hour to begin
to understand it.

In the process of cleaning that up, I wrote several more tests,
to ensure that every case was handled correctly. Coverage was about 50%
of the cases, and should now be 100%.

To add to the fun, a while ago I had dealt with a bug on FAT/Windows where
it sometimes lost the symlink bit during automatic merge resolution. Except
it turned out my test case for it had a heisenbug, and I had not actually
fixed it (I think). In any case, my old fix for it was a large part
of the ugliness I was cleaning up, and had to be rewritten.
Fully tracking down and dealing with that took a large part of today.

Finally this evening, I added support for automatically handling merge
conflicts where one side is an annexed file, and the other side has the
same filename committed to git in the normal way. This is not an important
case, but it's worth it for completeness. There was an unexpected benefit
to doing it; it turned out that the weird asymmetric part of the code went
away.

The final core of the automatic merge conflict resolver has morphed from
a mess I'd not want to paste here to a quite consise and easy to follow
bit of code.

[[!format haskell """
        case (kus, kthem) of
                -- Both sides of conflict are annexed files
                (Just keyUs, Just keyThem) -> resolveby $
                        if keyUs == keyThem
                                then makelink keyUs
                                else do
                                        makelink keyUs
                                        makelink keyThem
                -- Our side is annexed file, other side is not.
                (Just keyUs, Nothing) -> resolveby $ do
                        graftin them file
                        makelink keyUs
                -- Our side is not annexed file, other side is.
                (Nothing, Just keyThem) -> resolveby $ do
                        graftin us file
                        makelink keyThem
                -- Neither side is annexed file; cannot resolve.
                (Nothing, Nothing) -> return Nothing
"""]]

Since the bug that started all this is so bad, I want to make a release
pretty soon.. But I will probably let it soak and whale on the test suite
a bit more first. (This bug is also probably worth backporting to old
versions of git-annex in eg Debian stable.)


code:
I've started a new page for my devblog, since I'm not focusing extensively
on the assistant and so keeping the blog [[here|design/assistant/blog]]
increasingly felt wrong. Also, my new year of
[crowdfunded development](https://campaign.joeyh.name)
formally starts in September, so a new blog seemed good.


code:
[[!comment format=mdwn
 username="http://pnijjar.livejournal.com/"
 ip="99.236.22.229"
 subject="comment 1"
 date="2013-08-31T00:05:16Z"
 content="""
Do we need to update our RSS feeds? I appear to be getting your devblog posts in my old feed, but I do not know whether that will continue working.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8"
 nickname="develop"
 subject="comment 2"
 date="2013-08-31T10:03:04Z"
 content="""
The old RSS feed will continue working.

So sit back, relax, and enjoy the show.
"""]]


code:
Pushed a release today. Rest of day spent beating head against Windows XMPP
brick wall.

Actually made a lot of progress -- Finally found the right approach, and
got a clean build of the XMPP haskell libraries. But.. ghc fails to
load the libraries when running Template Haskell.
"Misaligned section: 18206e5b".
Filed [a bug report](https://ghc.haskell.org/trac/ghc/ticket/8830),
and I'm sure this alignment problem can be fixed,
but I'm not hopeful about fixing it myself.

One workaround would be to use the EvilSplicer, building once without the
XMPP library linked in, to get the TH splices expanded, and then a second
time with the XMPP library and no TH. Made a `winsplicehack` branch with
tons of ifdefs that allows doing this. However, several dozen haskell libraries
would need to be patched to get it to work. I have the patches from
Android, but would rather avoid doing all that again on Windows.

Another workaround would be to move XMPP into a separate process from the
webapp. This is not very appealing either, the IPC between them would be
fairly complicated since the webapp does stuff like show lists of XMPP
buddies, etc. But, one thing this idea has to recommend it is I am already
considering using a separate helper daemon like this for
[[design/assistant/Telehash]].

So there could be synergies between XMPP and Telehash support, possibly
leading to some kind of plugin interface in git-annex for this sort of
thing. But then, once Telehash or something like it is available and
working well, I plan to deprecate XMPP entirely. It's been a flakey pain
from the start, so that can't come too soon.


code:
Normally, git-annex repositories consist of symlinks that are checked into
git, and in turn point at the content of large files that is stored in
`.git/annex/objects/`. Direct mode gets rid of the symlinks.

The advantage of direct mode is that you can access files directly,
including modifying them. The disadvantage is that most regular git
commands cannot be used in a direct mode repository.

Normally, git-annex repositories start off in indirect mode. With some
exceptions:

* Repositories created by the [[assistant]] use direct mode by default.
* Repositories on FAT and other less than stellar filesystems
  that don't support things like symlinks will be automatically put
  into direct mode.
* Windows always uses direct mode.

## enabling (and disabling) direct mode

Any repository can be converted to use direct mode at any time, and if you
decide not to use it, you can convert back to indirect mode just as easily.
Also, you can have one clone of a repository using direct mode, and another
using indirect mode.

To start using direct mode:

	git annex direct

To stop using direct mode:

	git annex indirect

## safety of using direct mode

With direct mode, you're operating without large swathes of git-annex's
carefully constructed safety net, which ensures that past versions of
files are preserved and can be accessed.
With direct mode, any file can be edited directly, or deleted at any time,
and there's no guarantee that the old version is backed up somewhere else.

So if you care about preserving the history of files, you're strongly
encouraged to tell git-annex that your direct mode repository cannot be
trusted to retain the content of a file. To do so:

	git annex untrust .

On the other hand, if you only care about the current versions of files,
and are using git-annex with direct mode to keep files synchronised between
computers, and manage your files, this should not be a concern for you.

## use a direct mode repository

You can use most git-annex commands as usual in a direct mode repository.

Direct mode also works well with the git-annex assistant.

The most important command to use in a direct mode repository is `git annex
sync`. This will commit any files you have run `git annex add` on, as well
as files that were added earlier and have been modified. It will push
the changes to other repositories for `git annex sync` there to pick up,
and will pull and merge any changes made on other repositories into the
local repository.

## what doesn't work in direct mode

A very few git-annex commands don't work in direct mode, and will refuse
to do anything. For example, `git annex unlock` doesn't make sense in
direct mode.

As for git commands, direct mode prevents using any git command that would
modify or access the work tree. So you cannot `git commit` or `git pull`
(use `git annex sync` for both instead), or run `git status` (use `git
annex status` instead). These git commands will complain "fatal: This
operation must be run in a work tree".

The reason for this is that git doesn't understand how git-annex uses the
work tree in direct mode. Where git expects the symlinks that get checked
into git to be checked out in the work tree, direct mode instead replaces
them with the actual content of files, as managed by git-annex.

There are still lots of git commands you can use in direct mode. For
example, you can run `git log` on files, run `git push`, `git fetch`,
`git config`, `git remote add` etc.

## forcing git to use the work tree in direct mode

This is for experts only. You can lose data doing this, or check enormous
files directly into your git repository, and it's your fault if you do!
Also, there should be no good reason to need to do this, ever.

Ok, with the warnings out of the way, all you need to do to make any
git command access the work tree in direct mode is pass it
`-c core.bare=false`


code:
[[!comment format=mdwn
 username="http://www.gl-como.it/author/valhalla/"
 nickname="valhalla"
 subject="Direct mode clone of an indirect repo"
 date="2013-08-18T08:47:35Z"
 content="""
I too have issues with mixing direct and indirect mode repositories.

 I have a regular, existing repository with ebooks, shared between various clones on proper :) filesystems; now I would need a copy of some of them on an ereader which only offers a FAT filesystem, so it has to be direct mode.

    mount $READER
    cd $reader
    git clone $REPO

I get a directory full of small files, the way git manages links on FAT.

    git annex init \"ebook reader\"

This detects the fact that it is working on a crippled filesystem, enables direct mode and disables ssh connection caching; up to now everything seems to be fine, but then

    git annex get $SOME_BOOK

seems to work, downloads the file somewhere, but when I try to open $SOME_BOOK it is still the fake link, and the file has been downloaded in its destination, as if the repo wasn't in direct mode.

I use version 4.20130723 on debian jessie
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.63"
 subject="comment 12"
 date="2013-08-23T17:48:54Z"
 content="""
There should be no obstacles to using direct mode on one clone of a git repository, and indirect mode on another clone. The data stored in git for either mode is identical, and I do this myself for some repositories.

@valhalla, you probably need to run `git annex fsck`, and if that does not solve your problem, you need to file a bug report.
"""]]


code:
[[!comment format=mdwn
 username="http://mildred.fr/"
 ip="82.247.184.53"
 subject="Fixing symlinks to the annex store in direct mode"
 date="2014-07-16T06:52:33Z"
 content="""
I have an issue with direct mode: I have tons of symlinks that points to the git-annex store, while in direct mode. After investigation, I found that these files don't seem to be part of the repository. I can check with:

    $ git -c core.bare=false status --porcelain tr.html
    ?? Documentation/Mozilla/developer.mozilla.org/tr.html

    $ ls -l tr.html
    lrwxrwxrwx 1 mildred mildred 205 10 juin  16:22 tr.html -> ../../../.git/annex/objects/gF/z1/SHA256E-s31895--c873982bd742ba8db6e026afee26b7ab2f75f54f587304d8c2d877db3900c0f6.html/SHA256E-s31895--c873982bd742ba8db6e026afee26b7ab2f75f54f587304d8c2d877db3900c0f6.html

The link is valid, and is probably pointing to a unused file in the annex store. How to add these files back?

If I was in indirect mode, I could simply use `git add tr.html` (and **not** `git annex add`). This would stage to the git staging area the symlink, and all would be well.

I found that in direct mode, the same was true. The command is:

    git -c core.bare=false add tr.html

The file is added to the repository, and the link is converted to the target file. The question now is why? There is no hook on the add command. Could it be the annex assistant? If that didn't worked, my question would have been: how to checkout a file in direct mode?
"""]]


code:
[[!comment format=mdwn
 username="http://mildred.fr/"
 ip="82.247.184.53"
 subject="Re: Fixing symlinks to the annex store in direct mode"
 date="2014-07-16T07:04:08Z"
 content="""
What I said was just wrong. Instead of trying on tr.html, I tried with a copy of that file. But making the copy of that file had the wanted effect, transforming the symlink to the actual file. Probably the assistant monitoring file creation, and transforming indirect file to direct file. Or perhaps the `cp` command follows symlinks by default (I thought it didn't).
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.63"
 subject="comment 13"
 date="2013-08-23T17:50:15Z"
 content="""
@obergix asked:

> But then, how can a direct repo sync with changes made in other remotes, if there no pull/fetch available.

The answer is simple: By running `git annex sync`, which handles all that.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 14"
 date="2014-07-16T17:51:28Z"
 content="""
I kinda wish people would post questions to the forum, and not clutter up this page..

Anyway, there have been past bugs in the direct mode code that caused some files to not be checked out in direct mode, but stay as symlinks pointing at the content. That can be fixed by running `git annex fsck`. But, I am not aware of any problem that can leave a git-annex symlink that is not checked into git at all. Perhaps you copied the symlink from another location?
"""]]


code:
[[!comment format=mdwn
 username="http://olivier.berger.myopenid.com/"
 nickname="obergix"
 subject="Git annex copy needed before git annex sync"
 date="2013-08-23T19:59:35Z"
 content="""
Thanks for these details @joeyh. But AFAIU, one needs to proceed to the git annex copy before doing the git annex sync, otherwise, symlinks (or files containing the symlink path on SMB) will be created, instead of the plain \"direct\" files that are expected.

I'm still not sure whether the git annex sync needs to be issued on either of the indirect or direct remotes first, or both, then in which sequence. I think a \"walkthrough\" script would help.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.63"
 subject="comment 15"
 date="2013-08-24T15:56:47Z"
 content="""
No, you can sync before you copy, get, or whatever. git-annex will replace the symlinks with the actual files when they arrive at the repository.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl2Jj8q2upJL4ZQAc2lp7ugTxJiGtcICv8"
 nickname="Michael"
 subject="comment 3"
 date="2013-02-19T03:03:14Z"
 content="""
So, if I edit a \"content file\" (change a music file's metadata, say), what's the workflow to record that fact and then synchronise it to other repositories?

I can't do a `git add`, so I don't understand what has to happen as a first step.  (Thanks for your quick reply above, BTW.)

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.183"
 subject="comment 4"
 date="2013-02-19T03:05:35Z"
 content="""
<pre>
git annex add $file
git annex sync
git annex copy $file --to otherrepo
</pre>
"""]]


code:
[[!comment format=mdwn
 username="mildred"
 ip="2a01:e35:2f7b:8350:225:22ff:fe40:fdfc"
 subject="What happens to object database in direct mode?"
 date="2013-07-08T13:27:21Z"
 content="""
What happens to the object database (`.git/annex/objects`) when going to direct mode? Are the objects deleted, moved to another location, kept?

If the objects are kept, does it means that the file on the repository in direct mode is duplicated in the object database? If so, would it be relevant to use `cp --reflink=auto` to populate the working directory to enable copy on write on filesystems that supports it?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.254.222"
 subject="comment 6"
 date="2013-07-08T16:11:32Z"
 content="""
`.git/annex/objects` does not typically contain any file contents in direct mode. The file contents are stored directly in the working tree.
"""]]


code:
[[!comment format=mdwn
 username="http://caust1c.myopenid.com/"
 nickname="asbraithwaite"
 subject="comment 7"
 date="2013-08-12T18:06:21Z"
 content="""
Would it be safe to add largefiles to gitignore in direct mode?

Can git-annex still track large files ignored by git?

Thanks. :-)
"""]]


code:
[[!comment format=mdwn
 username="arand"
 ip="130.243.226.21"
 subject="comment 8"
 date="2013-08-12T18:12:32Z"
 content="""
asbraithwaite:
No, as far as I know it can not.
"""]]


code:
In git, there can be multiple clones of a repository, each clone can
be independently modified, and clones can push or pull changes to
one-another to get back in sync.

git-annex preserves that fundamental distributed nature of git, while
dropping the requirement that, once in sync, each clone contains all the data
that was committed to each other clone. Instead of storing the content
of a file in the repository, git-annex stores a pointer to the content.

Each git-annex repository is responsible for storing some of the content,
and can copy it to or from other repositories. [[Location_tracking]]
information is committed to git, to let repositories inform other
repositories what file contents they have available.

---

The [[walkthrough]] shows how to create a distributed set of git-annex
repositories with no central repository.

Prefer a central repository like GitHub? See the
[[tips/centralized_git_repository_tutorial]].


code:
The main git repository for git-annex is `git://git-annex.branchable.com/`

(You can push changes to this wiki from that anonymous git checkout.)

Other mirrors of the git repository:

* `git://git.kitenet.net/git-annex` [[gitweb](http://git.kitenet.net/?p=git-annex.git;a=summary)]
* [at github](https://github.com/joeyh/git-annex)

Releases of git-annex are uploaded
[to hackage](http://hackage.haskell.org/package/git-annex). Get your
tarballs there, if you need them.

Some operating systems include git-annex in easily prepackaged form and
others need some manual work. See [[install]] for details.

## git branches

The git repository has some branches:

* `ghc7.0` supports versions of ghc older than 7.4, which
  had a major change to filename encoding.
* `old-monad-control` is for systems that don't have a newer monad-control
  library.
* `no-ifelse` avoids using the IFelse library
  (merge it into master if you need it)
* `no-bloom` avoids using bloom filters. (merge it into master if you need it)
* `no-s3` avoids using the S3 library (merge it into master if you need it)
* `debian-stable` contains the latest backport of git-annex to Debian
  stable.
* `tweak-fetch` adds support for the git tweak-fetch hook, which has
  been proposed and implemented but not yet accepted into git.
* `setup` contains configuration for this website
* `pristine-tar` contains [pristine-tar](http://kitenet.net/~joey/code/pristine-tar)
  data to create tarballs of any past git-annex release.

----

Developing git-annex? Patches are very welcome.
You should read [[coding_style]].