code:
I've noticed people have some problems getting me logs when there'a a bug,
so I worked on improving the logging of the assistant.

While the assistant logged to `.git/annex/daemon.log` when started as a
daemon, when the webapp ran it didn't log there. It's somewhat tricky to
make the webapp redirect messages to the log, because it may start a web
browser that wants to write to the console. Took some file descriptor
juggling, but I made this work. Now the log is even used when the assistant
is started for the first time in a newly created repository. So, we have
total log coverage.

Next, I made a simple page in the webapp to display the accumulated logs.
It does not currently refresh as new things are logged. But it's easier
for me to tell users to click on `Current Repository -> View log` than
ask for them to look for the daemon.log file.

Finally, I made all the webapp's alerts also be written to the log.

---

Also did the requisite bug fixes.

[[!img /assistant/logs.png alt="screenshot of logs page"]]


code:
Only one bug fix today, but it was a doozie. It seems that gpg2 has an
incompatability with the gpg 1.x that git-annex was written for, that
causes large numbers of excess passphrase prompts, when it's supposed to be
using a remote's symmetric encryption key. Adding the --batch parameter
fixed this.

I also put together a page listing [[related_software]] to git-annex.

I've also updated [[direct_mode]]'s documentation, about when it's safe to
use direct mode. The intuition I've developed about direct mode is that if
you don't need full versioning of files (with the ability to get back old
versions), direct mode is fine and safe to use. If you want full
versioning, it's best to not use direct mode. Or a reasonable compromise is
to `git annex untrust` the direct mode repository and set up a backup remote.
With this compromise, only if you edit a file twice in a row might the old
version get lost before it can be backed up.

Of course, it would be possible to make direct mode fully version
preserving, but it'd have to back up every file in the repository locally
to do so. Going back to two local copies of every file, which is part of
git that git-annex happily avoids. Hmm, it might be possible to only back
up a file locally until it reaches the backup remote..

repo_name: kakafarm/git

license: unknown


code:
[[!comment format=mdwn
 username="https://launchpad.net/~rubiojr"
 nickname="rubiojr"
 subject="comment 1"
 date="2013-01-18T18:42:35Z"
 content="""
Awesome @joey, crisp clear.

I guess that most people used to the Dropbox way of doing things will not miss versioning that much, which is not very useful for medium/large binary files in most cases I'd say.

Being able to choose the mode for power users is pretty cool though.
"""]]


code:
When I wasn't dealing with the snowstorm today, I was fixing more bugs.
Rather serious bugs.

One actually involved corruption to git-annex's location tracking info, due
to a busted three-way merge. Takes an unusual set of circumstances for that
bug to be triggered, which is why it was not noticed before now. Also,
since git-annex is designed to not trust its location tracking info, and
recover from it becoming inconsistent, someone could have experienced the
bug and not really noticed it. Still it's a serious problem and I'm in debt
to user a-or-b for developing a good test case that let me reproduce it and
fix it. (Also added to the test suite.)
[[This is how to make a perfect bug report|bugs/Annex_thinks_file_exists_afer_being_dropped]]

Another bug made `git add; git commit` cause data loss in direct mode.
I was able to make that not lose data, although it still does something
that's unlikely to be desired, unless the goal is to move a file from an
annexed direct mode file to having its entire contents stored in git.

Also found a bug with sync's automatic resolution of git conflicts. It
failed when two repositories both renamed a file to different names.
I had neglected to explicitly `git rm` the old file name, which is
necessary to resolve such a conflict.


code:
On Friday, worked on several bugs in direct mode mapping code. Fixed it
to not crash on invalid unicode in filenames. Dealt with some bugs when
mappings were updated in subdirectories of the repository.

Those bugs could result in inconsistent mapping files, so today I
made `fsck` check mapping files for consistency.

Leaving for Australia tomorrow, but I also hope to get another bugfix
release out before my flight leaves. Then will be on vacation for several
days, more or less. Then at Linux Conf Australia, where there will
be a git-annex presentation on February 1st.

----

BTW, I've lined up my Android development hardware for next month. I will
be using an [Asus Transformer](http://en.wikipedia.org/wiki/ASUS_Eee_Pad_Transformer),
kindly lent to me by Mark H. This has the advantage
of having a real keyboard, and running the (currently) second most widely used
version of Android, 4.0.x. I have already experienced frustration getting photos
off the thing and into my photo annex; the file manager is the worst I've
seen since the 80's. I understand why so many want an Android port!

Interestingly, its main user filesystem is a FUSE mount point on `/sdcard`
backed by an ext4 filesystem on `/data` that a regular user is not allowed
to access. Whatever craziness this entails does not support symlinks.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 1"
 date="2013-01-19T20:40:52Z"
 content="""
The Nexus devices all come without support for SD cards and without FAT anywhere to be seen. Indirect mode should work nicely on those.

-- Richard
"""]]


code:
[[!comment format=mdwn
 username="EskildHustvedt"
 ip="84.48.83.221"
 subject="Presentation video"
 date="2013-01-19T22:09:50Z"
 content="""
Any chance your presentation will be filmed and made available?
"""]]


code:
[[!comment format=mdwn
 username="hamish"
 ip="203.0.139.23"
 subject="AU"
 date="2013-01-24T12:50:06Z"
 content="""
I dont know how busy your LCA week is looking, but I'd love to shout you a beer one evening - perhaps there would also be other git-annex people hanging around the conference and interested too
"""]]


code:
15 hours in a plane with in-seat power. Ok, time for some new features!

Added two new repository groups.

"manual" can be used to avoid the assistant downloading any file contents
on its own. It'll still upload and otherwise sync data. To download files,
you can use `git annex get` while the assistant is running. You can also
drop files using the command line.

"source" is for repositories that are the source of new files, but don't
need to retain a copy once the file has been moved to another repository.
A camera would be a good example.

Ok, those were easy features to code; I suck at being productive on planes.
Release coming up with those, once I find enough bandwidth here in AU.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlyFDd2xkx0uj738OArDj10CqEkVfWu_mQ"
 nickname="Johann"
 subject="comment 1"
 date="2013-01-24T15:36:34Z"
 content="""
Regarding 'manual': Maybe it is necessary to adjust the other standard settings for preferred content then.
For the 'transfer' for example it would still make sense to treat a manual repository as a client.
"""]]


code:
Got back to hacking today, and did something I've wanted to do for some
time. Made all the assistant's threads be managed by a thread manager. This
allows restarting threads if they crash, by clicking a button in the
webapp. It also will allow for other features later, like stopping and
starting the watcher thread, to pause the assistant adding local files.

[[!img /assistant/crashrecovery.png]]

I added the haskell async library as a dependency, which made this pretty
easy to implement. The only hitch is that async's documentation is not
clear about how it handles asyncronous exceptions. It took me quite a while
to work out why the errors I'd inserted into threads to test were crashing
the whole program rather than being caught!


code:
Hacking on a bus to Canberra for [LCA2013](https://lca2013.linux.org.au/),
I made the webapp's UI for pausing syncing to a repository also work for
the local repository. This pauses the watcher thread. (There's also an
annex.autocommit config setting for this.)

Ironically, this didn't turn out to the use the thread manager I built
yesterday. I am not sure that a ThreadKilled exception would never be
masked in the watcher thread. (There is some overly broad exception
handling in git-annex that dates to back before I quite understood haskell
exceptions.)


code:
Not doing significant coding here at LCA2013, but stuff is still happening:

* I'll be giving a talk and demo of git-annex and the assistant tomorrow.
  Right after a keynote by Tim Berners-Lee! There's no streaming, but
  a recording will be available later.
* I've met numerous git-annex users and git-annex curious folk from down
  under.
* I had a suggestion that direct mode rename the `.git` directory to
  something else, to prevent foot-shooting git commands being used.
  A wrapper around git could be used to run git commands, and limit
  to safe ones. Under consideration.
* I finally updated the OSX 10.8.2 build to last week's release.
  Been having some problems with the autobuilder, but it finally spat out
  a build. Hopefully this build is good, and it should fix the javascript
  issues with Safari and the webapp.
* Ulrik Sverdrup has written <https://github.com/blake2-ppc/git-remote-gcrypt>,
  which allows using gpg encrypted ssh remotes with git. The same idea
  could be expanded to other types of remotes, like S3. I'm excited
  about adding encrypted git remote support to the assistant!


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmnG4EuvZWse5hvgrl0XAK-U61e-0iGaao"
 nickname="David"
 subject="git commit in direct mode"
 date="2013-01-31T08:30:19Z"
 content="""
How about creating a .git/hooks/pre-commit script that warns the user or checks the commit before anything goes wrong?
"""]]


code:
Not much available time today, only a few hours.

Main thing I did was fixed up the failed push tracking to use a better data
structure. No need for a queue of failed pushes, all it needs is a map of
remotes that have an outstanding failed push, and a timestamp. Now it
won't grow in memory use forever anymore. :)

Finding the right thread mutex type for this turned out to be a bit of a
challenge. I ended up with a STM TMVar, which is left empty when there are
no pushes to retry, so the thread using it blocks until there are some. And,
it can be updated transactionally, without races.

I also fixed a bug outside the git-annex assistant code. It was possible to
crash git-annex if a local git repository was configured as a remote, and
the repository was not available on startup. git-annex now ignores such
remotes. This does impact the assistant, since it is a long running process
and git repositories will come and go. Now it ignores any that
were not available when it started up. This will need to be dealt with when
making it support removable drives.


code:
Back from Australia. Either later today or tomorrow I'll dig into the
messages I was not able to get to while traveling, and then the plan is to
get into the Android port.

Video of my LCA2013 [git-annex talk](http://mirror.linux.org.au/linux.conf.au/2013/mp4/gitannex.mp4)
is now available. I have not watched it yet, hope it turned out ok despite
some technical difficulties!


code:
Got fairly far along in my triage of my backlog, looking through everything
that happened after January 23rd. Still 39 or so items to look at.

There have been several reports of problems with ssh password prompts.
I'm beginning to think the assistant may need to prompt for the password
when setting up a ssh remote. This should be handled by `ssh-askpass` or
similar, but some Linux users probably don't have it installed, and there
seems to be no widely used OSX equivalent.

---

Fixed several bugs today, all involving (gasp) direct mode.

The tricky one involved renaming or deleting files in direct mode.
Currently nothing removes the old filename from the direct mode
mapping, and this could result in the renamed or deleted file
unexpectedly being put back into the tree when content is downloaded.

To deal with this, it now assumes that direct mode mappings may be out of
date compared to the tree, and does additional checks to catch
inconsistencies. While that works well enough for the assistant,
I'd also like to make the `pre-commit` hook update the mappings for files
that are changed. That's needed to handle things like `git mv`.


code:
I need an Android development environment. I briefly looked into rooting
the Asus Transformer so I could put a Debian chroot on it and build
git-annex in there, but this quickly devolved to the typical maze of
forum posts all containing poor instructions and dead links. Not worth it.

Instead, I'm doing builds on my Sheevaplug, and once I have a static armel
binary, will see what I need to do to get it running on Android.

Fixed building with the webapp disabled, was broken by recent improvements.
I'll be building without the webapp on arm initially, because ghci/template
haskell on arm is still getting sorted out. (I tried ghc 7.6.2 and ghci is
available, but doesn't quite work.)

From there, I got a binary built pretty quickly (well, it's arm, so not *too*
quickly). Then tried to make it static by appending
`-optl-static -optl-pthread` to the ghc command line.
This failed with a bunch of errors:

/usr/lib/gcc/arm-linux-gnueabi/4.6/../../../arm-linux-gnueabi/libxml2.a(nanohttp.o): In function `xmlNanoHTTPMethodRedir': (.text+0x2128): undefined reference to `inflateInit2_'
/usr/lib/gcc/arm-linux-gnueabi/4.6/../../../arm-linux-gnueabi/libxml2.a(xzlib.o): In function `xz_decomp': (.text+0x36c): undefined reference to `lzma_code'
...

Disabling DBUS and (temporarily) XMPP got around that.

Result!

joey@leech:~/git-annex>ldd tmp/git-annex
        not a dynamic executable
joey@leech:~/git-annex>ls -lha tmp/git-annex
-rwxr-xr-x 1 joey joey 18M Feb  6 16:23 tmp/git-annex*

Next: Copy binary to Android device, and watch it fail in some interesting way.
Repeat.

---

Also more bug triage this morning...

Got the pre-commit hook to update direct mode mappings.
Uses `git diff-index HEAD` to find out what's changed. The only
tricky part was detecting when `HEAD` doesn't exist yet. Git's
plumbing is deficient in this area. Anyway, the mappings get updated
much better now.

Fixed a wacky bug where `git annex uninit` behaved badly on a filesystem
that does not support hardlinks.


code:
Have not tried to run my static binary on Android yet, but I'm already
working on a plan B in case that doesn't work. Yesterday I stumbled upon
<https://github.com/neurocyte/ghc-android>, a ghc cross-compiler for
Android that uses the Android native development kit.
It first appeared on February 4th. Good timing!

I've gotten it to build and it emits arm executables, that seem to use the
Android linker. So that's very promising indeed.

I've also gotten cabal working with it, and have it chewing through
installing git-annex's build dependencies.

----

Also made a release today, this is another release that's mostly bugfixes,
and a few minor features. Including one bug fixed at 6 am this morning, urk.

I think I will probably split my days between working on Android porting
and other git-annex development.


code:
Have been working on getting all the haskell libraries git-annex uses
built with the android cross compiler. Difficulties so far are
libraries that need tweaks to work with the new version of ghc, and some
that use cabal in ways that break cross compilation. Haskell's network
library was the last and most challenging of those.

At this point, I'm able to start trying to build git-annex for android.
Here's the first try!

<pre>
joey@gnu:~/src/git-annex>cabal install -w $HOME/.ghc-android-14-arm-linux-androideabi-4.7/bin/arm-unknown-linux-androideabi-ghc --with-ghc-pkg=$HOME/.ghc-android-14-arm-linux-androideabi-4.7/bin/arm-unknown-linux-androideabi-ghc-pkg --with-ld=$HOME/.ghc-android-14-arm-linux-androideabi-4.7/bin/arm-linux-androideabi-ld --flags="-Webapp -WebDAV -XMPP -S3 -Dbus"
Resolving dependencies...
Configuring git-annex-3.20130207...
Building git-annex-3.20130207...
Preprocessing executable 'git-annex' for git-annex-3.20130207...
on the commandline: Warning:
    -package-conf is deprecated: Use -package-db instead

Utility/libdiskfree.c:28:26:
     fatal error: sys/statvfs.h: No such file or directory
compilation terminated.
</pre>

Should not be far from a first android build now..

----

While I already have Android "hello world" executables to try, I have not yet
been able to run them. Can't seem to find a directory I can write to on the
Asus Transformer, with a filesystem that supports the +x bit. Do you really
have to root Android just to run simple binaries? I'm crying inside.

It seems that the blessed Android NDK way would involve making a Java app,
that pulls in a shared library that contains the native code. For haskell,
the library will need to contain a C shim that, probably, calls an entry
point to the Haskell runtime system. Once running, it can use the FFI to
communicate back to the Java side, probably. The good news is that CJ van
den Berg, who already saved my bacon once by developing ghc-android, tells
me he's hard at work on that very thing.

----

In the meantime, downloaded the Android SDK. Have gotten it to build a
`.apk` package from just javascript code, and managed to do it without
using eclipse (thank god). Will need this later, but for now want to wash
my brain out with soap after using it.


code:
[[!comment format=mdwn
 username="hhm"
 ip="108.17.80.177"
 subject="Write and exec on unrooted Android."
 date="2013-02-10T05:43:41Z"
 content="""
See <http://kevinboone.net/android_nonroot.html> for info on where in the android filesystem you have write, exec ability.

Basically you have these abilities in `/data/local` from `adb shell` (and in debuggable app's folders using run-as with `adb shell`), and in `/data/data/<app_id>/` for each app (for example the terminal emulator's data dir when using the terminal emulator).

It might be possible to just have temporary root using an old exploit and writing up a bit of code to just setuid on the process and exec sh (source for nachoroot exploit <https://github.com/CunningLogic/NachoRoot>).

It *is* possible to install a distro without root using a chroot faker.

I am working here <https://sites.google.com/site/taldewandroid/> on a way to install the OpenWrt distro on Android. This is working on my phone already for quite a while! So far only sketchy notes are up, but more to come, G-d willing.
"""]]


code:
Thanks to hhm, who pointed me at [KBOX](http://kevinboone.net/kbox.html),
I have verified that I can build haskell programs that work on Android.

After hacking on it all day, I've succeeded in making an initial build of
git-annex for Android. It links! It runs!

Which is not to say it's usable yet; for one thing I need to get a port
of git before it can do anything useful. (Some of the other things git-annex
needs, like ssh and sha256sum, are helpfully provided by KBOX.)

Next step will be to find or built a git port for Android. I know there's
one in the "Terminal IDE" app. Once I can use git-annex at the command line
on Android, I'll be able to test it out some (I can also port the test
suite program and run it on Android), and get a feeling for what is needed
to get the port to a usable command-line state.

And then on to the webapp, and an Android app, I suppose. So far, the port
doesn't include the webapp, but does include the assistant. The webapp
needs ghci/template haskell for arm. A few people have been reporting they
have that working, but I don't yet.


code:
[[!comment format=mdwn
 username="http://mlinden.myopenid.com/"
 ip="2001:470:d1c3:0:5c01:5562:e242:37b9"
 subject=""Build native GNU/Linux applications: the easy way""
 date="2013-02-10T22:02:19Z"
 content="""
Maybe this helps:
[[http://forum.xda-developers.com/showthread.php?t=1444792]]
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnbBRfl5F8gKRr1ko8Ai6FbEZStXXNF1S4"
 nickname="Áron"
 subject="another android git app"
 date="2013-02-11T08:20:42Z"
 content="""
https://play.google.com/store/apps/details?id=com.madgag.agit&hl=en

I used it and its decent and certainly runs git as a backend (although I havent seen its internals)
"""]]


code:
I'm now successfully using git-annex at the command line on Android.
`git annex watch` works too.

For now, I'm using a git repository under `/data`, which is on a real,
non-cripped filesystem, so symlinks work there.

There's still the issue of running without any symlinks on `/mnt/sdcard`.
While direct mode gets most of the way, it still uses symlinks in a few
places, so some more work will be needed there. Also, git-annex uses hard
links to lock down files, which won't work on cripped filesystems.

Besides that, there's lots of minor porting, but no big show-stoppers
currently.. Some of today's porting work:

* Cross-compiled git for Android. While the Terminal IDE app has some git
  stuff, it's not complete and misses a lot of plumbing commands git-annex
  uses. My git build needs some tweaks to be relocatable without setting
  `GIT_EXEC_PATH`, but it works.

* Switched git-annex to use the Haskell glob library, rather than PCRE. This
  avoids needing libpcre, which simplifies installation on several platforms
  (including Android).

* Made git-annex's `configure` hardcode some settings when cross-compiling
  for Android, rather than probing the build system.

* Android's built-in `lsof` doesn't support the -F option to use a
  machine-readable output format. So wrote a separate lsof output parser for
  the standard lsof output format. Unfortunatly, Android's lsof does not
  provide any information about where a file is open for read or write, so
  for safety, git-annex has to assume any file that's open might be written
  to, and avoid annexing it. It might be better to provide my own lsof
  eventually.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 1"
 date="2013-02-12T10:24:50Z"
 content="""
heh, +1 for removing the need for pcre
"""]]


code:
Ported all the utilities git-annex needs to run on Android:
git, rsync, gnupg, dropbear (ssh client), busybox. Built a
Makefile that can download, patch, and cross build these from source.

While all the utilities work, dropbear doesn't allow git-annex to use ssh
connection caching, which is rather annoying especially since these systems
tend to be rather slow and take a while to start up ssh connections.
I'd sort of like to try to get openssh's client working on Android instead.
Don't know how realistic that is.

Dealt with several parts of git-annex that assumed `/bin/sh` exists,
so it instead uses `/system/bin/sh` on Android. Also adapted `runshell`
for Android.

Now I have a 8 mb compressed tarball for Android.
Uncompressed it's 25 mb. This includes a lot of git and busybox
commands that won't be used, so it could be trimmed down further.
16 mb of it is git-annex itself.

[[Instructions for using the Android tarball|install/Android]]
This is for users who are rather brave, not afraid of command line and
keyboard usage. Good first step.


code:
[[!comment format=mdwn
 username="http://a-or-b.myopenid.com/"
 ip="203.45.2.230"
 subject="comment 1"
 date="2013-02-13T22:16:17Z"
 content="""
In addition to dropbear, there is ConnectBot <https://code.google.com/p/connectbot/> which might make the ssh side of the equation simpler.  I prefer it because it is half the size of dropbear.
"""]]


code:
There are at least three problems with using git-annex
on `/sdcard` on Android, or on a FAT filesystem, or on (to a first
approximation) Windows:

1. symlinks
2. hard links
3. unix permissions

So, I've added an `annex.crippledfilesystem` setting. `git annex init` now
probes to see if all three are supported, and if not, enables that, as well
as direct mode.

In crippled filesystem mode, all the permissions settings are skipped.
Most of them are only used to lock down content in the annex in indirect
mode anyway, so no great loss.

There are several uses of hard links, most of which can be dealt with by
making copies. The one use of permissions and hard links I absolutely
needed to deal with was that they're used to lock down a file as it's being
ingested into the annex. That can't be done on crippled filesystems, so I
made it instead check the metadata of the file before and after to detect
if it changed, the same way direct mode detects when files are modified.
This is likely better than the old method anyway.

The other reason files are hardlinked while they're being ingested is that
this allows running lsof on a single directory of files that are in the
process of being added, to detect if anything has them open for write.
I still need to adjust the lsof check to work in crippled filesystem mode.
It seems this won't make it much slower to run lsof on the whole repository.

At this point, I can use git-annex with a repository on `/sdcard` or a FAT
filesystem, and at least `git annex add` works.

Still several things on the TODO list before crippled filesystem mode is
complete. The only one I'm scared about is making `git merge` do something
sane when it wants to merge two trees full of symlinks, and the filesystem
doesn't let it create a symlink..


code:
[[!comment format=mdwn
 username="http://nullroute.eu.org/~grawity/"
 nickname="Mantas"
 subject="comment 1"
 date="2013-02-15T00:38:00Z"
 content="""
Windows supports hardlinks on NTFS filesystems. Not sure about the privileges needed; I think all users can create them.

Recent versions also support symlinks, but that's limited to Administrators by default, so Cygwin-Git (like the rest of Cygwin) uses special files with the \"system\" flag.

As far as I know, Git does not try to dereference symlinks when merging – it only tracks the paths they point to. Where symlinks are not supported (e.g. vfat filesystems in general, or msysGit on Windows) it simply creates regular files containing the target path (i.e. the raw contents of the respective blobs).

Regarding the use of Unix permissions: The \"open file\" syscall on Windows has a \"share mode\" flag, which can be used to disallow other programs from opening the file for writing (and/or even reading) as long as git-annex keeps it open. (I guess this is similar to mandatory locks in Linux.)
"""]]


code:
Finished crippled filesystem support, except for symlink handling.
This was straightforward, just got lsof working in that mode, made
`migrate` copy key contents, and adapted the rsync special remote to
support it. Encrypted rsync special remotes have no more overhead on
crippled filesystems than normally. Un-encrypted rsync special remotes
have some added overhead, but less than encrypted remotes. Acceptable
for now.

I've now successfully run the assistant on a FAT filesystem.

----

Git handles symlinks on crippled filesystems by setting
`core.symlinks=false` and checking them out as files containing the link
text. So to finish up crippled filesystem support, git-annex needs to
do the same whenever it creates a symlink, and needs to read file contents
when it normally reads a symlink target.

There are rather a lot of calls to `createSymbolicLink`,
`readSymbolicLink`, `getSymbolicLinkStatus`, `isSymbolicLink`, and `isSymLink`
in the tree; only ones that are used in direct mode
need to be converted. This will take a while.

Checking whether something is a symlink, or where it points is especially
tricky. How to tell if a small file in a git working tree is intended to be
a symlink or not? Well, it can look at the content and see if it makes
sense as a link text pointing at a git-annex key. As long as the
possibility of false positives is ok. It might be possible, in some cases,
to query git to verify if the object stored for that file is really a
symlink, but that won't work if the file has been renamed, for example.

Converted some of the most commonly used symlink code to handle this.
Much more to do, but it basically works; I can `git annex get` and `git
annex drop` on FAT, and it works.

-----

Unfortunately, got side-tracked when I discovered that the last release
introduced a bug in direct mode. Due to the bug, "git annex get file; git annex
drop file; git annex get file" would end up with the file being an indirect
mode symlink to the content, rather than a direct mode file. No data loss,
but not right. So, spent several hours fixing that reversion, which was
caused by me stupidly fixing another bug at 5 am in the morning last week..
and I'll probably be pushing out another release tomorrow with the fix.


code:
Worked on automatic merge conflict resolution today. I had expected to be
able to use git's merge driver interface for this, but that interface is
not sufficient. There are two problems with it:

1. The merge program is run when git is in the middle of an operation
   that locks the index. So it cannot delete or stage files. I need to
   do both as part of my conflict resolution strategy.
2. The merge program is not run at all when the merge conflict is caused
   by one side deleting a file, and the other side modifying it. This is
   an important case to handle.

So, instead, git-annex will use a regular `git merge`, and if it fails, it
will fix up the conflicts.

That presented its own difficulty, of finding which files in the tree
conflict. `git ls-files --unmerged` is the way to do that, but its output
is a quite raw form:

	120000 3594e94c04db171e2767224db355f514b13715c5 1	foo
	120000 35ec3b9d7586b46c0fd3450ba21e30ef666cfcd6 3	foo
	100644 1eabec834c255a127e2e835dadc2d7733742ed9a 2	bar
	100644 36902d4d842a114e8b8912c02d239b2d7059c02b 3	bar

I had to stare at the rather impenetrable documentation for hours and
write a lot of parsing and processing code to get from that to these mostly
self explanatory data types:

	data Conflicting v = Conflicting
	        { valUs :: Maybe v
	        , valThem :: Maybe v
	        } deriving (Show)

	data Unmerged = Unmerged
	        { unmergedFile :: FilePath
	        , unmergedBlobType :: Conflicting BlobType
	        , unmergedSha :: Conflicting Sha
	        } deriving (Show)

Not the first time I've whined here about time spent parsing unix command
output, is it? :)

From there, it was relatively easy to write the actual conflict cleanup
code, and make `git annex sync` use it. Here's how it looks:

	$ ls -1
	foo.png
	bar.png
	$ git annex sync
	commit
	# On branch master
	nothing to commit (working directory clean)
	ok
	merge synced/master
	CONFLICT (modify/delete): bar.png deleted in refs/heads/synced/master and modified in HEAD. Version HEAD of bar.png left in tree.
	Automatic merge failed; fix conflicts and then commit the result.
	bar.png: needs merge
	(Recording state in git...)
	[master 0354a67] git-annex automatic merge conflict fix
	ok
	$ ls -1
	foo.png
	bar.variant-a1fe.png
	bar.variant-93a1.png

There are very few options for ways for the conflict resolution code to
name conflicting variants of files. The conflict resolver can only use data
present in git to generate the names, because the same conflict needs to
be resolved the same everywhere.

So I had to choose between using the full key name in the filenames produced
when resolving a merge, and using a shorter checksum of the key, that would be
more user-friendly, but could theoretically collide with another key.
I chose the checksum, and weakened it horribly by only using 32 bits of it!

Surprisingly, I think this is a safe choice. The worst that can
happens if such a collision happens is another conflict, and the conflict
resolution code will work on conflicts produced by the conflict resolution
code! In such a case, it does fall back to putting the whole key in
the filename:
"bar.variant-SHA256-s2550--2c09deac21fa93607be0844fefa870b2878a304a7714684c4cc8f800fda5e16b.png"

Still need to hook this code into `git annex assistant`.


code:
[[!comment format=mdwn
 username="http://droggl.myopenid.com/"
 ip="2001:638:602:1181:a6ba:dbff:fedd:8041"
 subject="comment 1"
 date="2013-09-11T11:55:27Z"
 content="""
Nice!
However I didnt find any documentation yet on how to actually resolve the situation as a user (sorry if I'm being blind).
To be more specific: Say a conflict occurs on foo.txt and i now have two files foo.variant-dead.txt and foo.variant-beef.txt.
Now what? Say I used my favorite merge diffmerge tool to create a merged version foo-merged.txt, should I rename that to foo.txt?
Should I delete the variant files? Or drop them? Or would that lead to another merge for systems that still have the old foo.txt?
(You see I'm kind of confused of what happens here ;))

TIA for any helpers
"""]]


code:
Pushed out a release yesterday mostly for a bug fix. I have to build
git-annex 5 times now when releasing. Am wondering if I could get rid of
the Linux 64 bit standalone build. The 32 bit build should run ok on 64 bit
Linux systems, since it has all its own 32 bit libraries. What I really
need to do is set up autobuilders for Linux and Android, like we have for OSX.

Today, dealt with all code that creates or looks at symlinks. Audited every
bit of it, and converted all relevant parts to use a new abstraction layer
that handles the pseudolink files git uses when core.symlinks=false.
This is untested, but I'm quite happy with how it turned out.

----

Where next for Android? I want to spend a while testing command-line
git-annex. After I'm sure it's really solid, I should try to get the webapp
working, if possible.

I've heard rumors that Ubuntu's version of ghc somehow supports template
haskell on arm, so I need to investigate that. If I am unable to get
template haskell on arm, I would need to either wait for further
developments, or try to expand yesod's template haskell to regular haskell
and then build it on arm, or I could of course switch away from hamlet
(using blaze-html instead is appealing in some ways) and
use yesod in non-template-haskell mode entirely. One of these will work,
for sure, only question is how much pain.

After getting the webapp working, there's still the issue of bundling it
all up in an Android app that regular users can install.


code:
[[!comment format=mdwn
 username="http://jasonwoof.com/"
 nickname="JasonWoof"
 subject="phonegap might help"
 date="2013-02-17T22:43:06Z"
 content="""
In your research to see how to bundle up a graphical android app, I suggest you read up on phonegap: http://phonegap.com/

Essentially they bundle webkit so you can turn a web app into an android app. Not sure if they help you ship native/binary stuff along side the html/javascript/css, but phonegap should at least be useful as a reference for how to do some of the bundling-related stuff.
"""]]


code:
[[!comment format=c
 username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA"
 nickname="Laszlo"
 subject="Two direct mode repository on the same computer"
 date="2013-02-19T09:24:21Z"
 content="""
Hi,

I just tried the newest git annex webapp (20130216),
I am still not able to configure two direct mode repositories (one normal, one backup) on the same computer.

It does recognize my other partition (/mnt/the) as a removable drive, but when I add it
it gives me internal server error:
\"\"\"
Internal Server Error

there is no available git remote named \"the\"
git-annex version 3.20130216
\"\"\"

Could you please (pretty please) add a possibility to specify the removable drive as a simple folder?
So I could choose arbitrary folder(/mnt/the/annex-backup) on my computer and
it would treat it as a direct mode repository.

Best,
 Laszlo

"""]]


code:
Felt spread thin yesterday, as I was working on multiple things
concurrently & bouncing around as compiles finished. Been working to get
openssh to build for Android, which is quite a pain, starting with getting
openssl to build and then dealing with the Cyanogenmod patches, some of
which are necessary to build on Android and some of which break builds
outside Cyanogenmod. At the same time was testing git-annex on Android.
Found and fixed several more portability bugs while doing that. And on the
back burner I was making some changes to the webapp..

(Forgot to commit my blog post yesterday too..)

Today, that all came together.

* When adding another local repository in the webapp,
  it now allows you to choose whether it should be combined with
  your current repository, or kept separate. Several people had requested
  a way to add local clones with the webapp, for various reasons, like
  wanting a backup repository, or wanting to make a repository on a NFS
  server, and this allows doing that.

[[!img /assistant/combinerepos.png]]

* More porting fun. FAT filesystems and other things used on Android can
  get all new inode numbers each time mounted. Made git-annex use a
  sentinal file to detect when this has happened, since in direct mode
  it compares inodes. (As a bonus this also makes copying direct mode
  repositories between filesystems work.)

* Got openssh building for Android. Changed it to use $HOME/.ssh rather
  than trusting pwent.

* Got git-annex's ssh connection caching working on Android. That needs
  a place where it can create a socket. When the
  repository is on a crippled filesystem, it instead puts the socket
  in a temporary directory set up on the filesystem where the git-annex
  program resides.

With ssh connection caching, transferring multiple files off my Android
tablet *screams*! I was seeing 6.5 megabytes transferred per second,
sustained over a whole month's worth of photos.

Next problem: `git annex assistant` on Android is for some reason crashing
with a segfault on startup. Especially odd since `git annex watch` works.
I'm so close to snap-photo-and-it-syncs-nirvana, but still so far away...


code:
<video controls src="http://downloads.kitenet.net/videos/git-annex-android.ogv" width="100%"></video>
video

Today's work:

* Fixed `git annex add` of a modified file in direct mode.
* Fixed bugs in the inode sentinal file code added yesterday.
* With some help from Kevin Boone, I now understand how KBOX works and
  how to use similar techniques to build my own standalone Android app
  that includes git-annex.

  Kevin is using a cute hack; he ships a tarball and some other stuff
  as (pseudo-)library files (`libfoo.so`), which are the only files
  the Android package manager deigns to install. Then the app runs one
  of these, which installs the programs.

  This avoids needing to write Java code that extracts the programs from
  one of its assets and writes it to an executable file, which is the
  canonical way to do this sort of thing. But I noticed it has a benefit too
  (which KBOX does not yet exploit). Since the pseudo-library file is installed
  with the X bit set, if it's really a program, such as busybox or git-annex,
  that program can be run without needing to copy it to an executable file.
  This can save a lot of disk space. So, I'm planning to include all
  the binaries needed by git-annex as these pseudo-libraries.
* Got the Android Terminal Emulator to build. I will be basing my first
  git-annex Android app on this, since a terminal is needed until there's
  a webapp.
* Wasted several hours fighting with `Android.mk` files to include
  my pseudo shared library. This abuse of Makefiles by the NDK is what CDBS
  wants to grow up to be.. or is it the other way around? Anyway, it
  sucks horribly, and I finally found a way to do it without
  modifying the file at all. Ugh.
* At this point, I can build a `git-annex.apk` file containing a
  `libgit-annex.so`, and a `libbusybox.so`, that can both be directly
  run. The plan from here is to give git-annex the ability to
  auto-install itself, and the other pseudo-libraries, when it's run as
  `libgit-annex.so`.


code:
Well, it's built. [Real Android app for git-annex](http://downloads.kitenet.net/git-annex/android/current/).

[[!img /android/appinstalled.png]]

When installed, this will open a terminal in which you have access to
git-annex and all the git commands and busybox commands as well. No webapp
yet, but command line users should feel right at home.

[[!img /android/terminal.png]]

Please test it out, at least as far as installing it, opening the terminal,
and checking that you can run `git annex`; I've only been able to test on
one Android device so far. I'm especially keen to know if it works with
newer versions of Android than 4.0.3. (I know it only supports arm based
Android, no x86 etc.) Please comment if you tried it.

----

Building this went mostly as planned, although I had about 12 builds of
the app in the middle which crashed on startup with no error message ora
logs. Still, it took only one day to put it all together,
 and I even had time to gimp up a quick icon. (Better icons welcome.)

Kevin thinks that my space-saving hack won't work on all Androiden, and he
may be right. If the `lib` directory is on a different filesystem on some
devices, it will fail. But I used it for now anyhow. Thanks to the hack,
the 7.8 mb compressed .apk file installs to use around 23 mb of disk space.

----

Tomorrow: Why does `git-annex assistant` on Android re-add all existing
files on startup?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.253.75"
 subject="comment 10"
 date="2013-02-22T17:30:59Z"
 content="""
@rillian could you try to install KBOX <http://kevinboone.net/Term-debug.apk> and see if its terminal accepts input? It's the same terminal app, so will be good to know if I broke it or this is a wider problem.

Also, did you try the soft keyboard? There's a menu item to enable it.
"""]]


code:
[[!comment format=mdwn
 username="rillian"
 ip="64.213.70.194"
 subject="Re: no keybourd events on Galaxy Tab 10.1"
 date="2013-02-22T18:28:11Z"
 content="""
KBOX has the same input problem so you didn't break anything with your changes.

I was testing with the soft keyboard. Turns out the spacebar and number keys on the soft keyboard work, as well as return, just not the alphabetic keys.

Trying on a hardware keyboard (Samsung docking station) is similar. Space, Enter and numbers work, but not letters. There's some additional confusion about shift state, where it gets confused and reports shifted digits from the hardware keys when the soft keyboard is in some shift states. E.g. switching the soft keyboard to numeric mode results in shift-digits (punctuation) from the hardware numeric keys.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.253.75"
 subject="comment 12"
 date="2013-02-22T18:35:35Z"
 content="""
@rillian the upstream for this terminal is <https://github.com/jackpal/Android-Terminal-Emulator/>
and I found a bug that looks similar. <https://github.com/jackpal/Android-Terminal-Emulator/issues/129>
You might add your info there, and try the workaround documented there.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.253.75"
 subject="comment 13"
 date="2013-02-22T19:39:15Z"
 content="""
@Karsten does KBOX work on this version of Android? <http://kevinboone.net/Term-debug.apk>

It may well be that this is just too old and incompatable for the terminal to work.
"""]]


code:
[[!comment format=mdwn
 username="http://meep.pl/"
 ip="193.23.174.18"
 subject="comment 14"
 date="2013-02-22T21:45:54Z"
 content="""
On Android 2.3.3 (Samsung Galaxy S+) KBOX works, git-annex displays the \"Terminal session is running\" notification and dies immediately.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.253.75"
 subject="comment 15"
 date="2013-02-22T22:26:22Z"
 content="""
I'm seeing a pattern with Android 2.3 failing..

This failure mode suggests that the bug is in the installation script. The next build will make the terminal not exit immediately if that script fails, so you can go in to the shell and run it by hand to see what's going wrong.
"""]]


code:
[[!comment format=mdwn
 username="http://andrew.aylett.co.uk/"
 nickname="andrew"
 subject="Works on Samsung Galaxy S3"
 date="2013-02-21T23:23:08Z"
 content="""
Installed, executed and managed to get the help text on a Samsung Galaxy S3 running 4.1.2.  Not tried anything else yet, but looks good so far, thank you very much :).
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlmNbexnCgLywD0MKyT9mG9mWFb_qVj-8I"
 nickname="Joshua"
 subject="Toshiba Excite 7.7 (Android 4.1.1)"
 date="2013-02-22T00:15:46Z"
 content="""
Seems to run fine. I managed to create a repository, git-annex init it (had to set email and name per repository).

The first time I tried git-annex init it said \"Detected a crippled file system.\" \"Enabling direct mode.\" The final try that actually worked only mentioned the crippled filesystem and not direct mode.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnBJ6Dv1glxzzi4qIzGFNa6F-mfHIvv9Ck"
 nickname="Jim"
 subject="Installs and runs"
 date="2013-02-22T00:17:30Z"
 content="""
No problem installing on a Galaxy Nexus with 4.2.2

    1|u0_a162@android:/sdcard $ git clone https://git.jim.sh/jim/annex.git
    Cloning into 'annex'...
    fatal: Unable to find remote helper for 'https'

Too much to hope for at this stage, I suppose :)

    128|u0_a162@android:/sdcard $ git annex assistant
    git-annex: Not in a git repository.
    1|u0_a162@android:/sdcard $ git init test
    Initialized empty Git repository in /storage/emulated/legacy/test/.git/
    u0_a162@android:/sdcard $ cd test
    u0_a162@android:/sdcard/test $ git annex init
    init
      Detected a crippled filesystem.

      Enabling direct mode.

    *** Please tell me who you are.

    Run

      git config --global user.email \"you@example.com\"
      git config --global user.name \"Your Name\"

    to set your account's default identity.
    Omit --global to set the identity only in this repository.

    fatal: unable to auto-detect email address (got 'u0_a162@localhost.(none)')

    git-annex: user error (git [\"--git-dir=/storage/emulated/legacy/test/.git\",\"--work-tree=/storage/emulated/legacy/test\",\"commit-tree\",\"4b825dc642cb6eb9a060e54bf8d69288fbee4904\"] exited 128)
    failed
    git-annex: init: 1 failed
    1|u0_a162@android:/sdcard/test $

However, I get the same error even after setting `user.email` and `user.name` (which seems to work fine)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.253.75"
 subject="comment 4"
 date="2013-02-22T00:54:56Z"
 content="""
@jim git http support is disabled to save space

What seems to work is: git config user.email \"you@example.com\"

It's probably looking in the wrong place for the global .gitconfig so --global doesn't work. The assistant
also automates dealing with this problem BTW.
"""]]


code:
[[!comment format=mdwn
 username="rillian"
 ip="174.6.6.167"
 subject="no keybourd events on Galaxy Tab 10.1"
 date="2013-02-22T06:54:54Z"
 content="""
Hey, thanks for working on this.

I tried the Feb 21 apk on my Samsung galaxy Tab 10.1 with Android 4.0.4. It installed and launched fine, bringing up a terminal window, keyboard, etc, but only the return key works. It's not possible to type znything. The menus, new window button etc all seem to be functional.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn7gQ1zZDdWhXy9H51W2krZYShNmKL3qfM"
 nickname="Karsten"
 subject="comment 6"
 date="2013-02-22T07:00:41Z"
 content="""
Installed on Cyanogenmod 7.2 (Android 2.3.7) on my HTC Desire. When starting the App \"Terminalsitzung wird ausgeführt\" (Terminalsession is started) shows in the statusline, then the app immediately closes.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm7AuSfii_tCkLyspL6Mr0ATlO6OxLNYOo"
 nickname="Georg"
 subject="installed successfully on galaxy nexus stock 4.2.2 rom"
 date="2013-02-22T08:28:36Z"
 content="""
could run git annex and get the help screen. didn't do much else after that since I am at work currently.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="nexus 4, android 4.2.2"
 date="2013-02-22T11:59:13Z"
 content="""
Installs but can not detect email even when set:

    --global user.name \"Your Name\"                               <
    --global user.email \"you@example.com\"                         <
    u0_a135@android:/sdcard/git-annex.home $ git annex init
    init
      Detected a crippled filesystem.

    *** Please tell me who you are.

    Run

      git config --global user.email \"you@example.com\"
      git config --global user.name \"Your Name\"

    to set your account's default identity.
    Omit --global to set the identity only in this repository.

    fatal: unable to auto-detect email address (got 'u0_a135@localhost.(none)')

    git-annex: user error (git [\"--git-  dir=/storage/emulated/legacy/git-annex.home/.git\",\"--work-tree=/storage/emulated/legacy/git-annex.home\",\"commit-tree\",\"4b825dc642cb6eb9a060e54bf8d69288fbee4904\"] exited 128)
failed
    git-annex: init: 1 failed
    1|u0_a135@android:/sdcard/git-annex.home $

Sorry for bad copy paste... On tour bus in Stockholm and only have phone, no laptop...

RichiH
"""]]


code:
[[!comment format=mdwn
 username="http://ertai.myopenid.com/"
 nickname="npouillard"
 subject="comment 9"
 date="2013-02-22T15:19:25Z"
 content="""
Same trouble with --global config, however local config works.
"""]]


code:
So it seems the Android app works pretty well on a variety of systems.
Only report of 100% failure so far is on Cyanogenmod 7.2 (Android 2.3.7).

Worked today on some of the obvious bugs.

* Turns out that getEnvironment is broken on Android, returning no
  environment, which explains the weird git behavior where it complains
  that it cannot determine the username and email (because it sees no USER
  or HOST), and suggests setting them in the global git config (which it
  ignores, because it sees no HOME). Put in a work around for this
  that makes `git annex init` more pleasant, and opened a bug report on
  ghc-android.
* Made the Android app detect when it's been upgraded, and re-link all
  the commands, etc.
* Fixed the bug that made `git-annex assistant` on Android re-add all
  existing files on startup.
* Enabled a few useful things in busybox. Including vi.
* Replaced the service notification icon with one with the git-annex logo.
* Made the terminal not close immediately when the shell exits, which
  should aid in debugging of certain types of crashes.

I want to set up an autobuilder for Android, but to do that I need to
install all the haskell libraries on my server. Since getting them built
for Android involved several days of hacking the first time, this will
be an opportunity to make sure I can replicate that. Hopefully in less time.
;)


code:
Set up an autobuilder for the linux standalone binaries.
Did not get an Android autobuilder set up yet, but I did update
the Android app with recent improvements, so [[upgrade|install/Android]].

----

Investigated further down paths to getting the webapp built for Android.

* Since recent ghc versions support ghci and thus template haskell on arm,
  at least some of the time, I wonder what's keeping the ghc-android build
  from doing so? It might be due to it being a cross compiler. I tried
  recompiling it with the stage 2, native compiler enabled. While I was
  able to use that ghc binary on Android, it refused to run --interactive,
  claiming it was not built with that enabled. Don't really understand
  the ghc build system, so might have missed something.

  Maybe I need to recompile ghc using the native ghc running on Android.
  But that would involve porting gcc and a lot of libraries and toolchain
  stuff to Android.

* [yesod-pure](http://hackage.haskell.org/package/yesod-pure) is an option,
  and I would not mind making all the code changes to use it, getting
  rid of template haskell entirely. (Probably around 1 thousand lines of
  code would need to be written, but most of it would be trivial
  conversion of hamlet templates.)

  Question is, will yesod install at all without template haskell? Not
  easily. `vector`, `monad-logger`, `aeson`, `shakespeare`,
  `shakespeare-css`, `shakespeare-js`, `shakespeare-i18n`, `hamlet`
  all use TH at build time. Hacked them all to just remove the TH parts.

  The hack job on `yesod-core` was especially rough, involving things like
  404 handlers. Did get it to build tho!

  Still a dozen packages before I can build yesod, and then will try
  building [this yesod-pure demo](https://gist.github.com/snoyberg/3870834/raw/212f0164de36524291df3ab35788e2b72d8d1e75/fib.hs).


code:
[[!comment format=mdwn
 username="http://meep.pl/"
 ip="193.23.174.18"
 subject="Android 2.3"
 date="2013-02-24T15:16:51Z"
 content="""
The updated version lets me see the failure(s), and it looks like this:

    cd: can't cd to /data/data/ga.androidterm/lib/lib.runshell.so/../..
    [: not found
    [: not found
    could not open //installed-version, No such file or directory
    /data/data/ga.androidterm/lib/lib.runshell.so: cannot create
    /sdcard/git-annex.home/git-annex-install.log: directory nonexistent

    [Terminal session finished]

"""]]


code:
Wrote a C shim to get the Android app started. This avoids it relying on
the Android /system/bin/sh to run its shell script, or indeed relying on
any unix utilities from Android at all, which may help on some
systems. Pushed a new build of the Android app.

Tracked down a failure a lot of people are reporting with WebDAV support
to a backported security fix in the TLS library, and filed an upstream bug
about it.

Various other misc fixing and stuff.
My queue of bug reports and stuff only has 47 items in it now. Urk..


code:
[[!comment format=mdwn
 username="http://meep.pl/"
 ip="193.23.174.18"
 subject="Android 2.3 again"
 date="2013-02-24T23:07:28Z"
 content="""
I got _a lot_ further with the latest upgrade.

* installation worked
* `git init` worked
* first `git annex init` asked to set user.{email,name}
* `git config --global ...` worked
* second `git annex init` segfaulted, but not immediately:

      init
        Detected a crippled filesystem.
      error: git-annex died of signal 11

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA"
 nickname="Laszlo"
 subject="comment 2"
 date="2013-02-25T11:22:53Z"
 content="""
Is the autiobuilder for linux available already?
I can't find it, and I'm eager to try the thumbdrive related fixes (where I can specify a simple folder).

Best,
 Laszlo
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.210"
 subject="comment 3"
 date="2013-02-25T15:53:30Z"
 content="""
@meep try running it with --debug for some more hint of where it's crashing

@Laszlo see [[install/Linux_standalone]]
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA"
 nickname="Laszlo"
 subject="comment 4"
 date="2013-02-25T18:28:09Z"
 content="""
joeyh: Is the 20130216 is the newest? Because you just wrote about two days ago of direct mode fixes, I'm interested in.
I tried 20130216 already.

Laszlo

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.210"
 subject="comment 5"
 date="2013-02-26T16:33:11Z"
 content="""
I have not made a release with those fixes yet, but they are available in the autobuilds.
"""]]


code:
[[!comment format=mdwn
 username="http://meep.pl/"
 ip="193.23.174.18"
 subject="git annex init --debug"
 date="2013-02-27T06:39:19Z"
 content="""
The last thing before the signal 11 error that I can see with `--debug` is `read: uname [\"-n\"]` after reports of running 5 git commands.

"""]]


code:
[[!comment format=mdwn
 username="http://meep.pl/"
 ip="193.23.174.18"
 subject="comment 7"
 date="2013-02-27T06:53:15Z"
 content="""
But the error does not happen immediately after that. All the 6 log messages have the same timestamp and `git annex init --debug; date` prints a timestamp that is 3-4 seconds later.
"""]]


code:
[[!comment format=mdwn
 username="http://meep.pl/"
 ip="193.23.174.18"
 subject="comment 8"
 date="2013-02-27T06:55:24Z"
 content="""
Also, this was after a reinstall. The next time I start the app the terminal opens but prompt does not appear (at least not for a very long time). Then after uninstall and reinstall I get to the prompt again.

"""]]


code:
An Android autobuilder is now set up to run nightly. At this point
I don't see an immediate way to getting the webapp working on Android, so
it's best to wait a month or two and see how things develop in Haskell land.
So I'm moving on to other things.

Today:

* Fixed a nasty regression that made `*` not match files in subdirectories.
  That broke the preferred content handling, among other things. I will
  be pushing out a new release soon.
* As a last Android thing (for now), made the Android app automatically
  run `git annex assistant --autostart` , so you can manually set up
  an assistant-driven repository on Android, listing the repository in
  `.config/git-annex/autostart`
* Made the webapp display any error message from `git init` if it fails.
  This was the one remaining gap in the logging.
  One reason it could fail is if the system has a newer git in use, and
  `~/.gitconfig` is configured with some options the older git bundled
  with git-annex doesn't like.
* Bumped the major version to 4, and annex.version will be set to 4 in
  new direct mode repositories. (But version 3 is otherwise still used, to
  avoid any upgrade pain.) This is to prevent old versions that don't
  understand direct mode from getting confused. I hope direct mode is
  finally complete, too, after the work to make it work on crippled
  filesystems this month.
* Misc other bugfixes etc. Backlog down to 43.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="Automagic photo backups"
 date="2013-02-26T21:56:44Z"
 content="""
Did you investigate how hard it would be to only commit & upload once the phone is being charged _and_ on WiFi? That does not rely on the webapp, but it would make using git-annex on Android a lot more useful until you restart your work on this.

RichiH
"""]]


code:
Random improvements day..

Got the merge conflict resolution code working in `git annex assistant`.

Did some more fixes to the pushing and pulling code, covering some cases
I missed earlier.

Git syncing seems to work well for me now; I've seen it recover
from a variety of error conditions, including merge conflicts and repos
that were temporarily unavailable.

----

There is definitely a MVar deadlock if the merger thread's inotify event
handler tries to run code in the Annex monad. Luckily, it doesn't
currently seem to need to do that, so I have put off debugging what's going
on there.

Reworked how the inotify thread runs, to avoid the two inotify threads
in the assistant now from both needing to wait for program termination,
in a possibly conflicting manner.

Hmm, that *seems* to have fixed the MVar deadlock problem.

----

Been thinking about how to fix [[bugs/watcher_commits_unlocked_files]].
Posted some thoughts there.

It's about time to move on to data [[syncing]]. While eventually that will
need to build a map of the repo network to efficiently sync data over the
fastest paths, I'm thinking that I'll first write a dumb version. So, two
more threads:

1. Uploads new data to every configured remote. Triggered by the watcher
   thread when it adds content. Easy; just use a `TSet` of Keys to send.

2. Downloads new data from the cheapest remote that has it. Could be
   triggered by the
   merger thread, after it merges in a git sync. Rather hard; how does it
   work out what new keys are in the tree without scanning it all? Scan
   through the git history to find newly created files? Maybe the watcher
   triggers this thread instead, when it sees a new symlink, without data,
   appear.

Both threads will need to be able to be stopped, and restarted, as needed
to control the data transfer. And a lot of other control smarts will
eventually be needed, but my first pass will be to do a straightforward
implementation. Once it's done, the git annex assistant will be basically
usable.


code:
First day of [Kickstarter funded work](http://www.kickstarter.com/projects/joeyh/git-annex-assistant-like-dropbox-but-with-your-own/)!

Worked on [[inotify]] today. The `watch` branch in git now does a pretty
good job of following changes made to the directory, annexing files
as they're added and staging other changes into git. Here's a quick
transcript of it in action:

	joey@gnu:~/tmp>mkdir demo
	joey@gnu:~/tmp>cd demo
	joey@gnu:~/tmp/demo>git init
	Initialized empty Git repository in /home/joey/tmp/demo/.git/
	joey@gnu:~/tmp/demo>git annex init demo
	init demo ok
	(Recording state in git...)
	joey@gnu:~/tmp/demo>git annex watch &
	[1] 3284
	watch . (scanning...) (started)
	joey@gnu:~/tmp/demo>dd if=/dev/urandom of=bigfile bs=1M count=2
	add ./bigfile 2+0 records in
	2+0 records out
	2097152 bytes (2.1 MB) copied, 0.835976 s, 2.5 MB/s
	(checksum...) ok
	(Recording state in git...)
	joey@gnu:~/tmp/demo>ls -la bigfile
	lrwxrwxrwx 1 joey joey 188 Jun  4 15:36 bigfile -> .git/annex/objects/Wx/KQ/SHA256-s2097152--e5ced5836a3f9be782e6da14446794a1d22d9694f5c85f3ad7220b035a4b82ee/SHA256-s2097152--e5ced5836a3f9be782e6da14446794a1d22d9694f5c85f3ad7220b035a4b82ee
	joey@gnu:~/tmp/demo>git status -s
	A  bigfile
	joey@gnu:~/tmp/demo>mkdir foo
	joey@gnu:~/tmp/demo>mv bigfile foo
	"del ./bigfile"
	joey@gnu:~/tmp/demo>git status -s
	AD bigfile
	A  foo/bigfile

Due to Linux's inotify interface, this is surely some of the most subtle,
race-heavy code that I'll need to deal with while developing the git annex
assistant. But I can't start wading, need to jump off the deep end to make
progress!

The hardest problem today involved the case where a directory is moved
outside of the tree that's being watched. Inotify will still send events
for such directories, but it doesn't make sense to continue to handle them.

Ideally I'd stop inotify watching such directories, but a lot of state
would need to be maintained to know which inotify handle to stop watching.
(Seems like Haskell's inotify API makes this harder than it needs to be...)

Instead, I put in a hack that will make it detect inotify events from
directories moved away, and ignore them. This is probably acceptable,
since this is an unusual edge case.

----

The notable omission in the inotify code, which I'll work on next, is
staging deleting of files. This is tricky because adding a file to the
annex happens to cause a deletion event. I need to make sure there are no
races where that deletion event causes data loss.


code:
As well as making a new release, I rewrote most of the Makefile, so that it
uses cabal to build git-annex. This avoids some duplication, and most
importantly, means that the Makefile can auto-detect available libraries
rather than needing to juggle build flags manually. Which was becoming a
real pain.

I had avoided doing this before because cabal is slow for me on my little
netbook. Adding ten seconds to every rebuild really does matter! But I came
up with a hack to let me do incremental development builds without the
cabal overhead, by intercepting and reusing the ghc command that cabal
runs.

There was also cabal "fun" to get the Android build working with cabal.
And more fun involving building the test suite. For various reasons, I
decided to move the test suite into the git-annex binary. So you can run
`git annex test` at any time, any place, and it self-tests. That's a neat
trick I've seen one or two other programs do, and probably the nicest thing
to come out of what was otherwise a pretty yak shaving change that involved
babysitting builds all day.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 10"
 date="2013-03-01T04:23:48Z"
 content="""
@Brian, I follow your logic, it does seem that start.c is not finding itself. But, you checked the pid of the terminal, while start.c is the program it (usually) starts.

Anyway, I have tried making start.c fall back to a hard coded path if it cannot find busybox. It'll also say what path it detected, which may help debug the underlying problem. You can pick up an app with this change from the autobuilder.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmmLfBDEDFfEOba8Ra46nsnTmoNHFsLObo"
 nickname="Brian"
 subject="comment 11"
 date="2013-03-01T12:02:09Z"
 content="""
Good point--it didn't occur to me start.c runs as a child process. Makes sense. How do I download the output of the autobuild? Size-wise and behavior-wise, http://downloads.kitenet.net/git-annex/android/current/git-annex.apk looks like the same file I already had.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 13"
 date="2013-03-01T16:36:58Z"
 content="""
@Brian the autobuild is linked to on [[install/Android]]
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmmLfBDEDFfEOba8Ra46nsnTmoNHFsLObo"
 nickname="Brian"
 subject="comment 13"
 date="2013-03-01T23:52:53Z"
 content="""
I got the latest autobuild version installed and the fallback works for me--thanks. FYI, the fallback error message says: \"cannot find expected files in /data/app-lib\"
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnxp2XU8gIribhhGhGuYtU6eMMwHv5gUGI"
 nickname="Amitai"
 subject="comment 1"
 date="2013-02-28T02:53:53Z"
 content="""
That _is_ a neat trick, especially since you're deploying to odd platforms where you'll probably be even gladder of the in-app tests far in the future than you are now!
"""]]


code:
[[!comment format=mdwn
 username="http://meep.pl/"
 ip="193.23.174.18"
 subject="comment 2"
 date="2013-02-28T06:13:21Z"
 content="""
Maybe it's the build changes? Android apk disappeared from <http://downloads.kitenet.net/git-annex/android/current/>
"""]]


code:
[[!comment format=c
 username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA"
 nickname="Laszlo"
 subject="comment 3"
 date="2013-02-28T08:40:46Z"
 content="""
Two suggestion:
1. Possibility to restrict CPU usage (a 10-20% would be nice, or a slider)
2. Pager on Dashboard

I have a directory, it is 741MB, and contains 23381 files.
This directory is pretty much everything needed for my daily job.

I added a copy of this directory to annex, then I created a backup dir of annex,
and my laptop is using almost 100%CPU al the time, it is running since 1 hour already, and
only 140MB was copied over to the backup directory.

Don't get me wrong, this application just blows my mind, and it seems to *work* for the first time for real.

The suggestion is simply a slider in resource manager (preferably inside dashboard):
Restrict CPU usage [0---X---100]34%
Restrict upload bandwith [0-X----800]9kB/s
Restrict download bandwith [0-X----800]80kB/s

Also the pager is musthave in dashboard, because it renders firefox unusable with that many entries (21000).

Nice work, thank you for the release!

Laszlo

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA"
 nickname="Laszlo"
 subject="lot of stuck git processes"
 date="2013-02-28T09:01:25Z"
 content="""
Oh, and here is how it looks
ps -e |grep git
command after git-annex has been stopped (from the gui):
(54 process, from which 50 process <defunct>)
http://pastebin.com/5u74qTNU
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 5"
 date="2013-02-28T19:42:09Z"
 content="""
@meep That's not the autobuild. I put the wrong file there! Fixed.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 6"
 date="2013-02-28T19:52:59Z"
 content="""
Lazlo, could you please file a proper [[bug|bugs]] report and include details like what version of git-annex you're using there when you get the zombies.

I have modified the webapp to only ever show 10 queued transfers on the dashboard.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmmLfBDEDFfEOba8Ra46nsnTmoNHFsLObo"
 nickname="Brian"
 subject="link busybox: No such file or directory"
 date="2013-03-01T02:32:32Z"
 content="""
When I run the 28-Feb-2013 android version of git annex, I'm getting the error:

    link busybox: No such file or directory

Maybe this code is not finding the busybox.so file:

    link(\"lib/lib.busybox.so\", \"busybox\")

Any ideas how I can troubleshoot?

I'm running Android 4.2.2.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 8"
 date="2013-03-01T03:10:35Z"
 content="""
@Brian what you can do is go to the terminal's preferences menu and change the shell it runs to `/system/bin/sh` . This should let you open a new tab with a shell prompt. Then you can try to cd to `/data/data/ga.androidterm`, and see if both `lib/lib.start.so` and `lib/lib.busybox.so` exist.

You could then try to do the same thing it does to set up the system:

<pre>
ln lib/lib.busybox.so busybox
./busybox sh lib/lib.runshell.so
</pre>
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmmLfBDEDFfEOba8Ra46nsnTmoNHFsLObo"
 nickname="Brian"
 subject="comment 9"
 date="2013-03-01T03:55:46Z"
 content="""
@joey Both .so files were present and I was able to run the ln and busybox commands. And after that 'git annex' displayed the help screen. Thanks!

Then I ran 'ps | grep ga.androidterm' to find the pid and then 'ls -l /proc/<pid>/exe' and it showed the symlink points at '/system/bin/app_process'.

So the 'int n=readlink(\"/proc/self/exe\", buf, 1023);' in start.c isn't very useful in this case.
"""]]


code:
[[!meta title="day 201  real Android wrapup"]]

I got yesod-pure fully working on Android...

[[!img fib.png size=400x]]

As expected, this involved manually splicing some template haskell. I'm now
confident I can port the git-annex webapp to Android this way, and that it
will take about a week. Probably will start on that in a month or so. If
anyone has some spare Android hardware they'd like to loan me, possibly
sooner. (Returning loaner Asus Transformer tomorrow; thanks Mark.) Although
I'm inclined to let the situation develop; we may just get a ghc-android
that supports TH..

Also:

* Fixed several bugs in the Android installation process.
* Committed patches for all Haskell libraries I've modified to
  the git-annex git repo.
* Ran the test suite on Android. It found a problem; seems `git clone`
  of a local repository is broken in the Android environment.

Non-Android:

* Made the assistant check every hour if logs have grown larger than a
  megabyte, and rotate them to avoid using too much disk space.
* Avoided noise in log about typechanged objects when running
  git commit in direct mode repositories. Seems `git commit`
  has no way to shut that up, so I had to /dev/null it.
* When run with `--debug`, the assistant now logs more information
  about why it transfers or drops objects.
* Found and fixed a case where moving a file to an archive directory would
  not cause its content to be dropped.
* Working on a bug with the assistant where moving a file out of an
  archive directory in direct mode sometimes ends up with a symlink
  rather than a proper direct mode file. Have not gotten to the bottom
  of it entirely, but it's a race, and I think the race is between
  the direct mode mapping being updated, and the file being transferred.


code:
[[!comment format=mdwn
 username="http://ao2.it/"
 nickname="ao2"
 subject="Using the Emulator?"
 date="2013-03-04T13:58:18Z"
 content="""
Hi Joey,

while you wait for some actual Android device have you tried the emulator?

I've recently started playing with Android without any physical device, and the emulator is pretty responsive with hardware acceleration on (you need to run an x86 AVD).

Here are some instructions: http://git.ao2.it/android/android-app-development-getting-started.git/blob/HEAD:/android-app-development-getting-started.sh

Also, are you going to use a WebView connecting to the haskell web server for the WebApp?

Ciao,
   Antonio
"""]]


code:
Seems I am not done with the Android porting just yet after all. One more
porting day..

Last night I managed to get all of Yesod to build for Android.
I even successfully expanded some Template Haskell used in yesod-form. And
am fairly confident I could manually expand all the TH in there, so it's
actually useable without TH. Most of the TH is just commented out for now.

However, programs using Yesod didn't link; lots of missing symbols. I have
been fighting to fix those all day today.

Finally, I managed to build [the yesod-pure demo server](https://gist.github.com/snoyberg/3870834/raw/212f0164de36524291df3ab35788e2b72d8d1e75/fib.hs),
and I have a working web server on Android! It listens for requests, it logs
them correctly, and it replies to requests. I did cripple yesod's routing
code in my hack-n-slash port of it, so it fails to *display* any pages,
but never has "Internal Server Error" in a web browser been such a sweet
sight. ;-)

At this point, I estimate about 1 or 2 weeks work to get to an Android
webapp. I'd need to:

1. More carefully port Yesod, manually expanding all Template Haskell
   as I went, rather than commenting it all out like I did this time.
2. Either develop a tool to automatically expand Hamlet TH splices
   (preferred; seems doable), or convert all the webapp's templates
   to not use Hamlet.

-----

I've modified 38 Haskell libraries so far to port them to Android. Mostly
small hacks, but eep this is a lot of stuff to keep straight.


code:
Stuck on a bug or two, I instead built a new Preferences page:

[[!img /assistant/preferences.png]]

The main reason I wanted that was to allow enabling debug logging at
runtime. But I've also wanted to expose annex.diskreserve and
annex.numcopies settings to the webapp user. Might as well let them control
whether it auto-starts too.

Had some difficulty deciding where to put this. It could be considered
additional configuration for the local repository, and so go in the
local repository edit form. However, this stuff can only be configured for
local repositories, and not remotes, and that same form is used to edit
remotes, which would lead to inconsistent UI and complicate the code.
Also, it might grow to include things not tied to any repository,
like choice of key/value backends. So, I put the preferences on their own
page.

---

Also, based on some useful feedback from testing the assistant with a large
number of files, I made the assistant disable git-gc auto packing in
repositories it sets up. (Like fsck, git-gc always seems to run exactly
when you are in a hurry.) Instead, it'll pack at most once a day, and with
a rather higher threshold for the number of loose objects.


code:
Tracked down the bug that's been eluding me for days. It was indeed a race, and
could result in a file being transferred into a direct mode repository and
ending up in indirect mode. Was easy to fix once understood, just needed to
update the direct mode mapping before starting the transfer.

While I was in there, I noticed another potential race, also in direct
mode, where the watcher could decide to rewrite a symlink to fix its
target, and at just the wrong time direct mode content could arrive in its
place, and so get deleted. Fixed that too.

Seems likely there are some other direct mode races. I spent quite a while
hammering on dealing with the indirect mode races with the assistant
originally.

-----

Next on my list is revisiting XMPP.

Verified that git push over XMPP works between multiple repositories that
are sharing the same XMPP account. It does.

Seeing the XMPP setup process with fresh eyes, I found several places
wording could be improved. Also, when the user goes in and configures
(or reconfigures) an XMPP account, the next step is to do pairing,
so it now redirects directly to there.

Next I need to make XMPP get back into sync after a network disconnection
or when the assistant is restarted. This currently doesn't happen until
a XMPP push is received due to a new change being made.

### back burner: yesod-pure

Last night I made a yesod-pure branch, and did some exploratory conversion
away from using Hamlet, of the Preferences page I built yesterday.

I was actually finding writing pure Blaze worked *better* than Hamlet,
at first. Was able to refactor several things into functions that in Hamlet
are duplicated over and over in my templates, and built some stuff that makes
rendering type safe urls in pure Blaze not particularly ungainly. For example,
this makes a submit button and a cancel button that redirects to another page:

[[!format haskell """
        buttons = toWidget $ \redir ->
               "Save Preferences" <>|<> redir ConfigurationR []
"""]]

The catch is how to deal with widgets that need to be nested inside other
html. It's not possible to do this both cleanly and maximally
efficiently, with Blaze. For max efficiency, all the html before the widget
should be emitted, and then the widget run, and then all html after it be
emitted. To use Blaze, it would have to instead generate the full html,
then split it around the widget, and then emit the parts, which is less
efficient, doesn't stream, etc.

I guess that's the core of what Hamlet does; it allows a clean
representation and due to running TH at build time, can convert this into
an efficient (but ugly) html emitter.

So, I may give up on this experiment. Or I may make the webapp less than
maximally fast at generating html and go on with it. After all, these
sorts of optimisations are mostly aimed at high-volume websites, not local
webapps.


code:
Yesterday was all bug fixes, nothing to write about really.

Today I've been working on getting XMPP remotes to sync more reliably.
I left some big holes when I stopped work on it in November:

1. The assistant did not sync with XMPP remotes when it started up.
2. .. Or when it detected a network reconnection.
3. There was no way to trigger a full scan for transfers
   after receiving a push from an XMPP remote.

The asynchronous nature of git push over XMPP complicated doing this, but
I've solved all 3 issues today.


code:
More XMPP fixes. The most important change is that it now stores important
messages, like push requests, and (re)sends them when a buddy's client
sends XMPP presence. This makes XMPP syncing much more robust, all the
clients do not need to already be connected when messages are initially
sent, but can come and go. Also fixed a bug preventing syncing from working
immediately after XMPP pairing. XMPP seems to be working well now; I only
know of one minor bug.


code:
Fixed the last XMPP bug I know of. Turns out it was not specific to XMPP at
all; the assistant could forget to sync with any repository on startup
under certain conditions.

Also fixed bugs in `git annex add` and in the glob matching, and some more.

I've been working on some screencasts. More on them later.. But while doing
them I found a perfect way to reliably reproduce the webapp hang that
I've been chasing for half a year, and last saw at my presentation in
Australia. Seems the old joke about bugs only reproducible during
presentations is literally true here!

I have given this bug its [[own page|bugs/webapp_hang]] at last, and have a
tcpdump of it happening and everything. Am working on an hypotheses that it
might be caused by Warp's [slowloris](http://ha.ckers.org/slowloris/)
attack prevention code being falsely triggered by the repeated hits the web
browser makes as the webapp's display is updated.


code:
> And so we waited. Tick-tock, blink-blink, thirty seconds stretched
> themselves out one by one, a hole in human experience. -- The Bug

I *think* I've managed to fully track down the [[webapp_hang]]. It is,
apparently, a bug in the Warp web server's code intended to protect against
the [Slowloris](http://ha.ckers.org/slowloris/) attack. It assumes,
incorrectly, that a web browser won't reuse a connection it's left idle for
30 seconds. Some bad error handling keeps a connection open with no thread
to service it, leading to the hang.
<https://github.com/yesodweb/wai/issues/146>

Have put a 30 minute timeout into place as a workaround, and, unless
a web browser sits on an idle connection for a full 30 minutes and then
tries to reuse it, this should be sufficient.

I was chasing that bug, quietly, for 6 months. Would see it now and
then, but not be able to reproduce it or get anywhere with analysis.
I had nearly given up. If you enjoy stories like that, read Ellen
Ullman's excellent book The Bug.

> To discover that between the blinks of the machine’s shuttered eye—going
> on without pause or cease; simulated, imagined, but still not caught—was
> life.


code:
Today is a planning day. I have only a few days left before I'm off to
Nicaragua for [DebConf](http://debconf12.debconf.org/), where I'll only
have smaller chunks of time without interruptions. So it's important to get
some well-defined smallish chunks designed that I can work on later. See
bulleted action items below (now moved to [[syncing]]. Each
should be around 1-2 hours unless it turns out to be 8 hours... :)

First, worked on writing down a design, and some data types, for data transfer
tracking (see [[syncing]] page). Found that writing down these simple data
types before I started slinging code has clarified things a lot for me.

Most importantly, I realized that I will need to modify `git-annex-shell`
to record on disk what transfers it's doing, so the assistant can get that
information and use it to both avoid redundant transfers (potentially a big
problem!), and later to allow the user to control them using the web app.

While eventually the user will be able to use the web app to prioritize
transfers, stop and start, throttle, etc, it's important to get the default
behavior right. So I'm thinking about things like how to prioritize uploads
vs downloads, when it's appropriate to have multiple downloads running at
once, etc.


code:
Trying to record screencasts demoing the assistant is really helping me
see things that need to be fixed.

Got the version of the haskell TLS library in Debian fixed, backporting
some changes to fix a botched security fix that made it reject all
certificates. So WebDAV special remotes will work again on the next release.

Fixed some more problems around content being dropped when files are
moved to archive directories, and gotten again when files are
moved out.

Fixed some problems around USB drives. One was a real jaw-dropping
bug: "git annex drop --from usbdrive" when the drive was not
connected still updated the location log to indicate it did not have
the file anymore! (Thank goodness for fsck..)

I've noticed that moving around files in direct mode repos is inneficient,
because the assistant re-checksums the "new" file. One way to avoid
that would be to have a lookup table from (inode, size, mtime) to
key, but I don't have one, and would like to avoid adding one.

Instead, I have a cunning plan to deal with this heuristically. If the
assistant can notice a file was removed and another file added at the same
time, it can compare the (inode, size, mtime) to see if it's a rename, and
avoid the checksum overhead.

The first step to getting there was to make the assistant better at
batching together delete+add events into a single rename commit. I'm happy
to say I've accomplished that, with no perceptable delay to commits.


code:
Got renaming fully optimised in the assistent in direct mode. I even got it
to work for whole directory renames. I can drag files around all day in the
file manager and the assistant often finishes committing the rename before
the file manager updates. So much better than checksumming every single
renamed file! Also, this means the assistant makes just 1 commit when a
whole directory is renamed.

Last night I added a feature to `git annex status`. It can now be asked to
only show the status of a single directory, rather than the whole annex.
All the regular file filtering switches work, so some neat commands
are possible. I like `git annex status . --in foo --not --in bar` to see
how much data is in one remote but not another.

This morning, an important thought about [[bugs/smarter_flood_filling]],
that will avoid unnecessary uploads to transfer remotes when all that's
needed to get the file to its destination is a transfer over the LAN.
I found an easy way to make that work, at least in simple cases.
Hoping to implement it soon.

Less fun, direct mode turns out to be somewhat buggy when files with
duplicate content are in the repository. Nothing fails, but `git annex
sync` will re-checksum files each time it's run in this situation, and the
assistant will re-checksum files in certian cases. Need to work on this
soon too.


code:
Last night, revamped the web site, including making a [[/videos]]
page, which includes a new screencast introducing the git-annex assistant.

Worked on improving my Haskell development environment in vim.
hdevtools is an excellent but tricky thing to get working. Where before
it took around 30 seconds per compile for me to see type errors,
I now see them in under a second each time I save, and can also look up
types of any expression in the file. Since programming in Haskell is
mostly driven by reacting to type errors ;) this should speed me up a lot,
although it's not perfect. Unfortunatly, I got really caught up in tuning
my setup, and only finished doing that at 5:48 am.

Disasterously late this morning, fixed the assistant's
`~/.ssh/git-annex-shell` wrapper so it will work when the ssh key does
not force a command to be run. Also made the webapp behave better
when it's told to create a git repository that already exists.

After entirely too little sleep, I found a puzzling bug where copying files
to a local repo fails once the inode cache has been invalidated. This
turned out to involve running a check in the state monad of the wrong
repository. A failure mode I'd never encountered before.

Only thing I had brains left to do today was to record another screencast,
which is rendering now...


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl1D_4vD5ueaDw8gRsIYPO3UHRKEpFfg9I"
 nickname="Дмитрий"
 subject="About development in Vim"
 date="2013-03-13T02:18:20Z"
 content="""
Could you describe your vim configuration for haskell development, what plugins do you use, what is in you .vimrc file and maybe even record a screencast of typical \"session\" of work?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmU_2tE75oyG0h2ZPN4lcroIKEMC8G-otE"
 nickname="Michael"
 subject="Global preferences?"
 date="2013-03-13T08:00:55Z"
 content="""
I just did a \"cabal install git-annex --bindir=$HOME/bin\" on Ubuntu 12.10 and I don't see the global preferences (where you configure the autostart etc.) from the screencast. How can I enable this?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 3"
 date="2013-03-13T16:08:35Z"
 content="""
The global preferences page is a new feature that will be in the next release. You can use a daily build or build from source to try it now.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmU_2tE75oyG0h2ZPN4lcroIKEMC8G-otE"
 nickname="Michael"
 subject="comment 4"
 date="2013-03-13T16:34:46Z"
 content="""
Build from source is too hard for a non-haskell developer, I run into various errors I can not resolve. The daily build gives me \"git-annex: error while loading shared libraries: libyaml-0.so.2: cannot open shared object file: No such file or directory\". For Ubuntu a PPA with daily build would be handy.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 5"
 date="2013-03-13T16:58:04Z"
 content="""
In the standalone linux daily build, you need to use the \"runshell\" script to start a shell environment in which you can run git-annex.
"""]]


code:
Got the assistant to check again, just before starting a transfer, if
the remote still wants the object. This should be all that's needed to
handle the case where there is a transfer remote on the internet somewhere,
and a locally paired client on the LAN. As long as the paired repository
has a lower cost value, it will be sent any new file first, and if that
is the only client, the file will not be sent to the transfer remote at
all.

But.. locally paired repos did not have a lower cost set, at all.
So I made their cost be set to 175 when they're created. Anyone
who already did local pairing should make sure the Repositories
list shows locally paired repositories above transfer remotes.

Which brought me to needing an easy way to reorder that list of remotes,
which I plan to do by letting the user drag and drop remotes around,
which will change their cost accordingly. Implementing that has two
pain points:

1. Often a lot of remotes will have the same default cost value.
   So how to insert a remote in between two that have cost 100?
   This would be easy if git-annex didn't have these cost numbers,
   and instead just had an ordered list of remotes.. but it doesn't.
   Instead, dragging remotes in the list will sometimes need to change
   the costs of others, to make room to insert them in. It's BASIC
   renumbering all over again. So I wrote some code to do this with as
   little bother as possible.

2. Drag and drop means javascript. I got the basics going quickly with
   jquery-ui, only to get stuck for over an hour on some CSS issue
   that made lines from the list display all weird while being dragged.
   It is always something like this with javascript..

So I've got these 2 peices working, and even have the AJAX call
firing, but it's not quite wired up just yet. Tomorrow.


code:
Fighting with javascript all day and racing to get a release out. Unstuck
the OSX and Android autobuilders. Got drag and drop repository list
reordering working great. Tons of changes in this release!

Also put up a new podcast.


code:
I've reworked the UI of the webapp's dashboard. Now the repository list is
included, above the transfers. I found I was spending a lot of time
switching between the dashboard and repository list, so might as well
combine them into a single screen. Yesod's type safe urls and widgets
made this quite easy to do, despite it being a thousand line commit.
Liking the result ... Even though it does make all my screencasts dated.

[[!img /assistant/dashboard.png]]

----

Rest of my time was spent on XMPP pairing UI. Using the same pages for both
pairing with a friend and for self-pairing was confusing, so now the two
options are split.

Now every time an XMPP git push is received or sent, it checks if there's
a cloud repository configured, which is needed to send the contents of
files. If not, it'll display this alert. Hopefully this
will be enough to get users fully set up.

[[!img /assistant/cloudnudge.png]]

At this point I'm finally happy enough with the XMPP pairing + cloud
repository setup process to film a screencast of it. As soon as I have
some time & bandwidth someplace quiet. Expect one by the end of the month.


code:
A long time ago I made Remote be an instance of the Ord typeclass, with an
implementation that compared the costs of Remotes. That seemed like a good
idea at the time, as it saved typing.. But at the time I was still making
custom Read and Show instances too. I've since learned that this is *not* a
good idea, and neither is making custom Ord instances, without deep thought
about the possible sets of values in a type.

This Ord instance came around and bit me when I put Remotes into a Set,
because now remotes with the same cost appeared to be in the Set even if
they were not. Also affected putting Remotes into a Map. I noticed this
when the webapp got confused about which Remotes were paused.

Rarely does a bug go this deep. I've fixed it comprehensively, first
removing the Ord instance entirely, and fixing the places that wanted to
order remotes by cost to do it explicitly. Then adding back an Ord instance
that is much more sane. Also by checking the rest of the Ord (and Eq)
instances in the code base (which were all ok).

While doing that, I found lots of places that kept remotes in Maps and
Sets. All of it was probably subtly broken in one way or another before
this fix, but it would be hard to say exactly how the bugs would
manifest.

-----

Also fought some with Google Talk today. Seem to be missing presence
messages sometimes. Ugh. May have fixed it, but I've thought that before..

Made --debug include a sanitized dump of the XMPP protocol.

Made UI changes to encourage user to install git-annex on the server when
adding a ssh server, rather than just funneling them through to rsync.

Fixed UI glitches in XMPP username/password prompt.

Switched all forms in the webapp to use POST, to avoid sensitive
information leaking on the url bar.

----

Added an incremental backup group. Repositories in this group only want
files that have not been backed up somewhere else yet.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnR7hb8IaKB3IKZptRukje0yahmhfLOO98"
 nickname="Adam"
 subject="comment 1"
 date="2013-03-17T00:30:26Z"
 content="""
> Made UI changes to encourage user to install git-annex on the server when adding a ssh server, rather than just funneling them through to rsync.

What's the difference between having git-annex available versus not?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnxp2XU8gIribhhGhGuYtU6eMMwHv5gUGI"
 nickname="Amitai"
 subject="Google Talk silently drops non-Google invites (at least)"
 date="2013-03-17T13:44:18Z"
 content="""
Have you seen <http://www.fsf.org/blogs/sysadmin/google-backslides-on-federated-instant-messaging-on-purpose>? Maybe they're also doing other funny stuff that gets in your way.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnR7hb8IaKB3IKZptRukje0yahmhfLOO98"
 nickname="Adam"
 subject="comment 3"
 date="2013-03-25T01:58:23Z"
 content="""
About the UI prompt for the webapp:

> What's the difference between having git-annex available versus not?

I can see from the UI that if you make it be a git repository, you get this:

> All your data will be uploaded to the server, including the full git repository. This is a great choice if you want to set up other devices to use the same server, or share the repository with others.

Versus an rsync repo:

> The contents of your files will be stored, fully encrypted, on the server. The server will not store other information about your git repository. This is the best choice if you don't run the server yourself, or have sensitive data.

Unfortunately I'm looking for a shared server (as per the git repo description), but **also fully encrypted** because I'll be storing sensitive data.
"""]]


code:
Is what I planned to do on git-annex today. Instead I fixed several bugs,
but I'm drawing the line at blogging. Oops.


code:
Triaged some of the older bugs and was able to close a lot of them.

-----

Should mention that I will be in Boston this weekend, attending
[LibrePlanet 2013](http://libreplanet.org/wiki/LibrePlanet:Conference/2013).
Drop by and find me, I'll have git-annex stickers! ;)

-----

Did some UI work on the webapp. Minor stuff, but stuff that needed to be
fixed up. Like inserting zero-width spaces into filenames displayed in it
so very long filenames always get reasonably wrapped by the browser.
(Perhaps there's a better way to do that with CSS?)


code:
[[!comment format=mdwn
 username="http://jasonwoof.com/"
 nickname="JasonWoof"
 subject="wrapping css"
 date="2013-03-20T20:28:04Z"
 content="""
You can do this in css:

    word-wrap: break-word;

That should make the text not leave the box. Not sure if this makes it break \"reasonably\" though, eg a two letter word followed by an 200 letter \"word\" would probably leave the two letter word on a line by itself.

Worth a try though.

Seems to be well supported on browsers: [http://caniuse.com/#search=wrap](http://caniuse.com/#search=wrap)
"""]]


code:
Worked today on two action items from my last blog post:

* on-disk transfers in progress information files (read/write/enumerate)
* locking for the files, so redundant transfer races can be detected,
  and failed transfers noticed

That's all done, and used by the `get`, `copy`, and `move` subcommands.

Also, I made `git-annex status` use that information to display any
file transfers that are currently in progress:

	joey@gnu:~/lib/sound/misc>git annex status
	[...]
	transfers in progress:
		downloading Vic-303.mp3 from leech

(Webapp, here we come!)

However... Files being sent or received by `git-annex-shell` don't yet
have this transfer info recorded. The problem is that to do so,
`git-annex-shell` will need to be run with a `--remote=` parameter. But
old versions will of course fail when run with such an unknown parameter.

This is a problem I last faced in December 2011 when adding the `--uuid=`
parameter. That time I punted and required the remote `git-annex-shell` be
updated to a new enough version to accept it. But as git-annex gets more widely
used and packaged, that's becoming less an option. I need to find a real
solution to this problem.