code:
Now the webapp can set up encrypted repositories on removable drives.

[[assistant/encryptdrive.png]]

This UI needs some work, and the button to create a new key is not wired
up. Also if you have no gpg agent installed, there will be lots of password
prompts at the console.

Forked git-remote-gcrypt to fix a bug. Hopefully my patch will be merged;
for now I recommend installing my worked version.

Today's work was sponsored by Romain Lenglet.


code:
When generating a view, there's now a way to reuse part of the directory
hierarchy of the parent branch. For example, `git annex view tag=* podcasts/=*`
makes a view where the first level is the tags, and the second level is
whatever `podcasts/*` directories the files were in.

Also, year and month metadata can be automatically recorded when
adding files to the annex. I made this only be done when annex.genmetadata
is turned on, to avoid polluting repositories that don't want to use metadata.

It would be nice if there was a way to add a hook script that's run
when files are added, to collect their metadata. I am not sure yet if
I am going to add that to git-annex though. It's already possible to do via
the regular git `post-commit` hook. Just make it look at the commit to see
what files were added, and then run `git annex metadata` to set their
metadata appropriately. It would be good to at least have an example of
such a script to eg, extract EXIF or ID3 metadata. Perhaps someone can
contribute one?

repo_name: kakafarm/git

license: unknown


code:
Turns out that in the last release I broke making box.com, Amazon S3 and
Glacier remotes from the webapp. Fixed that.

Also, dealt with changes in the haskell DAV library that broke support for
box.com, and worked around an exception handling bug in the library.

I think I should try to enhance the test suite so it can run live tests
on special remotes, which would at least have caught the some of these
recent problems...

----

Since metadata is tied to a particular key, editing an annexed file,
which causes the key to change, made the metadata seem to get lost.

I've now fixed this; it copies the metadata from the old version to the new
one. (Taking care to copy the log file identically, so git can reuse its
blob.)

That meant that `git annex add` has to check every file it adds to see if
there's an old version. Happily, that check is fairly fast; I benchmarked my
laptop running 2500 such checks a second. So it's not going to slow things
down appreciably.


code:
More Windows porting. Made the build completely `-Wall` safe on Windows.
Fixed some DOS path separator bugs that were preventing WebDav from
working. Have now tested both box.com and Amazon S3 to be completely
working in the webapp on Windows.


code:
Not a lot accomplished today. Some release prep, followed up to a few bug
reports.

Split git-annex's .git/annex/tmp into two directories. .git/annex/tmp will
now be used only for partially transferred objects, while
.git/annex/misctmp will be used for everything else. In particular this
allows symlinking .git/annex/tmp to a ram disk, if you want to do that.
(It's not possible for .git/annex/misctmp to be on a different filesystem
from the rest of the repository for various reasons.)

Beat on Windows XMPP for several more painful hours. Got all the haskell
bindings installed, except for gnuidn. And patched network-client-xmpp to
build without gnuidn. Have not managed to get it to link.


code:
Did not plan to work on git-annex today..

Unexpectedly ended up making the webapp support HTTPS. Not by default,
but if a key and certificate are provided, it'll use them. Great for
using the webapp remotely! See the new tip: [[tips/remote_webapp_setup]].

Also removed support for --listen with a port, which was buggy and not
necessary with HTTPS.

Also fixed several webapp/assistant bugs, including one that let it be run in
a bare git repository.

And, made the quvi version be probed at runtime, rather than compile time.


code:
Worked on metadata and views. Besides bugfixes, two features of note:

Made git-annex run a hook script, pre-commit-annex. And I wrote a
sample script that extracts metadata from lots of kinds of files,
including photos and sound files, using extract(1) to do the heavy lifting.
See [[tips/automatically_adding_metadata]].

Views can be filtered to not include a tag or a field.
For example, `git annex view tag=* !old year!=2013`

Today's work was sponsored by Stephan Schulz


code:
Preparing for a release (probably tomorrow or Friday).

Part of that was updating the autobuilders. Had to deal with the gnutls
security hole fix, and upgrading that on the OSX autobuilder turned out to
be quite complicated due to library version skew. Also, I switched the
linux autobuilders over to building from Debian unstable, rather than
stable. That should be ok to do now that the standalone build bundles all
the libraries it needs... And the arm build has always used unstable, and
has been reported working on a lot of systems. So I think this will be
safe, but have backed up the old autobuilder chroots just in case.

Also been catching up on bug reports and traffic and
and dealt with quite a lot of things today. Smarter log file
rotation for the assistant, better webapp behavior when git is not
installed, and a fix for the webdav 5 second timeout problem.

Perhaps the most interesting change is a new `annex.startupscan` setting,
which can be disabled to prevent the assistant from doing the expensive
startup scan. This means it misses noticing any files that changed since it
last run, but this should be useful for those really big repositories.

(Last night, did more work on the test suite, including even more checking
of merge conflict resolution.)

----

Today's work was sponsored by Michael Alan Dorman.


code:
I decided to keep gpg key generation very simple for now. So it generates a
special-purpose key that is only intended to be used by git-annex. It
hardcodes some key parameters, like RSA and 4096 bits (maximum recommended
by gpg at this time). And there is no password on the key, although you can
of course edit it and set one. This is because anyone who can access the
computer to get the key can also look at the files in your git-annex
repository. Also because I can't rely on gpg-agent being installed
everywhere. All these simplifying assumptions may be revisited later, but
are enough for now for someone who doesn't know about gpg (so doesn't
have a key already) and just wants an encrypted repo on a
removable drive.

Put together a simple UI to deal with gpg taking quite a while to
generate a key ...

[[assistant/genkey.png]]

[[assistant/repoinfo.png]]

Then I had to patch git-remote-gcrypt again, to have a per-remote
signingkey setting, so that these special-purpose keys get used for signing
their repo.

Next, need to add support for adding an existing gcrypt repo as a remote
(assuming it's encrypted to an available key). Then, gcrypt repos on ssh
servers..

----

Also dealt with build breakage caused by a new version of the Haskell DNS
library.

----

Today's work was sponsored by Joseph Liu.


code:
[[!comment format=mdwn
 username="http://nullroute.eu.org/~grawity/"
 nickname="Mantas"
 subject="comment 1"
 date="2013-09-18T05:00:12Z"
 content="""
Should be \"PGP\" or *at least* \"GnuPG\".
"""]]


code:
Release made yesterday, but only finished up the armel build today.
And it turns out the OSX build was missing the webapp, so it's also been
updated today.

Post release bug triage including:

Added a nice piece of UI to the webapp on user request: A "Sync now" menu
item in the repository for each repo. (The one for the current repo syncs with
all its remotes.)

Copying files to a git repository on the same computer turns out to have
had a resource leak issue, that caused 1 zombie process per file. With
some tricky monad state caching, fixed that, and also eliminated 8% of the work
done by git-annex in this case.

Fixed `git annex unused` in direct mode to not think that files that were
deleted out of the work tree by the user still existed and were unused.


code:
Squashed three or four more bugs today. Unanswered message backlog is down
to 27.

The most interesting problem today is that the git-repair code was using
too much memory when `git-fsck` output a lot of problems (300 thousand!). I
managed to half the memory use in the worst case (and reduced it much more
in more likely cases). But, don't really feel I can close that bug yet,
since really big, really badly broken repositories can still run it out of
memory. It would be good to find a way to reorganize the code so that the
broken objects list streams through git-repair and never has to all be
buffered in memory at once. But this is not easy.


code:
Updated the Debian stable backport to the last release. Also it seems that
the last release unexpectedly fixed XMPP SIGILL on some OSX machines.
Apparently when I rebuilt all the libraries recently, it somehow fixed that
[[old_unsolved_bug|bugs/Share_with_friends_crash_in_osx]].

[RichiH](http://richardhartmann.de/) suggested "wrt ballooning memory on
repair: can you read in broken
stuff and simply stop reading once you reach a certain threshold, then
start repairing, re-run fsck, etc?" .. I had considered that but was
not sure it would work. I think I've gotten it to work.

Now working on a design for using a [[design/caching_database]]
for some parts of git-annex. My initial benchmarks using SQLite
indicate it would slow down associated file lookups by nearly an order of
magnitude compared with the current ".map files" implementation.
(But would scale better in edge cases). OTOH, using a SQLite
database to index metadata for use in views looks very promising.


code:
Did some more exploration and perf tuning and thinking on caching
databases, and am pretty sure I know how I want to implement it. Will be
several stages, starting with using it for generating views, and ending(?)
with using it for direct mode file mappings.

Not sure I'm ready to dive into that yet, so instead spent the rest of the
day working on small bugfixes and improvements. Only two significant ones..

Made the webapp use a constant time string comparison (from `securemem`)
to check if its auth token is valid. This could help avoid a potential
timing attack to guess the auth token, although that is theoretical.
Just best practice to do this.

Seems that openssh 6.5p1 had another hidden surprise (in addition to
its now-fixed bug in handing hostnames in `.ssh/config`) -- it broke
the method git-annex was using for stopping a cached ssh connection,
which led to some timeouts for failing DNS lookups. If git-annex seems
to stall for a few seconds at startup/shutdown, that may be why
(--debug will say for sure). I seem to have found a workaround that
avoids this problem.


code:
Added some power and convenience to [[preferred_content]] expressions.

Before, "standard" was a special case. Now it's a first-class keyword,
so you can do things like "standard or present" to use the standard
preferred content expression, modified to also want any file that happens
to be present.

Also added a way to write your own reusable preferred content expressions,
tied to groups. To make a repository use them, set its preferred
content to "groupwanted". Of course, "groupwanted" is also a first-class
keyword, so "not groupwanted" or something can also be done.

While I was at it, I made `vicfg` show the built-in standard preferred
content expressions, for reference. This little IDE should be pretty
self-explanatory, I hope.

So, preferred content is almost its own little programming language now.
Except I was careful to not allow recursion. ;)


code:
The website broke and I spent several hours fixing it, changing the
configuration to not let it break like this again, cleaning up after it,
etc.

Did manage to make a few minor bugfixes and improvements, but nothing
stunning.

----

I'll be attending LibrePlanet at MIT this weekend.


code:
Yesterday, worked on cleaning up the todo list. Fixed Windows slash problem
with rsync remotes. Today, more Windows work; it turns out to have been
quite buggy in its handling of non-ASCII characters in filenames. Encoding
stuff is never easy for me, but I eventually managed to find a way to fix
that, although I think there are other filename encoding problems lurking
in git-annex on Windows still to be dealt with.

Implemented an interesting metadata feature yesterday. It turns out that
metadata can have metadata. Particularly, it can be useful to know when a
field was last set. That was already beeing tracked, internally (to make
union merging work), so I was able to quite cheaply expose it as
"$field-lastchanged" metadata that can be used like any other metadata.

I've been thinking about how to implement [[todo/required_content]]
expressions, and think I have a reasonably good handle on it.


code:
Yesterday coded up one nice improvement on the plane -- `git annex unannex`
(and `uninit`) is now tons faster. Before it did a git commit after every
file processed, now there's just 1 commit at the end. This required using
some locking to prevent the `pre-commit` hook from running in a confusing
state.

Today. LibrePlanet and a surprising amount of development. I've
added [[tips/file_manager_integration]], only for Nautilus so far.
The main part of this was adding --notify-start and --notify-finish, which
use dbus desktop notifications to provide feedback.

(Made possible thanks to Max Rabkin for updating
[fdo-notify](http://hackage.haskell.org/package/fdo-notify) to use the
new dbus library, and ion for developing the initial Nautilus integration
scripts.)

Today's work and LibrePlanet visit was sponsored by Jürgen Lüters.


code:
Spent basically all of today getting the assistant to be able to handle
gcrypt special remotes that already exist when it's told to add a USB
drive. This was quite tricky! And I did have to skip handling gcrypt repos
that are not git-annex special remotes.

Anyway, it's now almost easy to set up an encrypted sneakernet
using a USB drive and some computers running the webapp. The only part
that the assistant doesn't help with is gpg key management.

Plan is to make a release on Friday, and then try to also add support for
encrypted git repositories on remote servers. Tomorrow I will try to get
through some of the communications backlog that has been piling up while I
was head down working on gcrypt.


code:
Attended at the f-droid sprint at LibrePlanet, and have been getting a
handle on how their build server works with an eye toward adding git-annex
to it. Not entirely successful getting vagrant to build an image yet.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmJuOOkYYguRbWhXzxihIPBavxITJIMyww"
 nickname="Matt"
 subject="Excellent News"
 date="2014-03-26T14:14:35Z"
 content="""
It will be great to see git-annex on f-droid!
"""]]


code:
Catching up on conference backlog. 36 messages backlog remains.

Fixed `git-annex-shell configlist` to automatically initialize a
git remote when a git-annex branch had been pushed to it. This is necessary
for gitolite to be easy to use, and I'm sure it used to work.

Updated the Debian backport and made a Debian package of the
fdo-notify haskell library used for notifications.

Applied a patch from Alberto Berti to fix support for tahoe-lafs
1.10.

And various other bug fixes and small improvements.


code:
Last week's trip was productive, but I came home more tired than I
realized. Found myself being snappy & stressed, so I have been on break.

I did do a little git-annex dev in the past 5 days. On Saturday I
implemented [[todo/preferred_content]] (although without the active checks
I think it probably ought to have.) Yesterday I had a long conversation
with the Tahoe developers about improving git-annex's tahoe integration.

Today, I have been wrapping up [building propellor](http://joeyh.name/code/propellor/).
To test its docker support, I used propellor to build and deploy a
container that is a git-annex autobuilder. I'll be replacing the old
autobuilder setup with this shortly, and expect to also publish docker
images for git-annex autobuilders, so anyone who wants to can run their
own autobuilder really easily.

---

I have April penciled in on the roadmap as the month to do telehash.
I don't know if telehash-c is ready for me yet, but it has had a lot of
activity lately, so this schedule may still work out!


code:
Got caught up on all recent bugs and questions, although I still have a
backlog of 27 older things that I really should find time for.

Fixed a couple of bugs. One was that the assistant set up ssh
`authorized_keys` that didn't work with the fish shell.

Also got caught up on the current state of telehash-c. Have not quite
gotten it to work, but it seems pretty close to being able to see it do
something useful for the first time.

Pushing out a release this evening with a good number of changes left over
from March.


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="70.82.37.38"
 subject="comment 1"
 date="2014-04-02T21:28:15Z"
 content="""
awesome! can't wait to see telehash land! :)
"""]]


code:
I have a plan for this month. While waiting for telehash, I am going to
build [[design/git-remote-daemon]], which is the infrastructure git-annex
will need, to use telehash. Since it's generalized to support other protocols,
I'll be able to start using it before telehash is ready.

In fact, I plan to first make it work with ssh:// remotes, where
it will talk with git-annex-shell on the remote server. This will let the
assistant immediately know when the server has received a commit, and that
will simplify using the assistant with a ssh server -- no more need for
XMPP in this case! It should also work with git-remote-gcrypt encrypted
repositories, so also covers the case of an untrusted ssh server where
everything is end-to-end encrypted.

Building the git-annex-shell part of this should be pretty easy, and
building enough of the [[design/git-remote-daemon]] design to support it
also not hard.


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="2001:1928:1:9::1"
 subject="look at mosh"
 date="2014-04-04T14:07:51Z"
 content="""
you may want to look at how \"mosh\" handles authentication: http://mosh.mit.edu/

from what i understand, it negociates an authentication token using SSH and then uses that to encrypt and authenticate UDP traffic. seems like similar issues here...
"""]]


code:
Added `git-annex-shell notifychanges` command, which uses inotify (etc)
to detect when git refs have changed, and informs the caller about the
changes. This was relatively easy to write; I reused the existing inotify
code, and factored out code for simple line-based protocols from the
external special remote protocol. Also implemented the git-remote-daemon
protocol. 200 lines of code total.

Meanwhile, Johan Kiviniemi improved the dbus notifications, making them
work on Ubuntu and adding icons. Awesome!

There's going to be some fun to get git-annex-shell upgraded so that the
assistant can use this new notify feaure. While I have not started working
on the assistant side of this, you can get a jump by installing today's
upcoming release of git-annex. I had to push this out early because there
was a bug that prevented the webapp from running on non-gnome systems. Since
all changes in this release only affected Linux, today's release will be a
Linux-only release.


code:
Built `git-annex remotedaemon` command today. It's buggy, but it already
works! If you have a new enough git-annex-shell on a remote server, you can
run "git annex remotedaemon" in a git-annex repository, and it will notice
any pushes that get made to that remote from any other clone, and pull down
the changes.


code:
Various bug triage today. Was not good for much after shuffling paper for
the whole first part of the day, but did get a few little things done.

Re <http://heartbleed.com/>, git-annex does not use OpenSSL itself,
but when using XMPP, the remote server's key could have been intercepted
using this new technique. Also, the git-annex autobuilds and this website
are served over https -- working on generating new https certificates now.
Be safe out there..


code:
[[design/git-remote-daemon]] is tied into the assistant, and working!
Since it's not really ready yet, this is in the `remotecontrol` branch.

My test case for this is two client repositories, both running
the assistant. Both have a bare git repository, accessed over ssh,
set up as their only remote, and no other way to keep in touch with
one-another. When I change a file in one repository,
the other one instantly notices the change and syncs.

This is gonna be *awesome*. Much less need for XMPP. Windows will be fully
usable even without XMPP. Also, most of the work I did today will be fully
reused when the telehash backend gets built. The telehash-c developer is
making noises about it being almost ready for use, too!

Today's work was sponsored by Frédéric Schütz.


code:
[[!meta title="day 150 signal"]]

The git-remote-daemon now robustly handles loss of signal, with
reconnection backoffs. And it detects if the remote ssh server has too
old a version of git-annex-shell and the webapp will display a warning
message.

[[!img /assistant/connection.png]]

Also, made the webapp show a network signal bars icon next to both
ssh and xmpp remotes that it's currently connected with. And, updated the
webapp's nudging to set up XMPP to now suggest either an XMPP or a ssh remote.

I think that the `remotecontrol` branch is nearly ready for merging!

Today's work was sponsored by Paul Tagliamonte.


code:
Spent a few hours improving gcrypt in some minor ways, including adding a
--check option that the assistant can use to find out if a given repo is
encrypted with dgit, and also tell if the necessary gpg key is available to
decrypt it. Also merged in a fix to support subkeys, developed by a
git-annex user who is the first person I've heard from who is using gcrypt.
I don't want to maintain gcrypt, so I am glad its author has shown up
again today.

Got mostly caught up on backlog. The main bug I was able to track down
today is git-annex using a lot of memory in certian repositories. This
turns out to have happened when a really large file was committed right
intoo to the git repository (by mistake or on purpose). Some parts of
git-annex buffer file contents in memory while trying to work out if
they're git-annex keys. Fixed by making it first check if a file in git is
marked as a symlink. Which was really hard to do!

At least 4 people ran into this bug, which makes me suspect that lots of
people are messing up when using direct mode (probably due to not reading
the documentation, or having `git commit -a` hardwired into their fingers,
and forcing git to commit large files into their repos, rather than having
git-annex manage them. Implementing [[todo/direct_mode_guard]] seems more
urgent now.

----

Today's work was sponsored by Amitai Schlair.


code:
Made a release on Friday. But I had to rebuild the OSX and Linux standalone
builds today to fix a bug in them.

Spent the past **three days** redoing the whole Android build environment.
I've been progressively moving from my first hacked up Android build env to
something more reproducible and sane. Finally I am at the point where I can
run a shell script (well, actually, 3 shell scripts) and get an Android
build chroot. It's still not immune to breaking when new versions of
haskell libs are uploaded, but this is much better, and should be
maintainable going forward.

This is a good starting point for getting git-annex into the F-Droid app
store, or for trying to build with a newer version of the Android SDK and
NDK, to perhaps get it working on Android 4.3. (Eventually. I am so sick
of building Android stuff right now..)

Friday was all spent struggling to get ghc-android to build. I had not built
it successfully since February. I finally did,
on Saturday, and I have made my own fork of it which builds using a
known-good snapshot of the current development version of ghc. Building
this in a Debian stable chroot means that there should be no possibility
that upstream changes will break the build again.

With ghc built, I moved on to building all the haskell libs git-annex
needs. Unfortunately my build script for these also has stopped working
since I made it in April. I failed to pin every package at a defined
version, and things broke.

So, I redid the build script, and updated all the haskell libs to the
newest versions while I was at it. I have decided not to pin the library
versions (at least until I find a foolproof way to do it), so this new
script will break in the future, but it should break in a way I can fix up
easily by just refreshing a patch.

The new ghc-android build has a nice feature of at least being able to
compile Template Haskell code (though still not run it at compile time.
This made the patching needed in the Haskell libs quite a lot less. Offset
somewhat by me needing to make general fixes to lots of libs to build with
ghc head. Including some fun with `==#` changing its type from `Bool` to
`Int#`. In all, I think I removed around 2.5 thousand lines of patches!
(Only 6 thousand lines to go...)

Today I improved ghc-android some more so it cross builds several C libraries
that are needed to build several haskell libraries needed for XMPP.
I had only ever built those once, and done it by hand, and very hackishly.
Now they all build automatically too.

And, I put together a script that builds the debian stable chroot and
installs ghc-android.

And, I hacked on the EvilSplicer (which is sadly still needed) to
work with the new ghc/yesod/etc.

At this point, I have git-annex successfully building, including the APK!

----

In a bored hour waiting for a compile, I also sped up `git annex add`
on OSX by I think a factor of 10. Using cryptohash for hash calculation
now, when external hash programs are not available. It's still a few
percentage points slower than external hash programs, or I'd use it by
default.

----

This period of important drudgery was sponsored by an unknown bitcoin
user, and by Bradley Unterrheiner and Andreas Olsson.


code:
Pushed out a new release today, fixing two important bugs, followed by a
second release which fixed the bugs harder.

Automatic upgrading was broken on OSX. The webapp will tell you upgrading
failed, and you'll need to manually download the .dmg and install it.

With help from Maximiliano Curia, finally tracked down a bug I have been
chasing for a while where the assistant would start using a lot of CPU
while not seeming to be busy doing anything. Turned out to be triggered by
a scheduled fsck that was configured to run once a month with no particular
day specified.

That bug turned out to affect users who first scheduled such a fsck job
after the 11th day of the month. So I expedited putting a release out to
avoid anyone else running into it starting tomorrow.

(Oddly, the 11th day of this month also happens to be my birthday. I did not
expect to have to cut 2 releases today..)


code:
Made ssh connection caching be used in several more places. `git annex
sync` will use it when pushing/pulling to a remote, as will the assistant.
And `git-annex remotedaemon` also uses connection caching. So, when
a push lands on a ssh remote, the assistant will immediately notice it, and
pull down the change over the same TCP connection used for the
notifications.

This was a bit of a pain to do. Had to set `GIT_SSH=git-annex` and then
when git invokes git-annex as ssh, it runs ssh with the connection caching
parameters.

Also, improved the network-manager and wicd code, so it detects when a
connection has gone down. That propagates through to the remote-daemon,
which closes all ssh connections. I need to also find out how to detect
network connections/disconnections on OSX..

Otherwise, the remote-control branch seems ready to be merged. But I want
to test it for a while first.

----

Followed up on yesterday's bug with writing some test cases for
Utility.Scheduled, which led to some more bug fixes. Luckily nothing
I need to rush out a release over. In the end, the code got a lot
simpler and clearer.

[[!format haskell """
-- Check if the new Day occurs one month or more past the old Day.
oneMonthPast :: Day -> Day -> Bool
new `oneMonthPast` old = fromGregorian y (m+1) d <= new
  where
        (y,m,d) = toGregorian old
"""]]

-------

Today's work was sponsored by Asbjørn Sloth Tønnesen.


code:
After fixing a few bugs in the `remotecontrol` branch, It's landed in
`master`. Try a daily build today, and see if the assistant can keep in
sync using nothing more than a remote ssh repository!

So, now all the groundwork for telehash is laid too. I only need a
telehash library to start developing on top of. Development on telehash-c
is continuing, but I'm more excited that
[htelehash](https://github.com/alanz/htelehash/tree/v2)
has been revived and is being updated to the v2 protocol, seemingly quite
quickly.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkC0W3ZQERUaTkHoks6k68Tsp1tz510nGo"
 nickname="Georg"
 subject=" remotedaemon in pre-built tarballs"
 date="2014-04-17T07:40:25Z"
 content="""
Hi Joey,

can you tell me when the pre-built Linux tarballs will include the remotedaemon?
Are they updated on a daily basis?

Best regards, Georg
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.191"
 subject="comment 2"
 date="2014-04-17T19:31:30Z"
 content="""
The daily builds are updated on a daily basis, so already include it.

The release builds are updated on each release, and so do not.
"""]]


code:
Worked through message backlog today. Got it down from around 70 to just
37. Was able to fix some bugs, including making the webapp start up more
robustly in some misconfigurations.

Added a new `findref` command which may be useful in a git `update` hook to
deny pushes of refs if the annexed content has not been sent first.

----

BTW, I also added a new `reinit` command a few days ago, which can be
useful if you're cloning back a deleted repository.

Also a few days ago, I made `uninit` a *lot* faster.


code:
Sometimes you don't notice something is missing for a long time until
it suddenly demands attention. Like today.

Seems the webapp never had a way to stop using XMPP and delete the XMPP
password. So I added one.

The new support for instantly noticing changes on a ssh remote forgot to
start up a connection to a new remote after it was created. Fixed that.

(While doing some testing on Android for unrelated reasons, I noticed that
my android tablet was pushing photos to a ssh server and my laptop
immediately noticed and downloaded them from tere, which is an excellent
demo. I will deploy this on my trip in Brazil next week. Yes, I'm spending
2 weeks in Brazil with git-annex users; more on this later.)

Finally, it turns out that "installing" git-annex from the standalone
tarball, or DMG, on a server didn't make it usable by the webapp. Because
git-annex shell is not in PATH on the server, and indeed git and rsync may
not be in PATH either if they were installed with the git-annex bundle.
Fixed this by making the bundle install a ~/.ssh/git-annex-wrapper, which
the webapp will detect and use.

Also, quite a lot of other bug chasing activity.

----

Today's work was sponsored by Thomas Koch.


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="72.0.72.144"
 subject="comment 1"
 date="2014-04-20T23:58:51Z"
 content="""
thanks!
"""]]


code:
I hope this will be a really good release. Didn't get all the way to
[[design/assistant/telehash]] this month, but the remotedaemon is pretty sweet. Updated [[design/roadmap]]
pushes telehash back again.

The files in this release are now gpg signed, after recently moving the
downloads site to a dedicated server, which has a dedicated gpg key.
You can verify the detached signatures as an additional security check
over trusting SSL. The automatic upgrade code doesn't check the gpg
signatures yet.

Sören Brunk has ported the webapp to Bootstrap 3.
<https://github.com/brunksn/git-annex/tree/bootstrap3>
The branch is not ready for merging yet (it would break the Debian stable
backports), but that was a nice surprise.


code:
Now git-annex's self-upgrade code will check the gpg signature of a
new version before using it.

To do this I had to include the gpg public keys into the
git-annex distribution, and that raised the question of which public keys
to include. Currently I have both the dedicated git-annex distribution
signing key, and my own gpg key as a backup in case I somehow misplace the
former.

Also spent a while looking at the recent logs on the web server. There
seem to be around 600 users of the assistant with
upgrade checking enabled. That breaks down to 68% Linux amd64, 20% Linux
i386, 11% OSX Mavericks, and 0.5% OSX Lion.

Most are upgrading successfully, but there are a few that seem to
repeatedly fail for some reason. (Not counting the OSX Lion, which will
probably never find an upgrade available.) I hope that someone who is
experiencing an upgrade failure gets in touch with some debug logs.

In the same time period, around 450 unique hosts manually downloaded a
git-anex distribution. Also compare with Debian popcon, which has 1200
reporting git-annex users.


code:
Next month the roadmap has me working on [[design/assistant/sshpassword]].
That will be a nice UI improvement and I'd be very surprised if it takes
more than a week, which is great.

Getting a jump on it today, investigating using `SSH_ASKPASS`. It seems this
will even work on Windows! Preliminary design in [[design/assistant/sshpassword]].

Time to get on a plane to a plane to a plane to Brasilia!

[[!meta date="Fri, 25 Apr 2014 16:32:36 -0400"]]


code:
Today was mostly spent driving across Brazil, but I had energy this evening
for a little work on git-annex.

Made the assistant delete old temporary files on startup. I've had
scattered reports of a few users whose `.git/annex/tmp` contained many
files, apparently put there by the assistant when it locks down a file
prior to annexing it. That seems it could possibly be a bug -- or it could
just be unclean shutdowns interrupting the assistant. Anyway, this will
deal with any source of tmp cruft, and I made sure to preserve
tmp files for partially downloaded content.

[[!meta date="Sun, 27 Apr 2014 22:12:55 -0300"]]


code:
Reviewed Sören's updated bootstrap3 patch, which appeared while I was
[traveling](http://joeyh.name/blog/entry/the_real_Brazil/). Sören
kindly fixed it to work with Debian stable's old version of Yesod,
which was quite a lot of work. The new new bootstrap3 UI looks nice,
found a few minor issues, but expect to be able to merge it soon.

Started on [[design/assistant/sshpassword]] groundwork. Added a simple
password cache to the assistant, with automatic expiration, and made
git-annex be able to be run by ssh as the `SSH_ASKPASS` program.

The main difficulty will be changing the webapp's UI to prompt for the ssh
password when one is needed. There are several code paths in ssh remote
setup where a password might be needed. Since the cached password expires,
it may need to be prompted for at any of those points. Since a new page is
loading, it can't pop up a prompt on the current page; it needs to redirect
to a password prompt page and then redirect back to the action that needed
the password. ...At least, that's one way to do it.
I'm going to sleep on it and hope I dream up a better way.

[[!meta date="Tue, 29 Apr 2014 18:33:53 -0400"]]


code:
I've moved out of implementation mode (unable to concentrate enough), and
into high-level design mode.

[[Syncing efficiency|design/assistant/syncing/efficiency]] has been an open TODO for years,
to find a way to avoid flood filling the network, and find more efficient
ways to ensure data only gets to the nodes that want it. Relatedly,
Android devices  often need a way to mark individual files they want to have.
Had a [very productive discussion with Vince and Fernao](http://joeyh.name/blog/entry/who_needs_whiteboards_when_you_have_strange_seed_pods_from_the_jungle/)
and I think we're heading toward a design that will address both these
needs, as well as some more Brazil-specific use cases, about which more
later.

Today's work was sponsored by Casa do Boneco.


code:
Sören Brunk's massive bootstrap 3 patch has landed! This is a 43 thousand
line diff, with 2 thousand lines after the javascript and CSS libraries are
filtered out. Either way, the biggest patch contributed by anyone to
git-annex so far, and excellent work.

Meanwhile, I built a [[haskell program|design/assistant/syncing/simroutes.hs]] to simulate
a network of highly distributed git-annex nodes with ad-hoc connections and
the selective file syncing algorythm now documented at the bottom of
[[design/assistant/syncing/efficiency]].

Currently around 33% of requested files never get to their destination
in this simulation, but this is probably because its network is randomly
generated, and so contains disconnected islands. So next, some data entry,
from a map that involves an Amazon not in .com, dotted with names of people
I have recently met... :)


code:
I have a preliminary design for [[design/requests_routing]]. Won't be
working on it immediately, but simulations show it can work well in a large
ad-hoc network.


code:
My backlog is massive -- 181 items to answer. Will probably take the rest
of the month to get caught back up. Rather than digging into that yet,
spent today working on the webapp's ssh password prompting.

I simplified it so the password is entered on the same form as the rest
of the server's information. That made the UI easy to build, but
means that when a user already has a ssh key they want to use, they need
to select "existing ssh key"; the webapp no longer probes to automatically
detect that case.

Got the ssh password prompting in the webapp basically working, and it's a
*really* nice improvement! I even got it to work on Windows (eventually...).
It's still only in the `sshpassword` branch, since I need to test it more
and probably fix some bugs. In particular, when enabling a remote that
already exists, I think it never prompts for the password yet.

Today's work was sponsored by Nicola Chiapolini.


code:
Spent the day testing the sshpasswd branch. A few interesting things:

* I was able to get rid of 10 lines of Windows specific code for
  rsync.net, which had been necessary for console ssh password prompting to
  work. Yay!
* git-remote-gcrypt turned out to be broken when there is no controlling
  tty. --no-tty has to be passed to gpg to avoid it falling over in this
  case, even when a gpg agent is available to be used. I fixed this with
  a new release of git-remote-gcrypt.

Mostly the new branch just worked! And is merged...

Merged a patch from Robie Basak that adds a new special remote that's sort
of like bup but supports deletion: [[special_remotes/ddar]]

Backlog: 172

Today's work was sponsored by Andrew Cant.


code:
Powered through the backlog today, and got it down to 67! Probably most of
the rest is the hard ones though.

A theme today was: It's stupid hard to get git-annex-shell installed into
PATH. While that should be the simplest thing in the world, I'm pinned
between two problems:

1. There's no single portable package format, so all the decades of
   development nice ways to get things into PATH don't work for everybody.
2. bash provides not a single dotfile that will work in all circumstances
   to configure PATH. In particular, "ssh $host git-annex-shell" causes bash
   to helpfully avoid looking at any dotfiles at all.

Today's flailing to work around that inluded:

* Merged a patch from Fraser Tweedale to allow `git config remote.origin.annex-shell /not/in/path/git-annex-shell`
* Merged a patch from Justin Lebar to allow symlinking the git-annex-shell
  etc from the standalone tarball to a directory that is in PATH. (Only
  on Linux, not OSX yet.)
* Improved the warning message git-annex prints when a remote server does
  not have git-annex-shell in PATH, suggesting some things the user could
  do to try to fix it.

I've found out why OSX machines were retrying upgrades repeatedly. The
version in the .info file did not match the actual git-annex version for
OSX. I've fixed the info file version, but will need to come up with a
system to avoid such mismatches.

Made a few other fixes. A notable one is that dragging and dropping
repositories in the webapp to reorder the list (and configure costs)
had been broken since November.

git-annex 5.20140421 finally got into Debian testing today, so I updated
the backport. I recommend upgrading, especially if you're using the
assistant with a ssh remote, since you'll get all of last month's nice
features that make XMPP unnecessary in that configuration.

----

Today's work was sponsored by Geoffrey Irving.


code:
Released git-annex 5.20140517 today. The changelog for this release
is very unusual, because it's full of contributions from others! There are
as many patches from others in this release as git-annex got in the first
entire two years of its existence.

I'd like to keep that going. Also, I could really use help triaging bug
reports right now. So I have updated the **[[contribute]]** page with more
info about easy ways to contribute to git-annex. If you read this devblog,
you're an ideal contributor, and you don't need to know how to write
haskell either.. So take a [[look at the page|contribute]] and see if
you can help out.


code:
Worked on triaging several bugs. Fixed an easy one, which involved the
assistant choosing the wrong path to a repository that has multiple
remotes. After today, backlog is down to 43, nearly pre-Brazil levels.

It seems that git-remote-gcrypt [[never quite worked on OSX|bugs/remote_gcrypt_based_repos_and_conflicting_uuids]].
It looked like it did, but a bug prevented anything being pushed to the remote.
Tracked down and fixed that bug.

This evening, getting back to working on the armel autobuilder setup
using propellor. The autobuilder will use a pair of docker containers, one
armel and a companion amd64, and their quite complex setup will be *almost*
fully automated (except for the haskell library patching part).

----

Today's work was sponsored by Mica Semrick.


code:
Keeping lots of things going these past few days..

* Rebootstrapping the armel autobuilder with propellor.
  Some qemu instability and the need to update haskell
  library patches meant this took a lot of hand-holding. Finally got a
  working setup today.
* Designing and ordering new git-annex stickers on clear viynl backing;
  have put off sending those to campaign contributors for too long.
* Added a new feature to the webapp: It now remembers the ssh remotes
  that it sets up, and makes it easy to enable them elsewhere, the same
  as other sorts of remotes. Had a very pleasant surprise building this,
  when I was able to reuse all the UI code for enabling rsync and gcrypt
  remotes. I think this will be a useful feature as we transition away
  from XMPP.


code:
Working on moving the android autobuilder to Docker & Propellor, which will
finish containerizing all the autobuilds that I run. Updated
ghc-android to use the released ghc 7.8.2, which will make it build more
reliably.

Also did bug triage. Bugs are now divided into [[bugs/confirmed]]
and [[bugs/unconfirmed]] categories.


code:
With some help from Sören, have been redoing the android build environment
for git-annex. This included making propellor put it in a docker container,
which was easy. But then much struggling with annoying stuff like getting
the gnutls linking to work, and working around some dependency issues on
hackage that make cabal's dependency resolver melt down.
Finally succeeded after much more time than I had wanted to spend on this.

[[!meta date="Mon May 27 16:36:40 JEST 2014"]]


code:
These themed days are inaverdent, but it happened again: Nearly everything
done today had to do with encoding issues.

The big news is that it turned out everything written to files in the
git-annex branch had unicode characters truncated to 8 bits. Now fixed so
you should always get out the same thing you put in, no matter what
encoding you use (but please use utf8). This affected things like storing
repository descriptions, but worse, it affected metadata. (Also preferred
content expressions, I suppose.)

With that fixed, there are still 7 source files left that use Char8 libraries.
There used to be more; nearly every use of those is a bug. I looked over
the remaining uses of it, and there might be a problem with Creds using it.
I should probably make a push to stamp out all remaining uses of Char8.

Other encoding bugs were less reproducible.

And just now, Sören made some progress on
[[bugs/Bootstrap3_icons_missing_on_Android]] ... and my current theory
is this is actually caused by an encoding issue too.


code:
Got a handle on the Android webapp static file problems (no, they were not
really encoding problems!), and hopefully
that's all fixed now. Also, only 3 modules use Char8 now. And updated the
git-annex backport. That's all I did today.

Meanwhile, a complete [[tips/ZSH_completion]] has been contributed by
Schnouki. And, Ben Gamari sent in a patch moving from the deprecated
MonadCatchIO-transformers library to the exceptions library.


code:
After making a release yesterday, I've been fixing some bugs in the
webapp, all to do with repository configuration stored on the git-annex
branch. I was led into this by a strange little bug where the webapp stored
configuration in the wrong repo in one situation. From there, I noticed
that often when enabling an existing repository, the webapp would stomp on
its group and preferred content and description, replacing them with
defaults.

This was a systematic problem, it had to be fixed in several places. And
some of the fixes were quite tricky. For example, when adding a ssh
repository, and it turns out there's already a git-annex repository at the
entered location, it needs to avoid changing its configuration. But also,
the configuration of that repo won't be known until after the first git
pull from it. So it doesn't make sense to show the repository edit form
after enabling such a repository.

Also worked on a couple other bugs, and further cleaned up the [[bugs]]
page. I think I am finally happy with how the bug list is displayed,
with confirmed/moreinfo/etc tags.

Today's work was sponsored by François Deppierraz.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 1"
 date="2014-06-02T13:36:00Z"
 content="""
Can you make the tags be displayed with background colours or some other mechanism for quick visual recognition? All \"done\" are green, all \"moreinfo\" blue(?), etc pp.

Thanks,
Richard
"""]]


code:
Yesterday I recorded a new screencast, demoing using the assistant on a
local network with a small server. [[videos/git-annex_assistant_lan]].
That's the best screencast yet; having a real framing story was nice;
recent improvements to git-annex are taken advantage of without being made
a big deal; and audio and video are improved. (But there are some minor
encoding glitches which I'd have to re-edit it to fix.)

The [[design/roadmap]] has this month dedicated to improving Android.
But I think what I'd more like to do is whatever makes the assistant usable
by the most people. This might mean doing more on Windows, since I hear
from many who would benefit from that. Or maybe something not related to
porting?


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm8BAEUyzYhORZmMuocRTk4M-3IumDm5VU"
 nickname="luciusf0"
 subject="Windows Support +1"
 date="2014-06-06T21:54:38Z"
 content="""
I just also (like others) wanna state my strong interest in a windows port. Actually it's the only thing which is holding me back of using git-annex for real, as I live and work in a mixed-OS environment. The git-annex features would be the answers to an awful lot of our day-to-day problems, and I cannot wait to have it ready for a production environment. So -> Windows Port +1
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnBJ6Dv1glxzzi4qIzGFNa6F-mfHIvv9Ck"
 nickname="Jim"
 subject="Android"
 date="2014-06-08T02:44:45Z"
 content="""
As I don't use Windows and git-annex is already fitting my Linux use cases perfectly, I'd have to vote for Android.  FWIW, I have previously posted some thoughts on what would I see as lacking on Android [here](http://git-annex.branchable.com/forum/Controlling_content_on_mobile_device/).
"""]]


code:
Did work on Windows porting today. First, fixed a reversion in the last
release, that broke the git-annex branch pretty badly on Windows, causing
\r to be written to files on that branch that should never have DOS line
endings. Second, fixed a long-standing bug that prevented getting a file
from a local bare repository on Windows.

Also refreshed all autobuilders to deal with the gnutls and openssl
security holes-of-the-week. (git-annex uses gnutls only for XMPP,
and does not use openssl itself, but a few programs bundled with it,
like curl, do use openssl.)

A nice piece of news: OSX Homebrew now contains git-annex, so it can be
easily installed with `brew install git-annex`


code:
[[!comment format=mdwn
 username="Tamschi"
 ip="178.27.213.202"
 subject="It's still impossible to use the git-annex repository on Windows."
 date="2014-06-09T14:44:07Z"
 content="""
Since the bug tracker is in the same repository and branch as the main program (why?), any bugs submitted with a colon in the title will prevent access on NTFS file systems.
There is at least one such instance, I assume it potentially affects blog posts too.
"""]]


code:
Spent most of today improving behavior when a sync or merge is
interrupted in direct mode. It was possible for an interrupt at the wrong
time to leave the merge committed, but the work tree not yet updated. And
then the next sync would make a commit that reverted the merged changes!

To fix this I had to avoid making any merge commit or indeed updating the
index until after the work tree is updated. It looked intractable for a
while; I'm still surprised I eventually succeeded.


code:
Have for the first time gotten git-annex to run as a proper Windows
service, using nssm.
([details](http://git-annex.branchable.com/todo/windows_support/#comment-a61be55862ea32e3dc30972f905bb987))
Not quite ready yet though; doesn't run as the right user.

And a few other windows porting bits.


code:
Spent all day on some horrible timestamp issues on legacy systems.

On FAT, timestamps have a 2s granularity, which is ok, but then Linux adds
a temporary higher resolution cache, which is lost on unmount. This
confused git-annex since the mtimes seemed to change and it had to
re-checksum half the files to get unconfused, which was not good.
I found a way to use the inode sentinal file to detect when on FAT
and put in a workaround, without degrading git-annex everywhere else.

On Windows, time zones are a utter disaster; it changes the mtime it reports
for files after the time zone has changed. Also there's a bug in the
haskell time library which makes it return old time zone data after a time
zone change. (I just finished developing a fix for that bug..)

Left with nothing but a few sticks, I rubbed them together, and
actually found a way to deal with this problem too. Scary details in
[[bugs/Windows_file_timestamp_timezone_madness]]. While I've implemented
it, it's stuck on a branch until I find a way to make git-annex notice when
the timezone changes while it's running.

----

Today's work was sponsored by Svenne Krap.


code:
It's officially a Windows porting month. Now that I'm half way through it
and with the last week of the month going to be a vacation, this makes
sense.

Today, finished up dealing with the timezone/timestamp issues on Windows.
This got stranger and stranger the closer I looked at it. After a timestamp
change, a program that was already running will see one timestamp, while a
program that is started after the change will see another one! My approach
works pretty much no matter how Windows goes insane though, and always
recovers a true timestamp. Yay.

Also fixed a regression test failure on Windows, which turned out to be
rooted in a bug in the command queue runner, which neglected to pass
along environment overrides on Windows.

Then I spent 5 hours tracking down a tricky
test suite failure on Windows, which turned out to also
affect FAT and be a recent reversion that has as it's
root cause a [fun bug in git itself](http://marc.info/?l=git&m=140262402204212&w=2).
Put in a not very good workaround. Thank goodness for test suites!

Also got the arm autobuilder unstuck. Release tomorrow.


code:
More work on [[todo/windows_git-annex_service]], but am stuck with a
permissions problem.

Fixed a bug that prevented two assistants from syncing when there was only
a uni-directional link between them. Only affected direct mode, and
was introduced back when I added the direct mode guard.


code:
After despairing of ever solving this yesterday (and for the past 6 months
really), I've got the webapp running on Windows with no visible DOS box.
Also have the assistant starting up in the background on login.

It turns out a service was not the way to do. There is a way to write a VB
Script that runs a "DOS" command in a hidden window, and this is what I
used. Amazing how hard it was to work this out, probably partly because I
don't have the Windows vocabulary to know what to look for.


code:
[[!comment format=mdwn
 username="https://launchpad.net/~maestro-alubia"
 nickname="maestro-alubia"
 subject="Congratulations!"
 date="2014-06-18T09:08:45Z"
 content="""
I really feel with you and I am glad I currently do not need to develop on/for Windows. I wish you good progress with Windows porting, so you can concentrate on the interesting stuff again :)

Best regards,
Fabian
"""]]


code:
[[!comment format=mdwn
 username="http://id.clacke.se/"
 nickname="Claes"
 subject="VBScript + wscript to avoid DOS window"
 date="2014-06-21T02:09:58Z"
 content="""
I use this technique too, to run cygwin xterm without an ugly DOS box popup.
"""]]


code:
[[!comment format=mdwn
 username="http://id.clacke.se/"
 nickname="Claes"
 subject="VBScript + wscript to avoid DOS window"
 date="2014-06-21T02:15:04Z"
 content="""
(sorry that I didn't see before that you had run into this annoying problem)
"""]]


code:
Last night, got logging to daemon.log working on Windows. Aside from XMPP
not working (but it's near to being deprecated anyway), and some possible
issues with unicode characters in filenames, the Windows port now seems in
pretty good shape for a beta release.

Today, mostly worked on fixing the release process so the metadata
accurarely reflects the version from the autobuilder that is included in
the release. Turns out there was version skew in the last release (now
manually corrected). This should avoid that happening again, and also
automates more of my release process.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlnoH5btjn_BLib3_IhES5uMhrzuOiwCYo"
 nickname="András"
 subject="sync options?"
 date="2014-06-19T13:16:51Z"
 content="""
What options will we have in this beta for windows <=> windows (local network) and windows <=> linux (remote server) synchronization?  I'm interested only in non-cloud sync possibilities.

(I tried to follow the developments but apparently failed...)

Also thank you for working on the windows port!

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.203"
 subject="comment 2"
 date="2014-06-19T22:13:23Z"
 content="""
windows to linux server (local or remote) to windows will work.
"""]]


code:
I am back from the beach, but my dev laptop is dead. A replacement is being
shipped, and I have spent today getting my old netbook into a usable state
so I can perhaps do some work using it in the meantime.

(Backlog is 95 messages.)


code:
Finally back to work with a new laptop!

Did one fairly major feature today: When using git-annex to pull down
podcasts, metadata from the feed is copied into git-annex's metadata store,
if annex.genmetadata is set. Should be great for views etc!

Worked through a lot of the backlog, which is down to 47 messages now.

Only other bug fix of note is a fix on Android. A recent change to git made
it try to chmod files, which tends to fail on the horrible /sdcard
filesystem. Patched git to avoid that.

For some reason the autobuilder box rebooted while I was away, and
somehow the docker containers didn't come back up -- so they got
automatically rebuilt. But I have to manually finish up building the
android and armel ones. Will be babysitting that build this evening.

Today's work was sponsored by Ævar Arnfjörð Bjarmason.


code:
Spent the morning improving behavior when `commit.gpgsign` is set.
Now git-annex will let gpg sign commits that are made when eg, manually
running `git annex sync`, but not commits implicitly made to the git-annex
branch. And any commits made by the assistant are not gpg signed. This was
slightly tricky, since lots of different places in git-annex ran `git
commit`, `git merge` and similar.

Then got back to a test I left running over vacation, that added millions
of files to a git annex repo. This was able to reproduce a problem where
`git annex add` blew the stack and crashed at the end. There turned out to
be two different memory issues, one was in git-annex and the other is in
Haskell's core `getDirectoryContents`. Was able to entirely fix it,
eventually.


code:
Got a bit distracted improving Haskell's directory listing code.

Only real git-annex work today was fixing [[bugs/Assistant_merge_loop]],
which was caused by changes in the last release (that made direct mode
merging crash/interrupt-safe). This is a kind of ugly bug, that can result
in the assistant making lots of empty commits in direct mode repositories.
So, I plan to make a new release on Monday.


code:
Got the release out. Had to fix various autobuilder issues. The arm autobuilder
is unfortunatly not working currently.

Updated git-annex to build with a new version of the bloomfilter library.


code:
**Important** A bug [[caused the assistant to sometimes remove all files|bugs/bad_merge_commit_deleting_all_files]] from the git repository.
You should check if your repository is ok. If the bug hit you, it should be
possible to revert the bad commit and recover your files with no data loss.
See the bug report for details.

This affected git-annex versions since 5.20140613, and only when using the
assistant in direct mode. It should be fixed in today's release,
5.20140709.

I'm available <urgent2014@joeyh.name> to help anyone hit by this
unfortunate bug.

This is another bug in the direct mode merge code. I'm not happy about it.
It's particularly annoying that I can't fix up after it automatically
(because there's no way to know if any given commit in the git history that
deletes all the files is the result of this bug, or a legitimate deletion
of all files).

The only good thing is that the design of git-annex is pretty robust, and
in this case, despite stupidly committing the deletion of all the files in
the repository, git-annex did take care to preserve all their contents and
so the problem should be able to be resolved without data loss.

Unfortunately, the main autobuilder is down and I've had to spin up
autobuilders on a different machine (thank goodness that's very automated
now!), and so I have not been able to build the fixed git-annex for android
yet. I hope to get that done later this evening.

---

Yesterday, I fixed a few (much less bad) bugs, and did some thinking about
plans for this month. The [[design/roadmap]] suggests working on some of
[[!traillink design/assistant/chunks]], [[!traillink design/assistant/deltas]] or [[!traillink design/assistant/gpgkeys]].
I don't know how to do deltas yet really. Chunks is pretty easily done.
The gpg keys stuff is pretty open ended and needs some more work to define
some use cases. But, after today, I am more inclined to want to spend
time on better testing and other means of avoiding this kind of situation.


code:
Spent past 2 days catching up on backlog and doing bug triage and some
minor bug fixes and features. Backlog is 27, lowest in quite a while so I
feel well on top of things.

I was saddened to find [this bug](http://git-annex.branchable.com/bugs/files_lost_during_upgrade/#comment-b265c796b1599d2dde649699cb54fa86)
where I almost managed to analize the [[ugy_bug|day_193-194__ugly_bug]]'s race
condition, but not quite (and then went on vacation). BTW, I have not heard
from anyone else who was hit by that bug so far.

The linux autobuilders are still down; their host server had a disk crash
in an electrical outage. Might be down for a while. I would not mind
setting up a redundant autobuilder if anyone else would like to donate a
linux VM with 4+ gb of ram.


code:
Yay, the Linux autobuilder is back! Also fixed the Windows build.

Fixed a reversion that prevented the webapp from starting
properly on Windows, which was introduced by some bad locking when I put in
the hack that makes it log to the log file on that platform.

Various other minor fixes here and there. There are almost enough
to do a release again soon.

I've also been trying to bootstrap ghc 7.8 on arm, for Debian. There's a script
that's supposed to allow building 7.8 using 7.6.3, dealing with a linker
problem by using the gold linker. Hopefully that will work since otherwise
Debian could remain stuck with an old ghc or worse lose the arm ports.
Neither would be great for git-annex..


code:
I have mostly been thinking about gcrypt today.
[This issue](https://github.com/blake2-ppc/git-remote-gcrypt/issues/9)
needs to be dealt with. The question is, does it really make sense to
try to hide the people a git repository is encrypted for? I have
[posted some thoughts](http://git-annex.branchable.com/bugs/using_gpg_encryption_with_multiple_keys_fails/?updated#comment-0c4f679d972c63b0b25b6aa5e851af62)
and am coming to the viewpoint that obscuring the identities of users
of a repository is not a problem git-annex should try to solve itself,
although it also shouldn't get in the way of someone who is able and
wants to do that (by using tor, etc).

Finally, I decided to go ahead and add a gcrypt.publish-participants
setting to git-remote-gcrypt, and make git-annex set that by default when
setting up a gcrypt repository.

Some promising news from the ghc build on arm. I got a working ghc, and
even ghci works. Which would make the template haskell in the webapp etc
avaialble on arm without the current horrible hacks. Have not managed to
build the debian ghc package successfully yet though.

Also, fixed a bug that made `git annex sync` not pull/push with a local
repository that had not yet been initialized for use with git-annex.

Today's work was sponsored by Stanley Yamane.


code:
[[!comment format=mdwn
 username="lex"
 ip="2620:149:7:303:796f:ae98:a0a1:94b"
 subject="comment 1"
 date="2014-07-16T21:13:06Z"
 content="""
glad my bug inspired a blog post about it.  =]
"""]]


code:
Spent hours today in a 10-minute build/test cycle, tracking down a bug that
caused the assistant to crash on Windows after exactly 10 minutes uptime.
Eventually found the cause; this is fallout from last month's work
that got it logging to the debug.log on Windows.

There was more, but that was the interesting one..


code:
Finished moving the Android autobuilder over to the new clean build
environment. Tested the Android app, and it still works. Whew!

There's a small chance that the issue with the Android app not working on
Android 4.3 has been fixed by this rebuild. I doubt it, but perhaps someone
can download the daily build and give it another try..

----

I have 7 days left in which I'd like to get remote gcrypt repositories
working in the assistant. I think that should be fairly easy, but a
prerequisite for it is making git-annex-shell support being run on a gcrypt
repository. That's needed so that the assistant's normal locked down ssh
key setup can also be used for gcrypt repositories.

At the same time, not all gcrypt endpoints will have git-annex-shell
installed, and it *seems* to make sense to leave in the existing support
for running raw rsync and git push commands against such a repository. So
that's going to add some complication.

It will also complicate git-annex-shell to support gcrypt repos. Basically,
everything it does in git-annex repos will need to be reimplemented in
gcrypt repositories. Generally in a more simple form; for example it
doesn't need to (and can't) update location logs in a gcrypt repo.

----

I also need to find a good UI to present the three available choices
(unencrypted git, encrypted git, encrypted rsync) when setting up a repo
on a ssh server. I don't want to just remove the encrypted rsync option,
because it's useful when using xmpp to sync the git repo, and is simpler to
set up since it uses shared encryption rather than gpg public keys.

My current thought is to offer just 2 choices, encrypted and non-encrypted.
If they choose encrypted, offer a choice of shared encryption or encrypting
to a specific key. I think I can word this so it's pretty clear what the
tradeoffs are.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmW0kg4uiMIhSHeVuvJFyo2VYMl7Qoej0s"
 nickname="Chris"
 subject="comment 1"
 date="2013-09-23T20:58:45Z"
 content="""
The new version of the Android apk doesn't work for me on my Nexus 4.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8"
 nickname="develop"
 subject="comment 2"
 date="2013-09-24T07:11:31Z"
 content="""
Yeah, no joy on Cyanogenmod 10.2(Android 4.3).

Would be pretty surprising if it had worked.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnR6E5iUghMWdUGlbA9CCs8DKaoigMjJXw"
 nickname="Efraim"
 subject="not working on my nexus 4 either"
 date="2013-09-24T07:37:28Z"
 content="""
terminal output reads:

Falling back to hardcoded app location; cannot find expected files in /data/app-lib

git annex webapp

u0_a124@mako:/sdcard/git-annex.home $ git annex webpp

CANNOT LINK EXECUTABLE: git-annex invalid R_ARM_COPY relocation against DT_SYMBOLIC shared library libc.so (built with -Bsymbolic?)

1|u0_a124@mako:/sdcard/git-annex.home $
"""]]


code:
I try hard to keep this devblog about git-annex development and not me.
However, it is a shame that what I wanted to be the beginning of my first
real month of work funded by the new campaign has been marred by my home's
internet connection being taken out by a lightning strike, and by illness.
Nearly back on my feet after that, and waiting for my new laptop to
finally get here.

Today's work: Finished up the `git annex forget` feature and merged it in.
Fixed the bug that was causing the commit race detection code to
incorrectly fire on the commit made by the transition code. Few other bits
and pieces.


code:
[[!comment format=mdwn
 username="rjc"
 ip="86.22.66.200"
 subject="laptop"
 date="2013-09-04T21:42:52Z"
 content="""
Are you retiring your Dell mini?

What kind of laptop are you getting?
"""]]


code:
Updated the Debian backport. (Also the git-remote-gcrypt backport.)

Made the assistant install a desktop file to integrate with Konqueror.

Improved `git annex repair`, fixing a bug that could cause it to leave
broken branch refs and yet think that the repair was successful.

----

A bit surprised to see that now been a full year since I started doing
development funded by my campaign. Not done yet!

Update on campaign rewards: <https://campaign.joeyh.name/blog/stickers_soon/>

----

Today's work was sponsored by Douglas Butts.


code:
Working on designs for better [[chunking|design/assistant/chunks]]. Having
a hard time finding a way to totally obscure file sizes, but otherwise a
good design seems to be coming together. I particularly like that the new
design puts the chunk count in the Key (which is then encrypted for special
remotes, rather than having it be some special extension.

While thinking through chunking, I realized that the current chunking
method can fail if two repositories have different chunksize settings for
the same special remote and both upload the same key at the same time.
Arn't races fun? The new design will eliminate this problem; in the
meantime updated the docs to recommend never changing a remote's
chunksize setting.


code:
The design for new style chunks seems done, and I laid the groundwork for it
today. Added chunk metadata to keys, reorganized the legacy chunking code
for directory and webdav so it won't get (too badly) in the way, and
implemented the chunk logs in the git-annex branch.

Today's work was sponsored by LeastAuthority.com.


code:
A lil bit in the weeds on the chunking rewrite right now. I did succeed in
writing the core chunk generation code, which can be used for every special
remote. It was pretty hairy (needs to stream large files in constant
memory, separating into chunks, and get the progress display right
across operations on chunks, etc). That took most of the day.

Ended up getting stuck in integrating the encryptable remote code, and had
to revert changes that could have led to rewriting (or perhaps
eliminating?) most of the per-remote encryption specific code.

Up till now, this has supported both encrypted and non-encrypted remotes;
it was simply passed encrypted keys for an encrypted remote:

[[!format haskell """
remove :: Key -> Annex Bool
"""]]

But with chunked encrypted keys, it seems it needs to be more complicated:

[[!format haskell """
remove' :: Maybe (Key -> Key) -> ChunkConfig -> Key -> Annex Bool
"""]]

So that when the remote is configured to use chunking, it can look up
the chunk keys, and then encrypt them, in order to remove all the encrypted
chunk keys.

I don't like that complication, so want to find a cleaner
abstraction. Will sleep on it.

----

While I was looking at the encryptable remote generator, I realized
the remote cost was being calculated wrongly for special
remotes that are not encrypted. Fixed that bug.

----

Today's work was sponsored by bak.


code:
Remained frustratingly stuck until 3 pm on the same stuff that puzzled
me yesterday. However, 6 hours later, I have the directory
special remote 100% working with both new chunk= and legacy chunksize=
configuration, both with and without encryption.

----

So, the root of why this is was hard, since I thought about it a lot today
in between beating my head into the wall: git-annex's internal API for remotes
is really, really simple. It basically comes down to:

[[!format haskell """
	Remote
		{ storeKey :: Key -> AssociatedFile -> MeterUpdate -> Annex Bool
		, retrieveKeyFile :: Key -> AssociatedFile -> FilePath -> MeterUpdate -> Annex Bool
		, removeKey :: Key -> Annex Bool
		, hasKey :: Key -> Annex (Either String Bool)
		}
"""]]

This simplicity is a Good Thing, because it maps very well to REST-type
services. And it allows for quite a lot of variety in implementations of
remotes. Ranging from reguar git remotes, that rsync files around without
git-annex ever loading them itself, to remotes like webdav that load
and store files themselves, to remotes like tahoe that intentionally do not
support git-annex's built-in encryption methods.

However, the simplicity of that API means that lots of complicated stuff,
like handling chunking, encryption, etc, has to be handled on a per-remote
basis. Or, more generally, by `Remote -> Remote` transformers that take
a remote and add some useful feature to it.

One problem is that the API is so simple that a remote transformer that adds
encryption is not feasible. In fact, every encryptable remote has
had its own code that loads a file from local disk, encrypts it, and sends
it to the remote. Because there's no way to make a remote transformer that
converts a `storeKey` into an encrypted `storeKey`. (Ditto for retrieving
keys.)

I almost made the API more complicated today. Twice. But both times
I ended up not, and I think that was the right choice, even though
it meant I had to write some quite painful code.

----

In the end, I instead wrote a little module that pulls together supporting
both encryption and chunking. I'm not completely happy because those
two things should be independent, and so separate. But, 120 lines of
code that don't keep them separate is not the end of the world.

That module also contains some more powerful, less general APIs,
that will work well with the kinds of remotes that will use it.

The really nice result, is that the implementation of the directory
special remote melts down from 267 lines of code to just 172! (Plus some
legacy code for the old style chunking, refactored out into a file I can
delete one day.) It's a lot cleaner too.

With all this done, I expect I can pretty easily add the new style chunking
to most git-annex remotes, and remove code from them while doing it!

----

Today's work was sponsored by Mark Hepburn.


code:
Last night, went over the new chunking interface, tightened up exception
handling, and improved the API so that things like WebDAV will be able to
reuse a single connection while all of a key's chunks are being downloaded.
I am pretty happy with the interface now, and except to convert more
special remotes to use it soon.

Just finished adding a killer feature: Automatic resuming of interrupted
downloads from chunked remotes. Sort of a poor man's rsync, that while less
efficient and awesome, is going to work on *every* remote that gets the new
chunking interface, from S3 to WebDAV, to all of Tobias's external special
remotes! Even allows for things like starting a download
from one remote, interrupting, and resuming from another one, and so on.

I had forgotten about resuming while designing the chunking API. Luckily, I
got the design right anyway. Implementation was almost trivial, and only
took about 2 hours! (See [[!commit 9d4a766cd7b8e8b0fc7cd27b08249e4161b5380a]])

I'll later add resuming of interrupted uploads. It's not hard to detect
such uploads with only one extra query of the remote, but in principle,
it should be possible to do it with no extra overhead, since git-annex
already checks if all the chunks are there before starting an upload.


code:
Zap! ... My internet gateway was [destroyed by lightning](https://identi.ca/joeyh/note/xogvXTFDR9CZaCPsmKZipA).
Limping along regardless, and replacement ordered.

Got resuming of uploads to chunked remotes working. Easy!

----

Next I want to convert the external special remotes to have these nice
new features. But there is a wrinkle: The new chunking interface works
entirely on ByteStrings containing the content, but the external special
remote interface passes content around in files.

I could just make it write the ByteString to a temp file, and pass the temp
file to the external special remote to store. But then, when chunking is
not being used, it would pointlessly read a file's content, only to write
it back out to a temp file.

Similarly, when retrieving a key, the external special remote saves it to a
file. But we want a ByteString. Except, when not doing chunking or
encryption, letting the external special remote save the content directly
to a file is optimal.

One approach would be to change the protocol for external special
remotes, so that the content is sent over the protocol rather than in temp
files. But I think this would not be ideal for some kinds of external
special remotes, and it would probably be quite a lot slower and more
complicated.

Instead, I am playing around with some type class trickery:

[[!format haskell """
{-# LANGUAGE Rank2Types TypeSynonymInstances FlexibleInstances MultiParamTypeClasses #-}

type Storer p = Key -> p -> MeterUpdate -> IO Bool

-- For Storers that want to be provided with a file to store.
type FileStorer a = Storer (ContentPipe a FilePath)

-- For Storers that want to be provided with a ByteString to store
type ByteStringStorer a = Storer (ContentPipe a L.ByteString)

class ContentPipe src dest where
        contentPipe :: src -> (dest -> IO a) -> IO a

instance ContentPipe L.ByteString L.ByteString where
        contentPipe b a = a b

-- This feels a lot like I could perhaps use pipes or conduit...
instance ContentPipe FilePath FilePath where
        contentPipe f a = a f

instance ContentPipe L.ByteString FilePath where
        contentPipe b a = withTmpFile "tmpXXXXXX" $ \f h -> do
                L.hPut h b
                hClose h
                a f

instance ContentPipe FilePath L.ByteString where
        contentPipe f a = a =<< L.readFile f
"""]]

The external special remote would be a FileStorer, so when a non-chunked,
non-encrypted file is provided, it just runs on the FilePath with no extra
work. While when a ByteString is provided, it's swapped out to a temp file
and the temp file provided. And many other special remotes are ByteStorers,
so they will just pass the provided ByteStream through, or read in the
content of a file.

I think that would work. Thoigh it is not optimal for external special
remotes that are chunked but not encrypted. For that case, it might be worth
extending the special remote protocol with a way to say "store a chunk of
this file from byte N to byte M".

---

Also, talked with ion about what would be involved in using rolling checksum
based chunks. That would allow for rsync or zsync like behavior, where
when a file changed, git-annex uploads only the chunks that changed, and the
unchanged chunks are reused.

I am not ready to work on that yet, but I made some changes to the parsing
of the chunk log, so that additional chunking schemes like this can be added
to git-annex later without breaking backwards compatability.


code:
It took 9 hours, but I finally got to make [[!commit c0dc134cded6078bb2e5fa2d4420b9cc09a292f7]],
which both removes 35 lines of code, and adds chunking support to all
external special remotes!

The groundwork for that commit involved taking the type scheme I sketched
out yesterday, completely failing to make it work with such high-ranked
types, and falling back to a simpler set of types that both I and GHC seem
better at getting our heads around.

Then I also had more fun with types, when it turned out I needed to
run encryption in the Annex monad. So I had to go convert several parts of
the utility libraries to use MonadIO and exception lifting. Yurk.

The final and most fun stumbling block caused git-annex to crash when
retriving a file from an external special remote that had neither
encryption not chunking. Amusingly it was because I had not put in an
optimation (namely, just renaming the file that was retrieved in this case,
rather than unnecessarily reading it in and writing it back out). It's
not often that a lack of an optimisation causes code to crash!

So, fun day, great result, and it should now be very simple to convert
the bup, ddar, gcrypt, glacier, rsync, S3, and WebDAV special remotes
to the new system. Fingers crossed.

But first, I will probably take half a day or so and write a
`git annex testremote` that can be run in a repository and does live
testing of a special remote including uploading and downloading files.
There are quite a lot of cases to test now, and it seems best to get
that in place before I start changing a lot of remotes without a way to
test everything.

----

Today's work was sponsored by Daniel Callahan.


code:
Built `git annex testremote` today.

That took a little bit longer than expected, because it actually found
several fence post bugs in the chunking code.

It also found a bug in the sample external special remote script.

I am very pleased with this command. Being able to run 640 tests against
any remote, without any possibility of damaging data already stored in the
remote, is awesome. Should have written it a looong time ago!


code:
Added support for gcrypt remotes to git-annex-shell. Now gcrypt special
remotes probe when they are set up to see if the remote system has a
suitable git-annex-shell, and if so all commands are sent to it. Kept the
direct rsync mode working as a fallback.

It turns out I made a bad decision when first adding gcrypt support to
git-annex. To make implementation marginally easier, I decided to not
put objects inside the usual `annex/objects` directory in a gcrypt remote.
But that lack of consistency would have made adding support to
git-annex-shell a lot harder. So, I decided to change this. Which
means that anyone already using gcrypt with git-annex will need to
[[manually_move_files_around|upgrades/gcrypt]].

Today's work was sponsored by Tobias Nix.


code:
Did various bug fixes and followup today. Amazing how a day can vanish that
way. Made 4 actual improvements.

I still have 46 messages in unanswered backlog. Although only 8 of
the are from this month.


code:
Being still a little unsure of the UI and complexity
for configuring gcrypt on ssh servers, I thought I'd start today with the
special case of gcrypt on rsync.net. Since rsync.net allows running some git
commands, gcrypt can be used to make encrypted git repositories on it.

Here's the UI I came up with. It's complicated a bit by needing to explain
the tradeoffs between the rsync and gcrypt special remotes.

[[!img /assistant/rsync.net.encryption.png]]

This works fine, but I did not get a chance to add support for enabling
existing gcrypt repos on rsync.net. Anyway, most of the changes to make
this work will also make it easier to add general support for gcrypt on ssh
servers.

Also spent a while fixing a bug in git-remote-gcrypt. Oddly
`gpg --list-keys --fast-list --fingerprint` does not show the fingerprints
of some keys.

Today's work was sponsored by Cloudier - Thomas Djärv.


code:
Worked on making the assistant able to merge in existing encrypted
git repositories from rsync.net.

This had two parts. First, making the webapp UI where you click to enable a
known special remote work with these encrypted repos. Secondly, handling
the case where a user knows they have an encrypted repository on rsync.net,
so enters in its hostname and path, but git-annex doesn't know about that
special remote. The second case is important, for example, when the
encrypted repository is a backup and you're restoring from it. It wouldn't
do for the assistant, in that case, to make a *new* encrypted repo and
push it over top of your backup!

Handling that was a neat trick. It has to do quite a lot of probing, including
downloading the whole encrypted git repo so it can decrypt it and merge it,
to find out about the special remote configuration used for it. This all
works with just 2 ssh connections, and only 1 ssh password prompt max.

Next, on to generalizing this rsync.net specific code to work with
arbitrary ssh servers!

----

Today's work was made possible by [RMS's vision 30 years ago](http://article.olduse.net/771@mit-eddie.UUCP).


code:
So close to being done with gcrypt support.. But still not quite there.

Today I made the UI changes to support gcrypt when setting up a repository
on a ssh server, and improved the probing and data types so it can tell
which options the server supports. Fairly happy with how that is turning
out.

Have not yet hooked up the new buttons to make gcrypt repos. While I was
testing that my changes didn't break other stuff, I found a bug in the
webapp that caused it to sometimes fail to transfer one file to/from a
remote that was just added, because the transferrer process didn't know
about the new remote yet, and crashed (and was restarted knowing about it,
so successfully sent any other files). So got sidetracked on fixing that.

Also did some work to make the gpg bundled with git-annex on OSX be
compatable with the config files written by MacGPG. At first I was going to
hack it to not crash on the options it didn't support, but it turned out
that upgrading to version 1.4.14 actually fixed the problem that was making
it build without support for DNS.

----

Today's work was sponsored by Thomas Hochstein.


code:
Long day, but I did finally finish up with gcrypt support. More or less.

Got both creating and enabling existing gcrypt repositories on ssh servers
working in the webapp. (But I ran out of time to make it detect when the
user is manually entering a gcrypt repo that already exists. Should be easy
so maybe tomorrow.)

Fixed several bugs in git-annex's gcrypt support that turned up in testing.
Made git-annex ensure that a gcrypt repository does not have
receive.denyNonFastForwards set, because gcrypt relies on always forcing
the push of the branch it stores its manifest on. Fixed a bug in
`git-annex-shell recvkey` when it was receiving a file from an annex in
direct mode.

Also had to add a new `git annex shell gcryptsetup` command, which is
needed to make setting up a gcrypt repository work when the assistant
has set up a locked-down ssh key that can only run git-annex-shell. Painted
myself into a bit of a corner there.

And tested, tested, tested. So many possibilities and edge cases in this
part of the code..

----

Today's work was sponsored by Hendrik Müller Hofstede.


code:
Did I say it would be easy to make the webapp detect when a gcrypt repository
already existed and enable it? Well, it wasn't exactly hard, but it took
over 300 lines of code and 3 hours..

So, gcrypt support is done for now. The glaring omission is gpg key
management for sharing gcrypt repositories between machines and/or people.
But despite that, I think it's solid, and easy to use, and covers some
great use cases.

Pushed out a release.

Now I really need to start thinking about
[[design/assistant/disaster_recovery]].

----

Today's work was sponsored by Dominik Wagenknecht.


code:
Started the day by getting the builds updated for yesterday's release. This
included making it possible to build git-annex with Debian stable's version
of cryptohash. Also updated the Debian stable backport to the previous
release.

----

The [[design/roadmap]] has this month devoted to improving git-annex's
support for recovering from disasters, broken repos, and so on. Today I've
been working on the first thing on [[the_list|design/assistant/disaster_recovery]],
stale git index lock files.

It's unfortunate that git uses simple files for locking, and does not use
fcntl or flock to prevent the stale lock file problem. Perhaps they want
it to work on broken NFS systems? The problem with that line of thinking is
is means all non-broken systems end up broken by stale lock files. Not a
good tradeoff IMHO.

There are actually two lock files that can end up stale when using
git-annex; both `.git/index.lock` and `.git/annex/index.lock`. Today I
concentrated on the latter, because I saw a way to prevent it from ever
being a problem. All updates to that index file are done by git-annex when
committing to the git-annex branch. git-annex already uses fcntl locking
when manipulating its journal. So, that can be extended to also cover
committing to the git-annex branch, and then the git `index.lock` file
is irrelevant, and can just be removed if it exists when a commit is
started.

To ensure this makes sense, I used the type system to prove that the journal
locking was in effect everywhere I need it to be. Very happy I was able to
do that, although I am very much a novice at using the type system for
interesting proofs. But doing so made it very easily to build up to a point
where I could unlink the `.git/annex/index.lock` and be sure it was safe to do
that.

----

What about stale `.git/index.lock` files? I don't think it's appropriate
for git-annex to generally recover from those, because it would change
regular git command line behavior, and risks breaking something. However, I
do want the assistant to be able to recover if such a file exists when it
is starting up, since that would prevent it from running. Implemented that
also today, although I am less happy with the way the assistant detects
when this lock file is stale, which is somewhat heuristic (but should work
even on networked filesystems with multiple writing machines).

----

Today's work was sponsored by Torbjørn Thorsen.


code:
[[!comment format=mdwn
 username="http://cstork.org/"
 nickname="Chris Stork"
 subject="News page not updated"
 date="2013-10-04T09:38:21Z"
 content="""
Already for the last three or so releases the News page wasn't updated. :-( I think many people (including me) used this page to check if their version is uptodate. Posting the newest version there motivates people to try the very latest which seems very desirable for the git-annex development.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.243"
 subject="comment 2"
 date="2013-10-04T20:13:37Z"
 content="""
Sorry about that. It turns out my release script was broken.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA"
 nickname="Laszlo"
 subject="comment 3"
 date="2013-10-07T09:17:30Z"
 content="""
Hi,

Can you please enlighten us mere mortals, when this stale file locking problem occcurs?
The typical situation. I tried to google up, and read locking files on wikipedia, but
still have no exact knowledge when the \"stale\" part happens.

How it gets outdated without noticing?

Best,
 Laszlo

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.22"
 subject="comment 4"
 date="2013-10-12T23:31:16Z"
 content="""
Git simply creates a file as a lock file, and does not use any form of locking on it, so if the git process dies for any reason before it gets a chance to remove the lock file, a stale lock file remains, and future git commands will fall over it.

It's really surprisingly bad..
"""]]


code:
Finished up the automatic recovery from stale lock files. Turns out git
has quite a few lock files; the assistant handles them all.

Improved URL and WORM keys so the filenames used for them
will always work on FAT (which has a crazy assortmeny of illegal
characters). This is a tricky thing to deal with without breaking backwards
compatability, so it's only dealt with when creating new URL or WORM keys.

-----

I think my next step in this disaster recovery themed month will be adding
periodic incremental fsck to the assistant. `git annex fsck` can already
do an incremental fsck, so this should mostly involve adding a user
interface to the webapp to configure when it should fsck. For example, you
might choose to run it for up 1 hour every night, with a goal of checking
all your files once per month. Also will need to make the assistant do
something useful when fsck finds a bad file (ie, queue a re-download).


code:
Spent most of the day building some generic types for scheduling recurring
events. Not sure if rolling my own was a good idea, but that's what I did.

In the incrementalfsck branch, I have hooked this up in `git-annex vicfg`,
which now accepts and parses scheduled events like
"fsck self every day at any time for 60 minutes" and
"fsck self on day 1 of weeks divisible by 2 at 3:45 for 120 minutes", and
stores them in the git-annex branch. The exact syntax is of course subject
to change, but also doesn't matter a whole lot since the webapp will have
a better interface.


code:
Now I can build git-annex twice as fast! And a typical incremental build is
down to 10 seconds, from 51 seconds.

Spent a productive evening working with Guilhem to get his encryption
patches reviewed and merged. Now there is a way to remove revoked gpg keys,
and there is a new encryption scheme available that uses public key
encryption by default rather than git-annex's usual approach. That's not
for everyone, but it is a good option to have available.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA"
 nickname="Laszlo"
 subject="comment 1"
 date="2013-09-05T15:07:44Z"
 content="""
out of curiosity, what laptop model have you choosen finally?

Laszlo
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="24.159.78.54"
 subject="comment 2"
 date="2013-09-06T18:02:08Z"
 content="""
Lenovo Yoga 11s
"""]]


code:
Lots of progress from yesterday's modest start of building data types for
scheduling. Last night I wrote the hairy calendar code to calculate when
next to run a scheduled event. (This is actually quite superior to `cron`,
which checks every second to see if it should run each event!) Today I
built a "Cronner" thread that handles spawning threads to handle each
scheduled event. It even notices when changes have been made to the its
schedule and stops/starts event threads appropriately.

Everything is hooked up, building, and there's a good chance it works
without too many bugs, but while I've tested all the pure code (mostly
automatically with quickcheck properties), I have not run the Cronner
thread at all. And there is some tricky stuff in there, like noticing
that the machine was asleep past when it expected to wake up, and deciding
if it should still run a scheduled event, or should wait until next time.
So tomorrow we'll see..

Today's work was sponsored by Ethan Aubin.


code:
[[!comment format=mdwn
 username="http://nullroute.eu.org/~grawity/"
 nickname="Mantas"
 subject="comment 1"
 date="2013-10-09T06:04:15Z"
 content="""
The SysV Unix *cron* learned this same thing [in 1979][1].

[1]: http://en.wikipedia.org/wiki/Cron#Multi-user_capability
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.22"
 subject="comment 2"
 date="2013-10-11T17:07:24Z"
 content="""
However, if you strace a traditional cron, you will be sad at how it's implemented. So much statting of files, and checking of entries every second..

0 polling or redundant calulcation in my code!
"""]]


code:
Some neat stuff is coming up, but today was a pretty blah day for me.
I did get the Cronner tested and working (only had a few little bugs). But
I got stuck for quite a while making the Cronner stop `git-annex fsck`
processes it was running when their jobs get removed. I had some code to do
this that worked when run standalone, but not when run from git-annex.

After considerable head-scratching, I found out this was due to
`forkProcess` masking aync exceptions, which seems to be probably
[a bug](http://ghc.haskell.org/trac/ghc/ticket/8433). Luckily was able to
work around it. Async exceptions continue to strike me as the worst part of
the worst part of Haskell (the worst part being exceptions in general).

Was more productive after that.. Got the assistant to automatically queue
re-downloads of any files that fsck throws out due to having bad contents,
and made the webapp display an alert while fscking is running, which will
go to the page to configure fsck schedules. Now all I need to do is
build the UI of that page.


code:
Last night, built this nice user interface for configuring periodic fscks:

[[!img assistant/fsckconfig.png]]

Rather happy that that whole UI needed only 140 lines of code to build.
Though rather more work behind it, as seen in this blog..

Today I added some support to git-annex for smart fscking of remotes.
So far only git repos on local drives, but this should get extended to
git-annex-shell for ssh remotes. The assistant can also run periodic fscks
of these.

Still need to test that, and find a way to make a removable drive's fsck
job run when the drive gets plugged in. That's where picking "any time"
will be useful; it'll let you configure fscking of removable drives when
they're available, as long as they have not been fscked too recently.

----

Today's work was sponsored by Georg Bauer.


code:
Built everything needed to run a fsck when a remote gets connected. Have
not tested it; only testing is blocking merging the incrementalfsck branch
now.

Also updated the OSX and Android builds to use a new gpg release (denial of
service security fix), and updated the Debian backport, and did a small
amount of bug fixing. I need to do several more days of bug fixing once
I get this incremental fsck feature wrapped up before moving on to recovery
of corrupt git repositories.


code:
Fixed a lot of bugs in the assistant's fsck handling today, and merged
it into master. There are some enhancments that could be added to it,
including fscking ssh remotes via git-annex-shell and adding the ability to
schedule events to run every 30 days instead of on a specific day of the
month. But enough on this feature for now.

Today's work was sponsored by Daniel Brockman.


code:
While I said I was done with fsck scheduling yesterday, I ended up adding
one more feature to it today: Full anacron style scheduling. So a fsck can
be scheduled to run once per week, or month, or year, and it'll run the
fsck the next time it's available after that much time has passed. The nice
thing about this is I didn't have to change Cronner *at all* to add this,
just improved the Recurrance data type and the code that calculates when
to run events.

Rest of the day I've been catching up on some bug reports. The main bug I
fixed caused git-annex on Android to hang when adding files. This turns out
to be because it's using a new (unreleased) version of git, and
`git check-attr -z` output format has changed in an incompatable way.

I am currently 70 messages behind, which includes some ugly looking bug
reports, so I will probably continue with this over the next couple days.


code: Productive day, but I'm wiped out. Backlog down to 51.


code:
A long day of bugfixing. Split into two major parts. First I got back to a
bug I filed in August to do with the assistant misbehaving when run in a
subdirectory of a git repository, and did a nice type-driven fix of the
underlying problem (that also found and fixed some other related bugs that
would not normally occur). Then, spent 4 hours in Windows purgatory working
around crazy path separator issues.


code:
Goal for the rest of the month is to build automatic recovery git
repository corruption. Spent today investigating how to do it and came up
with a fairly [[detailed_design|design/assistant/disaster_recovery]]. It
will have two parts, first to handle repository problems that can be fixed
by fetching objects from remotes, and secondly to recover from problems
where data never got sent to a remote, and has been lost.

In either case, the assistant should be able to detect the problem and
automatically recover well enough to keep running. Since this also affects
non-git-annex repositories, it will also be available in a standalone
`git-recover-repository` command.


code:
[[!comment format=mdwn
 username="http://cstork.org/"
 nickname="Chris Stork"
 subject="Idea: checksum left-over files in .git/annex/tmp for potential recovery"
 date="2013-10-19T12:30:58Z"
 content="""
This just came to mind when I thought about the second case.  I noticed quite often that files were left in .git/annex/tmp (seen on OS X) and sometimes these are the only instances of files on this particular computer.

(It's on my todo list to file several bug reports about this and other issues on OS X...)
"""]]