code:
I've been running some large transfers with the assistant, and looking at
ways to improve performance. (I also found and fixed a zombie process
leak.)

----

One thing I noticed is that the assistant pushes changes to the git-annex
location log quite frequently during a batch transfer. If the files being
transferred are reasonably sized, it'll be pushing once per file transfer.
It would be good to reduce the number of pushes, but the pushes are
important in some network topologies to inform other nodes
when a file gets near to them, so they can get the file too.

Need to see if I can find a smart way to avoid some of the pushes.
For example, if we've just downloaded a file, and are queuing uploads
of the file to a remote, we probably don't need to push the git-annex
branch to the remote.

----

Another performance problem is that having the webapp open while transfers
are running uses significant CPU just for the browser to update the progress
bar. Unsurprising, since the webapp is sending the browser a new `<div>`
each time. Updating the DOM instead from javascript would avoid that;
the webapp just needs to send the javascript either a full `<div>` or a
changed percentage and quantity complete to update a single progress bar.

I'd prefer to wait on doing that until I'm able to use Fay to generate
Javascript from Haskell, because it would be much more pleasant.. will see.

----

Also a performance problem when performing lots of transfers, particularly
of small files, is that the assistant forks off a `git annex transferkey`
for each transfer, and that has to in turn start up several git commands.

Today I have been working to change that, so the assistant maintains a
pool of transfer processes, and dispatches each transfer it wants to make
to a process from the pool. I just got all that to build, although untested
so far, in the `transferpools` branch.


code:
Was unsure yesterday if my transferrer pools code would just work, or
would be horribly broken and need a lot of work to get going. It was a
complex change involving both high-level STM code and low-level pipes and
fds. Well, it almost worked 100% first time, I just had a minor issue in
my fd setup to fix. Everything else seems to work perfectly.
Very happy how that went!

----

Improved support and documentation for using the OSX app and Linux
standalone tarball at the command line. Now it's sufficient to just put
their directory into `PATH`, rather than using `runshell`.

----

The webapp's form for adding a removable drive now allows specifying the
directory to use within the drive (default "annex").

When the drive's repository already exists, and it's not a repository
that git-annex knows about, it confirms that the user wants
to combine its contents into their repository.

(Should probably implement this same check when adding a ssh remote.)

----

Off to Boston!

repo_name: kakafarm/git

license: unknown


code:
Back from my trip. Spent today getting caught up.

Didn't do much while I was away. Pushed out a new release on Saturday.
Made `git annex` usage display nicer.

Fixed some minor webapp bugs today. The interesting bug was a
race that sometimes caused alerts or other notifications to be
missed and not be immediately displayed if they occurred while
a page was loading. You sometimes had to hit reload to see them,
but not anymore!

Checked if the `push.default=simple` change in upcoming git release will
affect git-annex. It shouldn't affect the assistant, or `git annex sync`,
since they always list all branches to push explicitly. But if you `git push`
manually, when the default changes that won't include the git-annex branch
in the push any longer.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm7AuSfii_tCkLyspL6Mr0ATlO6OxLNYOo"
 nickname="Georg"
 subject="something for the wishlist maybe?"
 date="2013-03-28T08:30:00Z"
 content="""
Something I was wondering with the assistant now getting really smart: would it be possible to have a cli-mode for the assistant that runs in the forground and only until there is nothing to do anymore? Something like a single-run-assistant`? Reason being that in my usecases most often I know exactly when the right moment for a run is - for example after processing a bunch of photographs - and I most definitely don't want a background process to work on those photographs while I do work on them. Or with archival media, I know when I will hook up the archive drives. In allmost all of my scenarios it would be much easier to not run the assistant in the background, but to fire it from the cli at the right moments, letting it work over night and have it shut down automatically when all is synced.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-03-28T15:50:47Z"
 content="""
That's an interesting thought. I think it would be hard to find a clear condition when it was done and could shut down. While there is easy visibility into the transfer queue to tell when file transfers are done, it's harder to tell when it's actively syncing git repositories or scanning for transfers to do, or finding new files to commit. Each thread knows what it's doing, but none of the other threads do. Maybe active threads could hold a lock to indicate they're busy, and this combined with visibility into the various work queues could do the job.
"""]]


code:
Went out and tried for the second time to record a screencast demoing
setting up syncing between two computers using just Jabber and a cloud
remote. I can't record this one at home, or viewers would think git-annex
was crazy slow, when it's just my dialup. ;) But once again I encountered
bugs, and so I found myself working on progress bars today, unexpectedly.

Seems there was confusion in different parts of the progress bar code
about whether an update contained the total number of bytes transferred, or
the delta of bytes transferred since the last update. One way this bug
showed up was progress bars that seemed to stick at 0% for a long time.
Happened for most special remotes, although not for rsync or git remotes.
In order to fix it comprehensively, I added a new BytesProcessed data type,
that is explicitly a total quantity of bytes, not a delta. And checked and
fixed all the places that used a delta as that type was knitted into
the code.

(Note that this doesn't necessarily fix every problem with progress bars.
Particularly, buffering can now cause progress bars to seem to run ahead
of transfers, reaching 100% when data is still being uploaded. Still,
they should be a lot better than before.)

I've just successfully run through the Jabber + Cloud remote setup process
again, and it seems to be working great now. Maybe I'll still get the
screencast recorded by the end of March.


code:
Built a feature for power users today. `annex.largefiles` can be
configured to specify what files `git annex add` and the assistant should
put into the annex. It uses the same syntax as [[/preferred_content]],
so arbitrarily complex expressions can be built.

For example, a game written in C with some large data files could
include only 100kb or larger files, that are not C code:

	annex.largefiles = largerthan=100kb and not (include=*.c or include=*.h)

The assistant will commit small files to git directly!
`git annex add`, being a lower level tool, skips small files
and leaves it up to you to `git add` them as desired.

It's even possible to tell the assistant that no file is too large to be
committed directly to git. `git config annex.largefiles 'exclude=*'`
The result should be much like using SparkleShare or dvcs-autosync.

-----

Also today, made the remote ssh server checking code in the webapp
deal with servers where the default shell is csh or some other non-POSIX
shell.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm7AuSfii_tCkLyspL6Mr0ATlO6OxLNYOo"
 nickname="Georg"
 subject="Great idea! "
 date="2013-03-30T08:00:16Z"
 content="""
This will be perfect for my raw image pool! I use Lightroom, so my photos are only imported once and only in case of DNG are they actually changed, in all other formats LR creates little sidecar XML files with processing instructions. So with this change the side car files can actually be fully Versionen and lage files are kept out of the repo and handled by archiving with annex. Cool!
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmWBvsZvSsAL8P2ye3F0OBStjFCVnOImzM"
 nickname="Jarno"
 subject="Making "git-annex add" commit largefiles, too?"
 date="2013-10-31T00:26:21Z"
 content="""
Is it possible to make `git annex add` also *not* skip largefiles but do an automatic `git add` instead? I'd like to keep .txt, .sh and the like in git directly (for easier diffing) but having to add them separately sounds like trouble.
"""]]


code:
I've posted a poll: [[polls/goals_for_April]]

----

Today added UI to the webapp to delete repositories, which many
users have requested. It can delete the local repository,
with appropriate cautions and sanity checks:

[[!img /assistant/deleterepository.png]]

More likely, you'll use it to remove a remote, which is done with no muss
and no fuss, since that doesn't delete any data and the remote can always
be added back if you change your mind.

It also has an option to fully delete the data on a remote. This doesn't
actually delete the remote right away. All it does is marks the remote
as untrusted[1], and configures it to not want any content.
This causes all the content on it to be sucked off to whatever
other repositories can hold it.

I had to adjust the preferred content expressions to make that work. For
example, when deleting an archive drive, your local (client) repository
does not normally want to hold all the data it has in "archive"
directories. With the adjusted preferred content expressions, any data on
an untrusted or dead repository is wanted. An interesting result is that
once a client repository has moved content from an untrusted remote, it
will decide it doesn't want it anymore, and shove it out to any other
remote that will accept it. Which is just the behavior we want. All it took
to get all this behavior is adding "or (not copies=semitrusted:1)" to the
preferred content expressions!

For most special remotes, just sucking the data from them is sufficient to
pretty well delete them. You'd want to delete an Amazon bucket or glacier
once it's empty, and git repositories need to be fully deleted. Since this
would need unique code for each type of special remote, and it would be
code that a) deletes possibly large quantities of data with no real way to
sanity check it and b) doesn't get run and tested very often; it's not
something I'm thrilled about fully automating. However, I would like to
make the assistant detect when all the content has been sucked out of a
remote, and pop up at least a message prompting to finish the deletion.
Future work.

-----

[1] I really, really wanted to mark it dead, but letting puns drive code
is probably a bad idea. I had no idea I'd get here when I started
developing this feature this morning.. Honest!


code:
[[!comment format=mdwn
 username="andy"
 ip="99.65.214.16"
 subject="Just wondering what I'm missing..."
 date="2013-04-02T21:43:33Z"
 content="""
I haven't played around with the preferred content expressions, but what is is that keeps \"or (not copies=semitrusted:1)\" from grabbing all the content from trusted repositories, as well as untrusted or dead ones?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-04-03T03:42:40Z"
 content="""
That's some seriously sharp eyes, Andy. Nothing. I forgot it didn't work that way. Now it uses semitrusted+:1 which works the right way.
"""]]


code:
Both the assistant and `git annex drop --auto` refused to drop files from
untrusted repositories. Got that fixed.

Finally recorded the xmpp pairing screencast. In one perfect take, which
somehow `recordmydesktop` lost the last 3 minutes of.
Argh! Anyway I'm editing it now, so, look for that screencast soon.

The [[polls/goals_for_April]] poll results are in.

* There have been no votes at all for working on
  cloud remotes. Seems that git-annex supports enough cloud remotes already.
* A lot of people want the Android webapp port to be done, so I will
  probably spend some time on that this month.
* Interest in other various features is split. I am surprised how many
  want git-remote-gcrypt, compared to the features that would make
  syncing use less bandwidth. Doesn't git push over xmpp cover most
  of the use cases where git-remote-gcrypt would need to be used with the
  assistant?
* Nearly as many people as want features, want me to work on bug
  fixing and polishing what's already there.
  So I should probably continue to make screencasts, since they often force
  me to look at things with fresh eyes and see and fix problems. And of course,
  continue working on bugs as they're reported.
* I'm not sure what to make of the 10% who want me to add direct mode support.
  Since direct mode is already used by default, perhaps they want
  me to take time off? :) (I certainly need to fix the
  [[bugs/Direct_mode_keeps_re-checksuming_duplicated_files]] bug, and one other
  direct mode bug I discovered yesterday.)


code:
[[!comment format=mdwn
 username="https://dzsino.myopenid.com/"
 nickname="dzsino"
 subject="I've voted for git-remote-gcrypt because.."
 date="2013-04-02T09:07:23Z"
 content="""
... as important a Windows port seems for git-annex to really take off (IMO), I would welcome encrypted repos for storing stuff on external drives: directory special remotes feel kinda cumbersome for me, I would rather use full blown repos.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-04-02T21:36:50Z"
 content="""
Thanks for reminding me about that use case. I have created a page [[encrypted_git_remotes]] for this.
"""]]


code:
The [[xmpp screencast|videos/git-annex_assistant_remote_sharing]]
is at long last done!

----

Fixed a bug that could cause the assistant to unstage files
from git sometimes. This happened because of a bad optimisation; adding a
file when it's already present and unchanged was optimised to do nothing.
But if the file had just been removed, and was put back, this resulted
in the removal being staged, and the add not being staged. Ugly bug,
although the assistant's daily sanity check automatically restaged the
files.

Underlying that bug was a more important problem: git-annex does not always
update working tree files atomically. So a crash at just the wrong instant
could cause a file to be deleted from the working tree. I fixed that too;
all changes to files in the working tree should now be staged in a temp
file, which is renamed into place atomically.

Also made a bunch of improvements to the dashboard's transfer display, and
to the handling of the underlying transfer queue.


code:
Getting back to the repository removal handling from Sunday, I made the
assistant detect when a repository that has been marked as unwanted becomes
empty, and finish the removal process.

I was able to add this to the expensive transfer scan without making it any
more expensive than it already was, since that scan already looks at the
location of all keys. Although when a remote is detected as empty, it then
does one more check, equivilant to `git annex unused`, to find any
remaining objects on the remote, and force them off.

I think this should work pretty well, but it needs some testing and
probably some UI work.

----

Andy spotted a bug in the preferred content expressions I was using to
handle untrusted remotes. So he saved me several hours dealing with an ugly
bug at some point down the line. I had misread my own preferred content
expression documentation, and `copies=semitrusted:1` was not doing what I
thought it was. Added a new syntax that does what I need,
`copies=semitrusted+:1`

----

The 64 bit linux standalone builds are back. Apparently the 32 bit builds
have stopped working on recent Fedora, for reasons that are unclear. I set
up an autobuilder to produce the 64 bit builds.


code:
Got caught up on bug reports and made some bug fixes.

The one bug I was really worried about, a strange file corruption problem
on Android, turned out not to be a bug in git-annex. (Nor is it a bug that
will affect regular users.)

The only interesting bug fixed was a mixed case hash directory name
collision when a repository is put on a VFAT filesystem (or other
filesystem with similar semantics). I was able to fix that nicely; since
such a repository will be in crippled filesystem mode due to other
limitations of the filesystem, and so won't be using symlinks,
it doesn't need to use the mixed case hash directory names.

Last night, finished up the repository removal code, and associated UI
tweaks. It works very well.

Will probably make a release tomorrow.


code:
Well, sometimes you just have to go for the hack. Trying to find a way
to add additional options to git-annex-shell without breaking backwards
compatibility, I noticed that it ignores all options after `--`, because
those tend to be random rsync options due to the way rsync runs it.

So, I've added a new class of options, that come in between, like
`-- opt=val opt=val ... --`

The parser for these will not choke on unknown options, unlike normal
getopt. So this let me add the additional info I needed to
pass to git-annex-shell to make it record transfer information. And
if I need to pass more info in the future, that's covered too.

It's ugly, but since only git-annex runs git-annex-shell, this is an
ugliness only I (and now you, dear reader) have to put up with.

Note to self: Command-line programs are sometimes an API, particularly
if designed to be called remotely, and so it makes sense consider
whether they are, and design expandability into them from day 1.

---

Anyway, we now have full transfer tracking in git-annex! Both sides of
a transfer know what's being transferred, and from where, and have
the info necessary to interrupt the transfer.

---

Also did some basic groundwork, adding a queue of transfers to perform,
and adding to the daemon's status information a map of currently running
transfers.

Next up: The daemon will use inotify to notice new and deleted transfer
info files, and update its status info.


code:
Made a release today. Releasing has sure gotten easier with all the
autobuilds to use!

I am now using git-annex to share files with my mom. Here's how the webapp
looks for our family's repository. Soon several of us will be using this
repository.

[[!img assistant/example.png]]

We're using XMPP and rsync.net, so pretty standard setup much like
shown in my last screencast.

Real-world deployments help find bugs, and I found a few:

* If you're running the webapp in `w3m` on a remote computer to set it up,
  some forms are lacking submit buttons. This must be a issue with
  Bootstrap, or HTML5, I guess. I switched to `lynx` and it offers a
  way to submit forms that lack an explicit button.

* Progress bars for downloads from encrypted rsync repos don't update
  during the actual download, but only when gpg is decrypting the
  downloaded file.

* XMPP pushes sometimes fail still. Especially when your mom's computer
  is saturating its limited outgoing network connection uploading hundreds of
  photos. I have not yet determined if this is a packet loss/corruption issue,
  or if the XMPP messages are getting out of order. My gut feeling is it's
  the latter, in which can I can fix this pretty easily by adding sequence
  numbers and some buffering for out of order packets. Or perhaps just
  make it retry failed pushes when this happens.

  Anyway, I found it was useful to set up a regular git repository on a
  server to suppliment the git pushes over XMPP. It's helpful to have
  such a git repository anyway, so that clients can push to there when the
  other client(s) are not online to be pushed to directly over XMPP.


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="68.35.120.66"
 subject="comment 1"
 date="2013-04-05T23:50:14Z"
 content="""
This is probably a Dumb Git Question, but is there a tag in the repo for the new release?  I go into my clone and type \"git tag\" and the latest one I see is 4.20130323.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="deletions by mistake?"
 date="2013-04-06T04:58:09Z"
 content="""
Let's say a family member deletes a file by mistake in a synced diretory?  Looks like this kind of error is easy to go unnoticed.  What would be a good way to handle this?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 3"
 date="2013-04-06T16:26:57Z"
 content="""
Weird, I don't know where the tag went. I've pushed a new one.

If a file is deleted, it's still there in git, and git-annex hangs onto its contents. You need a git history browser to get it back of course.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlUbH3eytydcwlWqv8oauE2Jg4NwcV9uA0"
 nickname="Anna"
 subject="me!"
 date="2013-04-06T18:22:30Z"
 content="""
I want in on the family repository! :-)
"""]]


code:
Finally fixed the bug causing repeated checksumming when a direct mode file
contains duplicate files. I may need to add some cleaning of stale inode
caches eventually.

Meanwhile, Guilhem made `git annex initremote` use higher quality entropy,
with `--fast` getting back to the old behavior of urandom quality entropy.
The assistant doesn't use high quality entropy since I have no way to
prompt when the user would need to generate more. I did have a fun idea to
deal with this: Make a javascript game, that the user can play while
waiting, which would generate enropy nicely. Maybe one day.. ;)

Also made a small but significant change to [[archive directory handling|todo/assistant_smarter_archive_directory_handling]].
Now the assistant syncs files that are in `archive` directories like any
other file, until they reach an archive repository. Then they get dropped
from all the clients. This way, users who don't set up archive repositories
don't need to know about this special case, and users who do want to use
them can, with no extra configuration.

After recent changes, the preferred content expression for transfer
repositories is becoming a bit unweidly, at 212 characters. Probably
time to add support for macros..

`(not (inallgroup=client and copies=client:2) and (((exclude=*/archive/* and exclude=archive/*) or (not (copies=archive:1 or copies=smallarchive:1))) or (not copies=semitrusted+:1))) or (not copies=semitrusted+:1)`

Still, it's pretty great how much this little language lets me express, so
easily.


code:
Developed a way to run the webapp on a remote or headless computer.

The webapp can now be started on a remote or headless computer, just
specify `--listen=address` to make it listen on an address other than
localhost. It'll print out the URL to open to access it.

This doesn't use HTTPS yet, because it'd need to generate a
certificate, and even if it generated a self-signed SSL certificate,
there'd be no easy way for the browser to verify it and avoid a MITM.

So `--listen` is a less secure but easier option; using ssh to forward
the webapp's port to the remote computer is more secure.

(I do have an idea for a way to do this entirely securely, making
the webapp set up the ssh port forwarding, which I have written down
in [[webapp]].. but it would be rather complicated to implement.)

----

Made the webapp rescan for transfers after it's been used to change a
repository's group. Would have been easy, but I had to chase down a
cache invalidation bug.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="Certificate storage"
 date="2013-04-09T00:43:12Z"
 content="""
Why not store the certificate within the repository? Either a common one in, e.g.

    .git-annex.key-cert.pem

or host-specific as per

    .git-annex.key-cert.pem.$HOST

-- Richard
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmU_2tE75oyG0h2ZPN4lcroIKEMC8G-otE"
 nickname="Michael"
 subject="comment 2"
 date="2013-04-09T07:27:53Z"
 content="""
Can you also set the port via listen= ?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 3"
 date="2013-04-09T18:58:40Z"
 content="""
@Richard besides the issues with where to store the private key part, that doesn't help with probably the most common use case for a headless webapp: Setting up a new repository on the remote system, which then might get joined into an existing repository network.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 4"
 date="2013-04-09T19:18:24Z"
 content="""
@Michael you can now! --listen=address:port
"""]]


code:
Did my taxes today. Not very pretty. Planning to run them by a professional.

Reproduced a bug that prevents git-annex from authenticating to the ejabberd
server, and passed the buck upstream with a test case to the author of the
haskell XMPP library.

Added some animations to the webapp to show when it's busy doing things.

Made `git annex webapp --listen=address:port` work.

Added a `annex.web-download-command` setting.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA"
 nickname="Laszlo"
 subject="comment 1"
 date="2013-04-10T13:35:30Z"
 content="""
I don't really want to nagging you, but I'm impatiently waiting
for a new build (for linux) to be able to try out your duplicated file fix.

Please don't delay your next release too much:)

Laszlo

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-04-11T14:57:24Z"
 content="""
You can already try that fix out using the [[install/Linux_standalone]] autobuild.
"""]]


code:
Short day because I spent 3 hours this morning explaining free software
and kickstarter to an accountant. And was away until 3 pm, so how did I get
all this done‽

Eliot pointed out that shutting down the assistant could leave transfers
running. This happened because `git annex transferkeys` is a separate
process, and so it was left to finish up any transfer that was in
process. I've made shutdown stop all transfers that the
assistant started. (Other paired computers could still be connecting to
make transfers even when the assistant is not running, and those are not
affected.)

Added sequence numbers to the XMPP messages used for git pushes. While
these numbers are not used yet, they're available for debugging, and will
help me determine if packets are lost or come out of order. So if you have
experienced problems with XMPP syncing sometimes failing, run tonight's
build of the assistant with `--debug` (or turn on debugging in the webapp
configuration screen), and send me a log by email to
<debuglogs201204@joeyh.name>

Changed the way that autobuilds and manual builds report their version
number. It now includes the date of the last commit, and the abbreviated
commit ID, rather than being some random date after the last release.

Frederik found a bug using the assistant on a FAT filesystem. It didn't
properly handle the files that git uses to stand-in for symlinks in that
situation, and annexed those files. I've fixed this, and even moving
around symlink stand-in files on a FAT filesystem now results in correct
changes to symlinks being committed.


code:
Felt like spending my birthday working on git-annex. Thanks again to
everyone who makes it possible for me to work on something I care about
every day.

----

Did some work on `git annex addurl` today. It had gotten broken in direct
mode (I think by an otherwise good and important bugfix). After fixing
that, I made it interoperate with the webapp. So if you have the webapp
open, it will display progress bars for downloads being run by `git annex
addurl`.

This enhancement meshes nicely with a FlashGot script Andy
contributed, which lets you queue up downloads into your annex from a web
browser. Andy described how to set it up in
[[this_tip|tips/Using_Git-annex_as_a_web_browsing_assistant]].

(I also looked briefly into ways to intercept a drag and drop of a link
into the webapp and make it lauch a download for you. It doesn't seem that
browsers allow javascript to override their standard behavior of loading
links that are dropped into them. Probably good to prevent misuse, but it
would be nice here...)

----

Also, I think I have fixed the progress bars displayed when downloading
a file from an encrypted remote. I did this by hooking up existing
download progress metering (that was only being used to display
a download percentage in the console) into the location log, so the
webapp can use it. So that was a *lot* easier than it could have been,
but still a pretty large patch (500+ lines). Haven't tested this; should work.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawksoT5jmbiZ9UghROilchqy3kNJ1XGqCAo"
 nickname="Rob"
 subject="Drag and Drop"
 date="2013-04-12T03:37:53Z"
 content="""
Is this along the lines of what you are looking for? http://www.dropzonejs.com/
"""]]


code:
[[!comment format=mdwn
 username="marvin"
 ip="91.152.75.65"
 subject="Happy Birthday"
 date="2013-04-12T17:52:43Z"
 content="""
Great work!
"""]]


code:
[[!comment format=mdwn
 username="robconnolly"
 ip="118.90.78.117"
 subject="comment 3"
 date="2013-04-13T03:12:14Z"
 content="""
Thanks for fixing the addurl bug and Happy Birthday!
"""]]


code:
[[!comment format=mdwn
 username="andy"
 ip="99.48.75.171"
 subject="Just in case it might be helpful..."
 date="2013-04-14T05:56:47Z"
 content="""
For whatever reason, dragging links onto the page text in the \"Using drag and drop for selecting\" section of <http://www.html5rocks.com/en/tutorials/file/dndfiles/#toc-selecting-files-dnd> does not trigger the usual page change. I don't really know why--I couldn't get it to work on my own with Firebug. And thanks for git-annex--love it!
"""]]


code:
Spent today bulding the Evil Splicer, a program that parses `ghc -ddump-splices`
output, and uses it to expand Template Haskell splices in source code.
I hope to use this crazy hack to get the webapp working on Android.

This was a good opportunity to use the
[Parsec](http://hackage.haskell.org/package/parsec) library for parsing the
ghc output. I've never really used it before, but found it quite nice to
work with. The learning curve, if you already know monads and applicatives,
is about 5 minutes. And instead of ugly regular expressions, you can work
with nice code that's easily composable and refactorable. Even the ugly
bits come out well:

[[!format haskell """
	{- All lines of the splice result will start with the same
	- indent, which is stripped. Any other indentation is preserved. -}
	i <- lookAhead indent
	result <- unlines <$> many1 (string i >> restOfLine)
"""]]

Anyway, it works.. sorta. The parser works great! The splices that ghc
outputs are put into the right places in the source files, and formatted in
a way that ghc is able to build. Often though, they contain code that
doesn't actually build as-is. I'm working to fix up the code to get closer
to buildable.

----

Meanwhile, guilhem has made ssh connection caching work for rsync special
remotes! It's very nice to have another developer working on git-annex. :)


code:
Fixed a bug where the locked down ssh key that the assistant sets up to
access the annex on a remote server was being used by ssh *by default* for
all logins to that server.

That should not have happened. The locked down key is written to a filename
that ssh won't use at all, by default. But, I found code in gnome-keyring
that watches for `~/.ssh/*.pub` to appear, and automatically adds all such
keys to the keyring. In at least some cases, probably when it has no other
key, it then tells ssh to go ahead and use that key. Astounding.

To avoid this, the assistant will store its keys in <s>`~/.ssh/annex/`</s>
`~/.ssh/git-annex/` instead. gnome-keyring does not look there (verified in
the source). If you use gnome-keyring and have set up a repository on a
remote server with the assistant, I'd recommend moving the keys it set up
and editing `~/.ssh/config` to point to their new location.

gnome-keyring is not the only piece of software that has a bad
interaction with git-annex. I've been working on a bug that makes git-annex
fail to authenticate to ejabberd. ejabberd 2.1.10 got support for
SCRAM-SHA-1, but its code violates the RFC, and chokes on an address
attribute that the haskell XMPP library provides. I hope to get this fixed
in ejabberd.

Also did some more work on the Evil Splicer today, integrating it into the
build of the Android app, and making it support incremental building.
Improved its code generation, and am at the milestone where it creates
valid haskell code for the entire `Assistant/WebApp/Types.hs` file,
where Template Haskell expands 2 lines into 2300 lines of code!


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="Confusing name for storage directory"
 date="2013-04-14T23:55:03Z"
 content="""
`~/.ssh/annex/` would make sense for ssh-annex. I would suggest using `~/.ssh/git-annex/` or similar, instead.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-04-14T23:59:44Z"
 content="""
I agree, change made
"""]]


code:
Back to really working toward an Android webapp now. I have been improving
the EvilSplicer, and the build machinery, and build environment all day.
Slow but steady progress.

First milestone of the day was when I got `yesod-form` to build with all
Template Haskell automatically expanded by the EvilSplicer. (With a few
manual fixups where it's buggy.)

At this point the Android build with the webapp enabled successfully
builds several files containing Yesod code.. And I suspect I am very close
to getting a first webapp build for Android.


code:
Several bug fixes today, and got mostly caught up on recent messages.
Still have a backlog of two known bugs that I cannot reproduce well enough
to have worked on, but I am thinking I will make a release tomorrow. There
have been a lot of changes in the 10 days since the last release.

----

I am, frustratingly, stuck building the webapp on Android with no forward
progress today (and last night) after such a productive day yesterday.

The expanded Template Haskell code of the webapp fails to compile,
whereever type safe urls are used.

<pre>
Assistant/WebApp/Types.hs:95:63:
    Couldn't match expected type `Route WebApp -> t2'
                with actual type `Text'
    The function `urender_a1qcK' is applied to three arguments,
    but its type `Route WebApp -> [(Text, Text)] -> Text' has only two
    In the expression: urender_a1qcK u_a1qcL [] LogR
    In the first argument of `toHtml', namely
      `(\ u_a1qcL -> urender_a1qcK u_a1qcL [] LogR)'
</pre>

My best guess is this is a mismatch between the versions of yesod (or other
libraries) used for the native and cross compiled ghc's. So I've been
slowly trying to get a fully matched set of versions in between working on
bugs.


code:
Starting to travel, so limited time today.

Yet Another Thread added to the assistant, all it does is watch for changes
to transfer information files, and update the assistant's map of transfers
currently in progress. Now the assistant will know if some other repository
has connected to the local repo and is sending or receiving a file's
content.

This seemed really simple to write, it's just 78 lines of code. It worked
100% correctly the first time. :) But it's only so easy because I've got
this shiny new inotify hammer that I keep finding places to use in the
assistant.

Also, the new thread does some things that caused a similar thread (the
merger thread) to go into a MVar deadlock. Luckily, I spent much of
[day 19](day_19__random_improvements) investigating and fixing that
deadlock, even though it was not a problem at the time.

So, good.. I'm doing things right and getting to a place where rather
nontrivial features can be added easily.

--

Next up: Enough nonsense with tracking transfers... Time to start actually
transferring content around!


code:
Late last night, I successfully built the full webapp for Android!

That was with several manual modifications to the generated code, which I
still need to automate. And I need to set up the autobuilder properly
still. And I need to find a way to make the webapp open Android's web browser
to URL. So it'll be a while yet until a package is available
to try. But what a milestone!

The point I was stuck on all day yesterday was generated code that
looked like this:

[[!format haskell """
(toHtml
  (\ u_a2ehE -> urender_a2ehD u_a2ehE []
    (CloseAlert aid)))));
"""]]

That just couldn't type check at all. Most puzzling. My best guess is that
`u_a2ehE` is the dictionary GHC passes internally to make a typeclass work,
which somehow leaked out and became visible. Although
I can't rule out that I may have messed something up in my build environment.
The EvilSplicer has a hack in it that finds such code and converts it to
something like this:

[[!format haskell """
(toHtml
  (flip urender_a2ehD []
    (CloseAlert aid)))));
"""]]

I wrote some more about the process of the Android port in my personal blog:
[Template Haskell on impossible architectures](http://joeyh.name/blog/entry/Template_Haskell_on_impossible_architectures/)

----

Release day today. The OSX builds are both not available yet for this
release, hopefully will come out soon.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA"
 nickname="Laszlo"
 subject="Duplicated files resolved"
 date="2013-04-18T05:52:35Z"
 content="""
Hi,

Seems like the duplicated file issue finally resolved in this release,
also the logging is not so detailed, so it does not explode the disk space either.

I think a new bug introduced in this release,
namely if you create a repository and a backup repository of it (pendrive), then this repository is not in direct mode by default.
(and there is a missing direct = True variable in that repository (/mnt/dat/annex_backup/.git/annex/config)).

Also I run into resource exhausted bug, and every thread is crashed.
Can git-annex only watch a directory, and not every file in it?

I will create a bugreport for resource exhausted.

Best,
 Laszlo

"""]]


code:
Finished the last EvilSplicer tweak and other fixes to make the
Android webapp build without any hand-holding.

Currently setting up the Android autobuilder to include the webapp in its
builds. To make this work I had to set up a new chroot with all the right
stuff installed.

Investigated how to make the Android webapp open a web browser when run.
As far as I can tell (without access to an Android device right now),
`am start -a android.intent.action.VIEW -d http://localhost/etc` should do
it.

Seems that git 1.8.2 broke the assistant. I've put in a fix but have not
yet tested it.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlJ2utMQgMEYAOs3Dfc6eZRyUzt4acNXUU"
 nickname="David"
 subject="am start -a android.intent.action.VIEW -d http://localhost/etc"
 date="2013-04-18T22:39:50Z"
 content="""
Just tested the command on an Android device, it works well.
"""]]


code:
Got WebDAV enabled in the Android build. Had to deal with some system calls
not available in Android's libc.

New poll: [[polls/Android_default_directory]]


code:
Today was not a work day for me, but I did get a chance to install
git-annex in real life while visiting. Was happy to download the standalone
Linux tarball and see that it could be unpacked, and git-annex webapp
started just by clicking around in the GUI. And in very short order got it
set up.

I was especially pleased to see my laptop noticed this new repository
had appeared on the network via XMPP push, and started immediately
uploading files to my rsync.net transfer repository so the new
repository could get them.

Did notice that the standalone tarball neglected to install a
FDO menu file. Fixed that, and some other minor issues I noticed.

I also got a brief chance to try the Android webapp. It fails to start;
apparently `getaddrinfo` doesn't like the flags passed to it and is
failing. As failure modes go, this isn't at all bad. I can certainly work
around it with some hardcoded port numbers, but I want to fix it the right
way. Have ordered a replacement battery for my dead phone so I can use it
for Android testing.


code:
Ported all the C libraries needed for XMPP to Android. (gnutls,
libgcrypt, libgpg-error, nettle, xml2, gsasl, etc). Finally
got it all to link. What a pain.

Bonus: Local pairing support builds for Android now, seems recent changes to
the network library for WebDAV also fixed it.


code:
Got the OSX autobuilder back running, and finally got a OSX build up for
the 4.20130417 release. Also fixed the OSX app build machinery to handle
rpath.

Made the assistant (and `git annex sync`) sync with git remotes that have
`annex-ignore` set. So, `annex-ignore` is only used to prevent using
the annex of a remote, not syncing with it. The benefit of this change
is that it'll make the assistant sync the local git repository with
a git remote that is on a server that does not have git-annex installed.
It can even sync to github.

Worked around more breakage on misconfigured systems that don't have GECOS
information.

... And other bug fixes and bug triage.


code:
[[!comment format=mdwn
 username="spwhitton"
 ip="192.76.7.217"
 subject="comment 1"
 date="2013-04-23T17:11:46Z"
 content="""
If I set my annex-ignore'd respository to `annex-sync = false` will this override the new behaviour?  My remote requires mounting a fuse mount, so this change will just result in my seeing lots of sync failure messages in the webapp.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-04-23T20:16:26Z"
 content="""
Yes, setting that, or just clicking to pause syncing to a remote in the webapp will still work.

Although I don't think you need to.. If the remote is on a filesystem mount point, the assistant will just ignore it when it's not mounted. And will automatically detect when it gets mounted and sync with it, too. Same as USB drives are handled.
"""]]


code:
[[!comment format=mdwn
 username="spwhitton"
 ip="163.1.167.50"
 subject="comment 3"
 date="2013-04-24T17:09:50Z"
 content="""
It's a userspace fuse mount so the assistant won't know it's a mountpoint unless mounted.  Also I don't want the automatic sync since I like to keep this respository manual as a kind of backup.  Thanks for the info.
"""]]


code:
There seem to be a steady state of enough bug reports coming in that I can
work on them whenever I'm not working on anything else. As I did all day
today.

This doesn't bother me if the bug reports are of real bugs that I can
reproduce and fix, but I'm spending a lot of time currently following up to
messages and asking simple questions like "what version number" and "can I
please see the whole log file". And just trying to guess what a vague
problem report means and read people's minds to get to a definite bug
with a test case that I can then fix.

I've noticed the overall quality of bug reports nosedive over the past
several months. My guess is this means that git-annex has found a less
technical audience. I need to find something to do about this.

With that whining out of the way ...
I fixed a pretty ugly bug on FAT/Android today, and
I am 100% caught up on messages right now!


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkQafKy7hNSEolLs6TvbgUnkklTctUY9LI"
 nickname="Zellyn"
 subject="Bugs..."
 date="2013-04-23T23:30:47Z"
 content="""
Perhaps if you made a bug report form that asked for the version number (with helpful command to get it printed below), and had a checkbox for \"full log attached\", etc. you might be able to weed out low-value bugs...  Not that adding UI to force good bugs really works, but there might be a couple of low-hanging fruit before the diminishing returns kick in.
"""]]


code:
Working on assistant's performance when it has to add a whole lot of files
(10k to 100k).

Improved behavior in several ways, including fixing display
of the alert in the webapp when the default inotify limit of 8192
directories is exceeded.

Created a new TList data type, a transactional DList. Much nicer
implementation than the TChan based thing it was using to keep track of the
changes, although it only improved runtime and memory usage a little bit.
The way that this is internally storing a function in STM and modifying
that function to add items to the list is way cool.

Other tuning seems to have decreased the time it would take to import 100k
files from somewhere in the range of a full day (too long to wait to see),
to around 3.5 hours. I don't know if that's good, but it's certainly better.


code:
Very productive & long day today, spent adding a new feature to the
webapp: Internet Archive support!

[[!img /assistant/iaitem.png]]

git-annex already supported using archive.org via its S3 special remotes,
so this is just a nice UI around that.

How does it decide which files to publish on archive.org? Well,
the item has a unique name, which is based on the description
field. Any files located in a directory with that name will be uploaded
to that item. (This is done via a new preferred content expression I added.)

So, you can have one repository with multiple IA items attached, and
sort files between them however you like.
I plan to make a screencast eventually demoing that.

Another interesting use case, once the Android webapp is done, would be add
a repository on the DCIM directory, set the archive.org repository to
prefer all content, and *bam*, you have a phone or tablet that
auto-publishes and archives every picture it takes.

Another nice little feature added today is that whenever a file is uploaded
to the Internet Archive, its public url is automatically recorded, same
as if you'd ran `git annex addurl`. So any users who can clone your
repository can download the files from archive.org, without needing any
login or password info. This makes the Internet Archive a nice way to
publish the large files associated with a public git repository.


code:
Quiet day. Only did minor things, like adding webapp UI for changing the
directory used by Internet Archive remotes, and splitting out an
`enableremote` command from `initremote`.

My Android development environment is set up and ready to go on my Motorola
Droid. The current Android build of git-annex fails to link at run time, so
my work is cut out for me. Probably broke something while enabling XMPP?


code:
In a series of airport layovers all day. Since I woke up at 3:45 am,
didn't feel up to doing serious new work, so instead I worked through some
OSX support backlog.

git-annex will now use Haskell's SHA library if the `sha256sum`
command is not available. That library is slow, but it's guaranteed to be
available; git-annex already depended on it to calculate HMACs.

Then I decided to see if it makes sense to use the SHA library
when adding smaller files. At some point, its slower implementation should
win over needing to fork and parse the output of `sha256sum`. This was
the first time I tried out Haskell's
[Criterion](http://hackage.haskell.org/package/criterion) benchmarker,
and I built this simple benchmark in short order.

[[!format haskell """
import Data.Digest.Pure.SHA
import Data.ByteString.Lazy as L
import Criterion.Main
import Common

testfile :: FilePath
testfile = "/tmp/bar" -- on ram disk

main = defaultMain
        [ bgroup "sha256"
                [ bench "internal" $ whnfIO internal
                , bench "external" $ whnfIO external
                ]
        ]

internal :: IO String
internal = showDigest . sha256 <$> L.readFile testfile

external :: IO String
external = pOpen ReadFromPipe "sha256sum" [testfile] $ \h ->
        fst . separate (== ' ') <$> hGetLine h
"""]]

The nice thing about benchmarking in Airports is when you're running a
benchmark locally, you don't want to do anything else with the computer,
so can alternate people watching, spacing out, and analizing results.

100 kb file:

	benchmarking sha256/internal
	mean: 15.64729 ms, lb 15.29590 ms, ub 16.10119 ms, ci 0.950
	std dev: 2.032476 ms, lb 1.638016 ms, ub 2.527089 ms, ci 0.950

	benchmarking sha256/external
	mean: 8.217700 ms, lb 7.931324 ms, ub 8.568805 ms, ci 0.950
	std dev: 1.614786 ms, lb 1.357791 ms, ub 2.009682 ms, ci 0.950

75 kb file:

	benchmarking sha256/internal
	mean: 12.16099 ms, lb 11.89566 ms, ub 12.50317 ms, ci 0.950
	std dev: 1.531108 ms, lb 1.232353 ms, ub 1.929141 ms, ci 0.950

	benchmarking sha256/external
	mean: 8.818731 ms, lb 8.425744 ms, ub 9.269550 ms, ci 0.950
	std dev: 2.158530 ms, lb 1.916067 ms, ub 2.487242 ms, ci 0.950

50 kb file:

	benchmarking sha256/internal
	mean: 7.699274 ms, lb 7.560254 ms, ub 7.876605 ms, ci 0.950
	std dev: 801.5292 us, lb 655.3344 us, ub 990.4117 us, ci 0.950

	benchmarking sha256/external
	mean: 8.715779 ms, lb 8.330540 ms, ub 9.102232 ms, ci 0.950
	std dev: 1.988089 ms, lb 1.821582 ms, ub 2.181676 ms, ci 0.950

10 kb file:

	benchmarking sha256/internal
	mean: 1.586105 ms, lb 1.574512 ms, ub 1.604922 ms, ci 0.950
	std dev: 74.07235 us, lb 51.71688 us, ub 108.1348 us, ci 0.950

	benchmarking sha256/external
	mean: 6.873742 ms, lb 6.582765 ms, ub 7.252911 ms, ci 0.950
	std dev: 1.689662 ms, lb 1.346310 ms, ub 2.640399 ms, ci 0.950

It's possible to get nice graphical reports out of Criterion, but
this is clear enough, so I stopped here. 50 kb seems a reasonable
cutoff point.

I also used this to benchmark the SHA256 in Haskell's Crypto package.
Surprisingly, it's a *lot* slower than even the Pure.SHA code.
On a 50 kb file:

	benchmarking sha256/Crypto
	collecting 100 samples, 1 iterations each, in estimated 6.073809 s
	mean: 69.89037 ms, lb 69.15831 ms, ub 70.71845 ms, ci 0.950
	std dev: 3.995397 ms, lb 3.435775 ms, ub 4.721952 ms, ci 0.950

There's another Haskell library, [SHA2](http://hackage.haskell.org/package/SHA2),
which I should try some time.


code:
Turns out my old Droid has such an old version of Android (2.2) that
it doesn't work with any binaries produced by my haskell cross-compiler. I
think it's using a symbol not in its version of libc. Since upgrading this
particular phone is a ugly process and the hardware is dying anyway (bad
USB power connecter), I have given up on using it, and ordered an Android
tablet instead to use for testing. Until that arrives, no Android. Bah.
Wanted to get the Android app working in April.

Instead, today I worked on making the webapp require less redundant
password entry when adding multiple repositories using the same cloud
provider. This is especially needed for the Internet Archive, since users
will often want to have quite a few repositories, for different IA items.
Implemented it for box.com, and Amazon too.

Francois Marier has built an Ubuntu PPA for git-annex, containing the
current version, with the assistant and webapp. It's targeted at Precise,
but I think will probably also work with newer releases.
<https://launchpad.net/~fmarier/+archive/ppa>

Probably while I'm waiting to work on Android again, I will try to
improve the situation with using a single XMPP account for multiple
repositories. Spent a while today thinking through ways to improve the
design, and have some ideas.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnLgNOfkl8c6oQu9b42U0Pm_uC4n3Gkgiw"
 nickname="Mark"
 subject="Thanks!"
 date="2013-04-29T01:24:55Z"
 content="""
I was not looking forward to figuring out how to come up with all of those haskell build deps on Precise, so thanks to François Marier for putting this together.  (Does Ubuntu/launchpad autobuild all of the architectures, or do they need to be contributed? I see that `armhf` isn't there at the moment...)
"""]]


code:
Took 2 days in a row off, because I noticed I have forgotten to do that
since February, or possibly earlier, not counting trips. Whoops!

Also, I was feeling overwhelmed with the complexity of fixing XMPP to not
be buggy when there are multiple separate repos using the same XMPP
account. Let my subconscious work on that, and last night it served up the
solution, in detail. Built it today.

It's only a partial solution, really. If you want to use the same XMPP
account for multiple separate repositories, you cannot use the "Share with
your other devices" option to pair your devices. That's because XMPP
pairing assumes all your devices are using the same XMPP account, in order
to avoid needing to confirm on every device each time you add a new device.
The UI is clear about that, and it avoids complexity, so I'm ok with that.

But, if you want to instead use "Share with a friend", you now can use the
same XMPP account for as many separate repositories as you like. The
assistant now ignores pushes from repositories it doesn't know about.
Before, it would merge them all together without warning.

----

While I was testing that, I think I found out the real reason why XMPP
pushes have seemed a little unreliable. It turns out to not be an XMPP
issue at all! Instead, the merger was simply not always
noticing when `git receive-pack` updated a ref, and not merging it into
master. That was easily fixed.

----

Adam Spiers has been getting a `.gitignore` query interface suitable for
the assistant to use into `git`, and he tells me it's landed in `next`.
I should soon check that out and get the assistant using it. But first,
Android app!


code:
Pushed out a release today. Looking back over April, I'm happy with it as a
bug fix and stabilization month. Wish I'd finished the Android app in April,
but let's see what happens tomorrow.

Recorded part of a screencast on using Archive.org, but `recordmydesktop`
lost the second part. Grr. Will work on that later.


code:
[[!img /android/webapp.png alt="git-annex webapp on Android"]]

I fixed what I thought was keeping the webapp from working on Android, but
then it started segfaulting every time it was started. Eventually I
determined this segfault happened whenever haskell code called
`getaddrinfo`. I don't know why. This is particularly weird since I had
a demo web server that used `getaddrinfo` working way back in
[[day_201__real_Android_wrapup]]. Anyway, I worked around it by not using
`getaddrinfo` on Android.

Then I spent 3 hours stuck, because the webapp seemed to run, but
nothing could connect to the port it was on. Was it a firewall? Was
the Haskell threaded runtime's use of `accept()` broken? I went all the way
down to the raw system calls, and back, only to finally notice I had `netstat`
available on my Android. Which showed it was not listening to the port I
thought it was!

Seems that `ntohs` and `htons` are broken somehow. To get the
screenshot, I fixed up the port manually. Have a build running that
should work around the issue.

Anyway, the webapp works on Android!


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnxp2XU8gIribhhGhGuYtU6eMMwHv5gUGI"
 nickname="Amitai"
 subject="Woohoo!"
 date="2013-05-03T03:13:25Z"
 content="""
(See above)
"""]]


code:
There's a new page [[/Android]] that documents using git-annex on Android
in detail.

The Android app now opens the webapp when a terminal window is opened.
This is good enough for trying it out easily, but far from ideal.

Fixed an EvilSplicer bug that corrupted newlines in
the static files served by the webapp. Now the icons in the webapp
display properly, and the javascript works.

Made the startup screen default to `/sdcard/annex` for the repository
location, and also have a button to set up a camera repository. The camera
repository is put in the "source" preferred content group, so it will only
hang onto photos and videos until they're uploaded off the Android device.

Quite a lot of other small fixes on Android. At this point I've tested the
following works:

* Starting webapp.
* Making a repository, adding files.
* All the basic webapp UI.

However, I was not able to add any remote repository using only the webapp,
due to some more problems with the network stack.

* Jabber and Webdav don't quite work ("getProtocolByname: does not exist (no
  such protocol name: tcp)").
* SSH server fails.
  ("Network/Socket/Types.hsc:(881,3)-(897,61): Non-exhaustive patterns in case")
  I suspect it will work if I disable the DNS expansion code.

So, that's the next thing that needs to be tackled.

If you'd like to play with it in its current state, I've updated the
Android builds to incorporate all my work so far.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkGCmVc5qIJaQQgG82Hc5zzBdAVdhe2JEM"
 nickname="Bruno"
 subject="comment 1"
 date="2013-05-04T05:30:26Z"
 content="""
> The camera repository is put in the \"source\" preferred content group, so it will only hang onto photos and videos until they're uploaded off the Android device.

This is very nice but I'm wondering if I take a picture that I want to share using an Android app like Facebook, would the picture get deleted before I could share it?

Would there be a way to access old pictures? For example, to show them to people you meet.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-05-06T15:08:19Z"
 content="""
As it stands, using the \"source\" repository type will tend to remove the picture from that repository pretty quickly, assuming a fast network connection. I think if you want to do things on your phone with pictures you take, it's best to use \"client\" instead. Perhaps with an archive repository set up, so you can move stuff to an \"archive\" folder when done with it to get it removed from the phone.

Whether \"source\" is a good default in this case remains to be seen...

It might also be possible to change \"source\" in some way to make it more useful. For example, if it held onto new files for a day after they were created, you'd have plenty of time to use them before they expire off your phone. I have posted some thoughts about this to [[partial_content]].
"""]]


code:
Created a backport of the latest git-annex release for Debian 7.0 wheezy.
Needed to backport a dozen haskell dependencies, but not too bad.
This will be available in the backports repository once Debian
starts accepting new packages again. I plan to keep the backport up-to-date
as I make new releases.

The cheap Android tablet I bought to do this last Android push with came
pre-rooted from the factory. This may be why I have not seen this bug:
[[bugs/Android_app_permission_denial_on_startup]]. If you have Android
4.2.2 or a similar version, your testing would be helpful for me to know if
this is a widespread problem. I have an idea about a way to work around the
problem, but it involves writing Java code, and probably polling a file,
ugh.

Got S3 support to build for Android. Probably fails to work
due to the same network stack problems affecting WebDAV and Jabber.

Got removable media mount detection working on Android. Bionic has an
amusing stub for `getmntent` that prints out "FIX ME! implement
getmntent()". But, `/proc/mounts` is there, so I just parse it.
Also enabled the app's `WRITE_MEDIA_STORAGE` permission to allow
access to removable media. However, this didn't seem to do anything. :(

Several fixes to make the Android webapp be able to set up repositories on
remote ssh servers. However, it fails at the last hurdle with what
looks like a `git-annex-shell` communication problem. Almost there..


code:
This seems a very auspicious day to have finally gotten the Android app
doing something useful! I've fixed the last bugs with using it to set up a
remote ssh server, which is all I need to make my Android tablet
sync photos I take with a repository on my laptop.

[[!img /android/DCIM.png alt="git-annex webapp running on Android"]]

I set this up entirely in the GUI, except for needing to switch to the
terminal twice to enter my laptop's password.

How fast is it? Even several minute long videos transfer before I can
switch from the camera app to the webapp. To get this screenshot with it in
the process of syncing, I had to take a dozen pictures in a minute. Nice
problem to have. ;)

Have fun trying this out for yourself after tonight's autobuilds. But a
warning: One of the bugs I fixed today had to be fixed in `git-annex-shell`,
as run on the ssh server that the Android connects to. So the Android app
will only work with ssh servers running a new enough version of git-annex.

----

Worked on geting git-annex into Debian testing, which is needed before
the wheezy backport can go in. Think I've worked around most of the issues
that were keeping it from building on various architectures.

Caught up on some bug reports and fixed some of them.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm8wY171R5c4u_jPmB6LU6n6Px2xePM4sE"
 nickname="Efraim"
 subject="comment 1"
 date="2013-05-07T06:20:35Z"
 content="""
that's amazing.  I saw in one of the conference videos that you said you didn't use dropbox, and I wanted to let you know (in case you were curious) that this is one of the features that they have on their android app, that pictures and videos are uploaded to one's dropbox.  Two questions, is there a toggle or option somewhere to upload over wifi only? and did you set up the tablet's camera folder as a source directory?
"""]]


code:
Put in a fix for `getprotobyname` apparently not returning anything for
"tcp" on Android. This might fix all the special remotes there, but I don't
know yet, because I have to rebuild a lot of Haskell libraries to try it.

So, I spent most of today writing a script to build all the Haskell
libraries for Android from scratch, with all my patches.


code:
Fixed a nasty bug that affects at least some FreeBSD systems. It misparsed
the output of `sha256`, and thought every file had a SHA256 of "SHA256".
Added multiple layers of protection against checksum programs not having
the expected output format.

Lots more building and rebuilding today of Android libs than I wanted to do.
Finally have a completly clean build, which might be able to open TCP
connections. Will test tomorrow.

In the meantime, I fired up the evil twin of my development laptop.
It's identical, except it runs Windows.

I installed the Haskell Platform for Windows on it, and removed
some of the bloatware to free up disk space and memory for development.
While a rather disgusting experience, I certainly have a usable Haskell
development environment on this OS a lot faster than I did on Android!
Cabal is happily installing some stuff, and other stuff wants me to install
Cygwin.

So, the clock on my month of working on a Windows port starts now. Since
I've already done rather a lot of ground work that was necessary for a
Windows port (direct mode, crippled filesystem support), and for general
sanity and to keep everything else from screeching to a halt, I plan to
only spend half my time messing with Windows over the next 30 days.


code:
It all came together for Android today. Went from a sort of working app
to a fully working app!

* rsync.net works.
* Box.com appears to work -- at least it's failing with the same
  timeout I get on my linux box here behind the firewall of dialup doom.
* XMPP is working too!

These all needed various little fixes. Like loading TLS certificates from
where they're stored on Android, and ensuring that totally crazy file
permissions from Android (----rwxr-x for files?!) don't leak out into rsync
repositories. Mostly though, it all just fell into place today.
Wonderful..

The Android autobuild is updated with all of today's work, so try it out.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkGCmVc5qIJaQQgG82Hc5zzBdAVdhe2JEM"
 nickname="Bruno"
 subject="comment 1"
 date="2013-05-10T23:52:43Z"
 content="""
I have a 'Connection timed out' when I try to set-up my Gtalk account on my Nexus 4 (4.2.2). I tested it at home without a web proxy.

Here's a screen capture: [http://i.imgur.com/RBcNjSv.png](http://i.imgur.com/RBcNjSv.png).
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmmLfBDEDFfEOba8Ra46nsnTmoNHFsLObo"
 nickname="Brian"
 subject="I also see the connection timed out"
 date="2013-05-15T00:19:34Z"
 content="""
Same Android device and same behavior as the above comment. While the connection attempt was being made, I switched to the terminal window and saw a connection to 173.194.46.86:5222 in the SYN_SENT state (and that connection wasn't there prior to the jabber configuration attempt).

Even from my desktop, manual connection attempts to 173.194.46.86:5222 timed out.

Then I ran the same jabber configuration from my desktop and it worked fin. Using netstat, I could see it connected to 74.125.133.125:5222.

So is the Android version using the wrong address?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 3"
 date="2013-05-19T20:56:25Z"
 content="""
For the record, there was a bug with XMPP and SRV records. This got fixed in time for the last release.
"""]]


code:
So as not to bury the lead, I've been hard at work on my first day in
Nicaragua, and ** the git-annex assistant fully syncs files (including
their contents) between remotes now !! **

Details follow..

Made the committer thread queue Upload Transfers when new files
are added to the annex. Currently it tries to transfer the new content
to *every* remote; this inefficiency needs to be addressed later.

Made the watcher thread queue Download Transfers when new symlinks
appear that point to content we don't have. Typically, that will happen
after an automatic merge from a remote. This needs to be improved as it
currently adds Transfers from every remote, not just those that have the
content.

This was the second place that needed an ordered list of remotes
to talk to. So I cached such a list in the DaemonStatus state info.
This will also be handy later on, when the webapp is used to add new
remotes, so the assistant can know about them immediately.

Added YAT (Yet Another Thread), number 15 or so, the transferrer thread
that waits for transfers to be queued and runs them. Currently a naive
implementation, it runs one transfer at a time, and does not do anything
to recover when a transfer fails.

Actually transferring content requires YAT, so that the transfer
action can run in a copy of the Annex monad, without blocking
all the assistant's other threads from entering that monad while a transfer
is running. This is also necessary to allow multiple concurrent transfers
to run in the future.

This is a very tricky piece of code, because that thread will modify the
git-annex branch, and its parent thread has to invalidate its cache in
order to see any changes the child thread made. Hopefully that's the extent
of the complication of doing this. The only reason this was possible at all
is that git-annex already support multiple concurrent processes running
and all making independent changes to the git-annex branch, etc.

After all my groundwork this week, file content transferring is now
fully working!


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawldKnauegZulM7X6JoHJs7Gd5PnDjcgx-E"
 nickname="Matt"
 subject="Source code"
 date="2012-07-06T00:12:15Z"
 content="""
Hi Joey,

Is the source code for git-annex assistant available somewhere?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 subject="comment 2"
 date="2012-07-06T00:21:43Z"
 content="""
It's in the `assistant` branch of git://git-annex.branchable.com/
"""]]


code:
Set up my Windows development environment. For future reference, I've
installed:

* haskell platform for windows
* cygwin
* gcc and a full C toolchain in cygwin
* git from upstream (probably git-annex will use this)
* git in cygwin (the other git was not visible inside cygwin)
* vim in cygwin
* vim from upstream, as the cygwin vim is not very pleasant to use
* openssh in cygwin (seems to be missing a ssh server)
* rsync in cygwin
* Everything that `cabal install git-annex` is able to install successfully.
  This includes all the libraries needed to build regular git-annex,
  but not the webapp. Good start though.

Result basically feels like a linux system that can't decide which way
slashes in paths go. :P I've never used Cygwin before (I last used a
Windows machine in 2003 for that matter), and it's a fairly impressive hack.

----

Fixed up git-annex's configure program to run on Windows (or, at least, in
Cygwin), and have started getting git-annex to build.

For now, I'm mostly stubbing out functions that use unix stuff. Gotten the
first 44 of 300 source files to build this way.

Once I get it to build, if only with stubs, I'll have a good
idea about all the things I need to find Windows equivilants of.
Hopefully most of it will be provided by
<http://hackage.haskell.org/package/unix-compat-0.3.0.1>.

----

So that's the plan. There is a possible shortcut, rather than doing a full
port. It seems like it would probably not be too hard to rebuild ghc inside
Cygwin, and the resulting ghc would probably have a full POSIX emulation
layer going through cygwin. From ghc's documentation, it looks like that's
how ghc used to be built at some point in the past, so it would probably
not be too hard to build it that way. With such a cygwin ghc, git-annex
would probably build with little or no changes. However, it would be a
git-annex targeting Cygwin, and not really a native Windows port. So
it'd see Cygwin's emulated POSIX filesystem paths, etc. That
seems probably not ideal for most windows users.. but if I get really stuck
I may go back and try this method.


code:
After working on it all day, git-annex now builds on Windows!

Even better, `git annex init` works. So does `git annex status`, and
probably more. Not `git annex add` yet, so I wasn't able to try much more.

I didn't have to add many stubs today, either. Many of the missing Windows
features were only used in code paths that made git-annex faster, but I
could fall back to a slower code path on Windows.

The things that are most problimatic so far:

* POSIX file locking. This is used in git-annex in several places to
  make it safe when multiple git-annex processes are running. I put in
  really horrible dotfile type locking in the Windows code paths, but I
  don't trust it at all of course.
* There is, apparently, no way to set an environment variable in Windows
  from Haskell. It is only possible to set up a new process' environment
  before starting it. Luckily most of the really crucial environment
  variable stuff in git-annex is of this latter sort, but there were
  a few places I had to stub out code that tries to manipulate git-annex's
  own environment.

The `windows` branch has a diff of 2089 lines. It add 88 ifdefs to the code
base. Only 12 functions are stubbed out on Windows. This could be so much
worse.

Next step: Get the test suite to build. Currently ifdefed out because it
uses some stuff like `setEnv` and `changeWorkingDirectory` that I don't know
how to do in Windows yet.


code:
It's remarkable that a bad decision made in 1982 can cause me to waste an
entire day in 2013. Yes, `/` vs `\` fun time. Even though I long ago
converted git-annex to use the haskell `</>` operator wherever it builds
up paths (which transparently handles either type of separator), I still
spent most of today dealing with it. Including some libraries I use that
get it wrong. Adding to the fun is that git uses `/` internally, even on
Windows, so Windows separated paths have to be converted when being fed
into git.

Anyway, `git annex add` now works on Windows. So does `git annex find`,
and `git annex whereis`, and probably most query stuff.

Today was very un-fun and left me with a splitting headache, so I will
certainly *not* be working on the Windows port tomorrow.


code:
[[!comment format=mdwn
 username="http://jamesjustjames.wordpress.com/"
 nickname="purpleidea"
 subject="Isn't windows deprecated?"
 date="2013-05-13T05:51:20Z"
 content="""
I feel bad for you, however I respect you for keeping your promise to try and hack on Windows. I had to port over some code than ran beautifully on GNU/Linux and it was more trouble than it was worth. In the end it was never used :P

Cheers!
"""]]


code:
Spent some time today to get caught up on bug reports and website traffic.
Fixed a few things.

Did end up working on Windows for a while too. I got `git annex drop`
working. But nothing that moves content quite works yet..

I've run into a stumbling block with `rsync`. It thinks that
`C:\repo` is a path on a ssh server named "C". Seems I will need to translate
native windows paths to unix-style paths when running rsync.

[[!meta date="13 May 2013"]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkHscTHMCNvjJ6nLI1VpsBrJFI5FTwhUT4"
 nickname="David"
 subject="Just use Unix Paths like git does?"
 date="2013-05-15T20:43:34Z"
 content="""
Can't you just use unix-style paths for your internal data structures, and wrap/patch the filesystem I/O code to convert between the two?
"""]]


code:
The Windows port can now do everything in the [[walkthrough]]. It can use
both local and remote git repositories. Some special remotes work
(directory at least; probably rsync; likely any other special remote that
can have its dependencies built). Missing features include most special
remotes, gpg encryption, and of course, the assistant.

Also built a NullSoft installer for git-annex today. This was made very
easy when I found the Haskell ncis library, which provides a DSL embedding
the language used to write NullSoft installers into Haskell. So I didn't
need to learn a new language, yay! And could pull in all my helpful
Haskell utility libraries in the program that builds the installer.

The only tricky part was: How to get git-annex onto PATH? The standard way
to do this seems to be to use a multiple-hundred line include file. Of
course, that file does not have any declared license..  Instead of that,
I used a hack. The git installer for Windows adds itself to PATH, and is
a pre-requisite for git-annex. So the git-annex installer just installs
it into the same directory as git.

So.. I'll be including this first stage Windows port, with installer in
the next release. Anyone want to run a Windows autobuilder?


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlJ2utMQgMEYAOs3Dfc6eZRyUzt4acNXUU"
 nickname="David"
 subject="git on windows"
 date="2013-05-15T04:26:48Z"
 content="""
Adding git to the PATH variable is an optional step in the msysgit installer, fyi.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 2"
 date="2013-05-15T22:30:05Z"
 content="""
I was thinking maybe migrating away from gitbuilder to using something like buildbot as it seems to be a better fit for this project than gitbuilder.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 3"
 date="2013-05-15T22:56:27Z"
 content="""
@Jimmy, would be fine with me if it works better
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 4"
 date="2013-05-16T17:31:40Z"
 content="""
joey, I'll poke at it on the weekend, been meaning to look for an excuse to learn a bit more about buildbot.
"""]]


code:
Laid some groundwork for porting the test suite to Windows, and getting it
working in direct mode. That's not complete, but even starting to run the
test suite in direct mode and looking at all the failures (many of them
benign, like files not being symlinks) highlighted something
I have been meaning to look into for quite a while: Why, in direct mode,
`git-annex` doesn't operate on data staged in the index, but requires you
commit changes to files before it'll see them. That's an annoying
difference between direct and indirect modes.

It turned out that I introduced this behavior back on
[[January 5th|day_163__free_features]], working around a nasty
bug I didn't understand. Bad Joey, should have root caused the bug at the
time! But the commit says I was stuck on it for hours, and it was
presenting as if it was a bug in `git cat-file` itself, so ok. Anyway,
I quickly got to the bottom of it today, fixed the underlying bug (which
was in git-annex, not git itself), and got rid of the workaround and its
undesired consequences. Much better.

The test suite is turning up some other minor problems with direct mode.
Should have found time to port it earlier.

Also, may have fixed the issue that was preventing GTalk from working on
Android. (Missing DNS library so it didn't do SRV lookups right.)


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkF8_uQjLYm5Mf5F_JuVW-BxlvzpWjvR_o"
 nickname="Andrew"
 subject="Android success"
 date="2013-05-16T04:14:09Z"
 content="""
I now have my android phone and laptop talking with GTalk and used a remote ssh server to synchronise files.

Photos are flowing from my phone to my laptop quite happily. Magic.

Would help to exclude .thumbnails from the DCIM directory on the phone though...
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmubB1Sj2rwFoVdZYvGV0ACaQUJQyiJXJI"
 nickname="Paul"
 subject="GTalk/XMPP"
 date="2013-05-19T09:58:51Z"
 content="""
Google seems to be deprecating XMPP and GTalk. Can git-annex assistant use other XMPP services? And does it handle federation correctly?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 3"
 date="2013-05-19T19:12:59Z"
 content="""
git-annex can use any xmpp server.  The only thing that would need to be changed if google talk went away tomorrow is changing some text that suggests entering one's gmail address, to some other text that suggests some other xmpp server to use.

However, I am skeptical about echo chamber conclusions (and/or tech press) about what google is, or is not doing, or is or is not planning to do.
"""]]


code:
Made a release.

I am firming up some ideas for post-kickstarter. More on that later.

In the process of setting up a Windows autobuilder, using the same
jenkins installation that is used to autobuild msysgit.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmubB1Sj2rwFoVdZYvGV0ACaQUJQyiJXJI"
 nickname="Paul"
 subject="Continued work"
 date="2013-05-17T07:44:38Z"
 content="""
Can we continue to fund your work in some way?
"""]]


code:
git-annex is now autobuilt for Windows on the same Jenkins farm that
builds msysgit. Thanks for Yury V. Zaytsev for providing that! Spent about
half of today setting up the build.

Got the test suite to pass in direct mode, and indeed in direct mode
on a FAT file system. Had to fix one corner case in direct mode `git annex
add`. Unfortunately it still doesn't work on Android; somehow `git clone`
of a local repository is broken there. Also got the test suite to build,
and run on Windows, though it fails pretty miserably.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkDb5BRI8VzliQuKRVM7NwMdudvxiUNPHI"
 nickname="Richard"
 subject="git annex add on Win7 NTFS"
 date="2013-06-14T09:26:24Z"
 content="""
Hello I took latest build from download #243.

running in CMD or in GIT bash as admin user.

git annex add, fails on all files.

Example:
add IS Breast/Changing IP Address of The Server/flowplayer-3.2.4.min.js (checksum...)
git-annex: C:\development storage\.git\annex\objects\e4b\168\SHA256E-s15723--f6950
bd995674741c600d0465a333f5491d5713e8ac2e3fc57d61cccadba522d.min.js\SHA256E-s15723--f6950bd995674741c600d0465a333f5491d57
13e8ac2e3fc57d61cccadba522d.min.js.cache: openFile: does not exist (No such file or directory)

Am I doing something wrong? I somehow understood that this should be ok now.
Thanks for your kind answer
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.193"
 subject="comment 2"
 date="2013-06-25T17:12:17Z"
 content="""
That is the same problem that is currently making the test suite fail on the windows autobuilder.

I don't know what is causing this problem, and have not been able to reproduce it locally in order to debug it.
"""]]


code:
Today I had to change the implementation of the Annex monad. The old one
turned out to be buggy around exception handling -- changes to state
recorded by code that ran in an exception handler were discarded when it
threw an exception. Changed from a StateT monad to a ReaderT with
a MVar. Really deep-level change, but it went off without a
hitch!

Other than that it was a bug catch up day. Almost entirely caught up once
more.


code:
Worked on several important bug fixes today. One affects automatic merge
confict resolution, and can case data loss in direct mode, so I will be
making a release with the fix tomorrow.

Practiced TDD today, and good thing too. The new improved test suite
turned up a really subtle bug involving the git-annex branch vector
clocks-ish code, which I also fixed.

Also, fixes to the OSX autobuilds. One of them had a broken gpg, which is
now fixed. The other one is successfully building again. And, I'm switching
the Linux autobuilds to build against Debian stable, since testing has a
new version of libc now, which would make the autobuilds not work on older
systems. Getting an amd64 chroot into shape is needing rather a lot
of backporting of build dependencies, which I already did for i386.


code:
My laptop's SSD died this morning. I had some work from yesterday
committed to the git repo on it, but not pushed as it didn't build.
Luckily I was able to get that off the SSD, which is now a read-only
drive -- even mounting it fails with fsck write errors.

Wish I'd realized the SSD was dying before the day before my trip to
Nicaragua..
Getting back to a useful laptop used most of my time and energy today.

I did manage to fix transfers to not block the rest of the assistant's
threads. Problem was that, without Haskell's threaded runtime, waiting
on something like a rsync command blocks all threads. To fix this,
transfers now are run in separate processes.

Also added code to allow multiple transfers to run at once. Each transfer
takes up a slot, with the number of free slots tracked by a `QSemN`.
This allows the transfer starting thread to block until a slot frees up,
and then run the transfer.

This needs to be extended to be aware of transfers initiated by remotes.
The transfer watcher thread should detect those starting and stopping
and update the `QSemN` accordingly. It would also be nice if transfers
initiated by remotes would be delayed when there are no free slots for them
... but I have not thought of a good way to do that.

There's a bug somewhere in the new transfer code, when two transfers are
queued close together, the second one is lost and doesn't happen.
Would debug this, but I'm spent for the day.


code:
Got the bugfix release out.

Tobias contributed [[tips/megaannex]], which allows using mega.co.nz as a
special remote. Someone should do this with Flickr, using
[filr](https://github.com/ricardobeat/filr). I have improved the
[[special_remotes/hook]] special remote to make it easier to create
and use reusable programs like megaannex.

But, I am too busy rewriting lots of the XMPP code to join in the
special remote fun. Spent all last night staring at protocol traces and
tests, and came to the conclusion that it's working well at the basic
communication level, but there are a lot of bugs above that level. This
mostly shows up as one side refusing to push changes made to its tree,
although it will happily merge in changes sent from the other side.

The NetMessanger code, which handles routing messages to git commands and
queuing other messages, seems to be just wrong. This is code I wrote in the
fall, and have basically not touched since. And it shows. Spent 4 hours
this morning rewriting it. Went all Erlang and implemented message inboxes
using STM. I'm much more confident it won't drop messages on the
floor, which the old code certainly did do sometimes.

Added a check to avoid unnecessary pushes over XMPP. Unfortunately, this
required changing the protocol in a way that will make previous versions of
git-annex refuse to accept any pushes advertised by this version. Could not
find a way around that, but there were so many unnecessary pushes happening
(and possibly contributing to other problems) that it seemed worth the
upgrade pain.

Will be beating on XMPP a bit more. There is one problem I was seeing
last night that I cannot reproduce now. It may have been masked or even
fixed by these changes, but I need to verify that, or put in a workaround.
It seemed that sometimes this code in `runPush` would run the setup
and the action, but either the action blocked forever, or an exception
got through and caused the cleanup not to be run.

[[!format haskell """
        r <- E.bracket_ setup cleanup <~> a
"""]]


code:
Tobias has been busy again today, creating a [[/tips/flickrannex]]
special remote! Meanwhile, I'm thinking about providing a
[[more complete interface|/todo/support_for_writing_external_special_remotes]]
so that special remote programs not written in Haskell can do some of the
things the hook special remote's simplicity doesn't allow.

Finally realized last night that the main problem with the XMPP push code
was an inversion of control. Reworked it so now there are two new threads,
XMPPSendpack and XMPPReceivePack, each with their own queue of push
initiation requests, that run the pushes. This is a lot easier to
understand, probably less buggy, and lets it apply some smarts to squash
duplicate actions and pick the best request to handle next.

Also made the XMPP client send pings to detect when it has been disconnected
from the server. Currently every 120 seconds, though that may change. Testing
showed that without this, it did not notice (for at least 20 minutes) when
it lost routing to the server. Not sure why -- I'd think the TCP connections
should break and this throw an error -- but this will also handle any idle
disconnection problems that some XMPP servers might have.

While writing that, I found myself writing this jem using
[async](http://hackage.haskell.org/package/async), which has a comment
much longer than the code, but basically we get 4 threads that are all
linked, so when any dies, all do.

[[!format haskell """
pinger `concurrently` sender `concurrently` receiver
"""]]

Anyway, I need to run some long-running XMPP push tests to see if I've
really ironed out all the bugs.


code:
The Android app should work on some more devices now, where hard linking to
busybox didn't work. Now it installs itself using symlinks.

Pushed a point release so `cabal install git-annex` works again. And,
I'm really happy to see that the 4.20130521 release has autobuilt on all
Debian architectures, and will soon be replacing the old 3.20120629 version
in testing. (Well, once a libffi transition completes..)

TobiasTheMachine has done it again: [[tips/googledriveannex]]

-----

I spent most of today building a fuzz tester for the assistant. `git annex
fuzztest` will (once you find the special runes to allow it to run) create
random files in the repository, move them around, delete them, move
directory trees around, etc. The plan is to use this to run some long
duration tests with eg, XMPP, to make sure the assistant keeps things
in shape after a lot of activity. It logs in machine-readable format,
so if it turns up a bug I may even be able to use it to reproduce the same
bug (fingers crossed).

I was able to use QuickCheck to generate random data for some parts of the fuzz
tester. (Though the actual file names it uses are not generated using
QuickCheck.) Liked this part:

[[!format haskell """
instance Arbitrary FuzzAction where
        arbitrary = frequency
                [ (100, FuzzAdd <$> arbitrary)
                , (10, FuzzDelete <$> arbitrary)
                , (10, FuzzMove <$> arbitrary <*> arbitrary)
                , (10, FuzzModify <$> arbitrary)
                , (10, FuzzDeleteDir <$> arbitrary)
                , (10, FuzzMoveDir <$> arbitrary <*> arbitrary)
                , (10, FuzzPause <$> arbitrary)
                ]
"""]]


code:
Got caught up on some bug reports yesterday. The main one was odd behavior
of the assistant when the repository was in manual mode. A recent change to
the preferred content expression caused it. But the expression was not
broken. The problem was in the parser, which got the parentheses wrong
in this case. I had to mostly rewrite the parser, unfortunately. I've tested
the new one fairly extensively -- on the other hand this bug lurked in the
old parser for several years (this same code is used for matching files
with command-line parameters).

Just as I finished with that, I noticed another bug. Turns out git-cat-file
doesn't reload the index after it's started. So last week's changes to make
git-annex check the state of files in the index won't work when using the
assistant. Luckily there was an easy workaround for this.

Today I finished up some robustness fixes, and added to the test suite
checks for preferred content expressions, manual mode, etc.

I've started a stress test, syncing 2 repositories over XMPP, with the fuzz
tester running in each to create lots of changes to keep in sync.


code:
Fuzz tester has found several interesting bugs that I've now fixed. It's
even found a bug in my fixes. Most of the problems the fuzz testing has
found have had to do with direct mode merges, and automatic merge conflict
resoltion. Turns out the second level of automatic merge conflict
resolution (where the changes made to resolve a merge conflict themselves
turn out conflict in a later merge) was buggy, for example.

So, didn't really work a lot today -- was not intending to work at all
actually -- but have still accomplished a lot.

(Also, Tobias contributed [[tips/dropboxannex]] .. I'll be curious to see
what the use case for that is, if any!)


code:
The fuzz testing found a file descriptor leak in the XMPP git push code.
The assistant seems to hold up under fuzzing for quite a while now.

Have started trying to work around some versions of Android not letting
the `am` command be used by regular users to open a web browser on an URL.
Here is my current crazy plan: Hack the terminal emulator's title setting
code, to get a new escape sequence that requests an URL be opened. This
assumes I can just use `startActivity()` from inside the app and it will
work. This may sound a little weird, but it avoids me needing to set up a
new communications channel from the assistant to the Java app. Best of all,
I have to write very little Java code. I last wrote Java code in 1995, so
writing much more is probably a good thing to avoid.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkGCmVc5qIJaQQgG82Hc5zzBdAVdhe2JEM"
 nickname="Bruno"
 subject="comment 1"
 date="2013-05-28T03:23:50Z"
 content="""
Does the Android application use a [WakeLock](https://developer.android.com/reference/android/os/PowerManager.WakeLock.html) to ensure syncing happens even if the device screen goes off?

I'm under the impression that it doesn't since it took a very long time to sync the first time (before I used the terminal's options to prevent the phone from sleeping).

I think that option shouldn't be left on since it would waste the battery and I think the application should block sleep mode only when syncing.

I might be wrong. I never had to use a WakeLock on Android yet.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-05-29T16:30:20Z"
 content="""
The terminal has options to take the wake lock, and also the wifi lock, in its menu.

Currently this has to be managed manually. The assistant does not try to deal with this.
"""]]


code:
Yeah, Java hacking today. I have something that I think should deal with
the [[bugs/Android_app_permission_denial_on_startup]] problem. Added a "Open
WebApp" item to the terminal's menu, which should behave as advertised.
This is available in the Android daily build now, if your device has that
problem.

I was not able to get the escape sequence hack to work. I had no difficulty
modifying the terminal to send an intent to open an url when it received a
custom escape sequence. But sending the intent just seemed to lock up the
terminal for a minute without doing anything. No idea why. I had to propigate a
context object in to the terminal emulator through several layers of objects.
Perhaps that doesn't really work despite what I [read on stackoverflow](http://stackoverflow.com/questions/9051849/opening-a-link-in-the-browser/9052208#9052208).

Anyway, that's all I have time to do. It would be nice if I, or some other
interested developer who is more comfortable with Java, could write a custom
Android frontend app, that embedded a web browser widget for the webapp,
rather than abusing the terminal this way. OTOH, this way does provide the
bonus of a pretty good terminal and git shell environment for Android to go
with git-annex.


code:
Winding down work for now, as I prepare for a week at the beach starting in
2 days. That will be followed by a talk about git-annex at
[SELF2013](http://www.southeastlinuxfest.org/) in Charlotte NC on June 9th.

Bits & pieces today.

Want to get a release out RSN, but I'm waiting for the previous release
to finally reach Debian testing, which should happen on Saturday. Luckily I
hear the beach house has wifi, so I will probably end up cutting the
release from there. Only other thing I might work on next week is updating
to yesod 1.2.


code:
Landed two final changes before the release..

First, made git-annex detect if any of the several long-running git process
it talks to have died, and, if yes, restart them. My stress test is reliably
able to get at least `git cat-file` to crash, and while I don't know why (and
obviously should follow up by getting a core dump and stack trace of it),
the assistant needs to deal with this to be robust.

Secondly, wrote rather a lot of Java code to better open the web browser
when the Android app is started. A thread listens for URLs to be written to
a FIFO. Creating a FIFO from fortran^Wjava code is .. interesting. Glad to
see the back of the `am` command; it did me no favors.

AFK


code:
Spent most of the day making file content transfers robust. There were lots
of bugs, hopefully I've fixed most of them. It seems to work well now,
even when I throw a lot of files at it.

One of the changes also sped up transfers; it no longer roundtrips to the
remote to verify it has a file. The idea here is that when the assistant is
running, repos should typically be fairly tightly synced to their remotes
by it, so some of the extra checks that the `move` command does are
unnecessary.

Also spent some time trying to use ghc's threaded runtime, but continue to
be baffled by the random hangs when using it. This needs fixing eventually;
all the assistant's threads can potentially be blocked when it's waiting on
an external command it has run.

Also changed how transfer info files are locked. The lock file is now
separate from the info file, which allows the TransferWatcher thread to
notice when an info file is created, and thus actually track transfers
initiated by remotes.

---

I'm fairly close now to merging the `assistant` branch into `master`.
The data syncing code is very brute-force, but it will work well enough
for a first cut.

Next I can either add some repository network mapping, and use graph
analysis to reduce the number of data transfers, or I can move on to the
[[webapp]]. Not sure yet which I'll do. It's likely that since DebConf
begins tomorrow I'll put off either of those big things until after the
conference.


code:
I didn't plan to work on git-annex much while at DebConf, because the conference
always prevents the kind of concentration I need. But I unexpectedly also had to deal
with [three dead drives](http://joeyh.name/blog/entry/I_am_become_Joey_destroyer_of_drives/)
and illness this week.

That said, I have been trying to debug a problem with git-annex and Haskell's threaded
runtime all week. It just hangs, randomly. No luck so far isolating why, although I now
have a branch that hangs fairly reliably, and in which I am trying to whittle the entire
git-annex code base (all 18 thousand lines!) into a nice test case.

This threaded runtime problem doesn't affect the assistant yet, but if I want to use
Yesod in developing the webapp, I'll need the threaded runtime, and using the threaded
runtime in the assistant generally would make it more responsive and less hacky.

Since this is a task I can work on without much concentration, I'll probably keep beating
on it until I return home. Then I need to spend some quality thinking time on where
to go next in the assistant.


code:
Today marks 1 year since I started working on the git-annex assistant. 280
solid days of work!

As a background task here at the beach I've been porting git-annex to yesod
1.2. Finished it today, earlier than expected, and also managed to keep it
building with older versions. Some tricks kept the number of
ifdefs reasonably low.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkGCmVc5qIJaQQgG82Hc5zzBdAVdhe2JEM"
 nickname="Bruno"
 subject="comment 1"
 date="2013-06-04T12:51:20Z"
 content="""
Congrats on the first year Joey!
"""]]