code:
[[!meta title="debian system runs assistant in tight loop, rather than using select"]]

### Please describe the problem.

After running for a while, the assistant maxes my CPU. I have evidence
here that it is just sitting there, doing apparently nothing (transfers
are finished), eating precious CPU.

Nowadays, I do not let the assistant run on its own, otherwise it makes
my office noisy because the CPU fan starts.

This makes the assistant and the webapp basically unuseable for me:
if i want to use its feature, I must remember to shut down the daemon
every time I am done with it, which seems contrary to the spirit of
the project. :)

### What steps will reproduce the problem?

It's a little unclear. There was a peculiar crash here (power cable
yanked out) during a `git annex copy`, which *was* eating up all CPU,
but mostly through I/O.

That is the purple block in this graph:

![CPU usage during the last 24 hours][cpu-day.png]

Power was yanked out about right after that purple block was gone:

![Uptime in the last 24 hours][uptime-day.png]

Files were copied to an external drive at the time of the crash,
which explains the high iowait. The last lines in daemon.log are:

[[!format txt """
sent 184184937 bytes  received 31 bytes  13643330.96 bytes/sec
total size is 184162304  speedup is 1.00
[2013-12-31 23:56:47 EST] Transferrer: Uploaded American ..Queen.avi
[2013-12-31 23:56:47 EST] Pusher: Syncing with backup
To /backup/srv/video
   5155be4..7d904d5  git-annex -> synced/git-annex
"""]]

And daemon.status:

[[!format txt """
lastRunning:1388569856.366405s
scanComplete:True
sanityCheckRunning:False
lastSanityCheck:1387169250.801117s
"""]]

When the machine rebooted, I logged in, and resumed the transfer using
`git annex copy`. The assistant was also started automatically during
my session, but I wasn't aware of that until this morning (7AM) when i
came back to the terminal and noticed how noisy it was.

At that point i did `killall git-annex` and CPU usage returned to
normal, as the frequency of the CPU:

![CPU frequency scaling in the last 24 hours][cpuspeed-day.png]

I am trying to see if I can reproduce this problem reliably - I
just started the assistant again now, and while the CPU usage is
already high, I will forgive it for now because it's doing its
initial scans... Let's see what happens in the afternoon! --[[anarcat]]

Update: I seem to be able to reproduce. I just started the webapp and
the scans were performed. They are now finished, yet the CPU usage is
still high:

[[!format txt """
top - 14:10:20 up 1 day,  1:16,  7 users,  load average: 5,51, 3,61, 2,53
Tasks: 292 total,   2 running, 288 sleeping,   0 stopped,   2 zombie
%Cpu(s): 60,9 us,  7,8 sy,  0,6 ni, 29,5 id,  0,8 wa,  0,0 hi,  0,5 si,  0,0 st
KiB Mem:   8162960 total,  6943100 used,  1219860 free,  1065464 buffers
KiB Swap:  4022268 total,    15452 used,  4006816 free,  1983252 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
18364 anarcat   20   0  152m  27m  18m S  53,7  0,3   0:35.61 git-annex
 8388 anarcat   20   0  192m  54m  21m S  47,8  0,7  12:55.19 git-annex
18671 anarcat   20   0  157m  33m  18m S  29,9  0,4   0:22.65 git-annex
"""]]

An strace on 18364 yields:

[[!format txt """
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 537245076}) = 0
[pid 18364] <... futex resumed> )       = ? ERESTARTSYS (To be restarted)
[pid 18364] --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
[pid 18364] rt_sigreturn(0x1a)          = 202
[pid 18364] futex(0x3de703c, FUTEX_WAIT_PRIVATE, 51, NULL <unfinished ...>
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 256339453}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 537670797}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 256748972}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 538079838}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 257202502}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 538533443}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 257611085}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 538941839}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 258018398}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 539349100}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 258433069}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 539763939}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 258863864}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 540195885}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 259295204}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 540626505}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 259729523}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 541060751}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 260131616}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 541463041}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 260539761}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 541871201}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 260970889}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 542302047}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 261378036}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 542709166}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 261780507}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 543111819}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 262194032}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 543525143}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 262610040}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 543941528}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 263042272}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 544374016}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 263532359}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 544864814}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18364] <... futex resumed> )       = ? ERESTARTSYS (To be restarted)
[pid 18364] --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
[pid 18364] rt_sigreturn(0x1a)          = 202
[pid 18364] futex(0x3de703c, FUTEX_WAIT_PRIVATE, 51, NULL <unfinished ...>
"""]]

 [cpu-day.png]: http://paste.anarc.at/cpu-day.png
 [uptime-day.png]: http://paste.anarc.at/uptime-day.png
 [cpuspeed-day.png]: http://paste.anarc.at/cpuspeed-day.png

### What version of git-annex are you using? On what operating system?

Debian wheezy.

[[!format sh """
anarcat@marcos:~$ git annex version
git-annex version: 5.20131109-gf2cb5b9
build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav glacier hook
"""]]

### Please provide any additional information below.

This *may* be related to [[bugs/webapp_takes_100__37___of_the_cpu/]],
but I didn't observe both problems simultaneously so far - ie. the
webapp doesn't need to run for this bug to happen.

Here's the output of top:

[[!format sh """
top - 04:55:01 up 16:01, 10 users,  load average: 3,96, 3,72, 3,59
Tasks: 293 total,   6 running, 284 sleeping,   0 stopped,   3 zombie
%Cpu(s): 87,4 us, 10,9 sy,  0,6 ni,  0,1 id,  0,3 wa,  0,0 hi,  0,7 si,  0,0 st
KiB Mem:   8162960 total,  6847120 used,  1315840 free,  1084756 buffers
KiB Swap:  4022268 total,     1320 used,  4020948 free,  2041108 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
 8099 anarcat   20   0  144m  28m  18m S  34,0  0,4 460:10.96 git-annex
 6525 anarcat   20   0  436m 295m  18m S  28,4  3,7 448:50.71 git-annex
11146 munin     20   0 25376 6540 1896 R  28,4  0,1   0:00.05 munin-update
 8045 anarcat   20   0  165m  41m  18m S  22,7  0,5 455:14.41 git-annex
11142 root      20   0 32316 7356 2016 R  22,7  0,1   0:00.05 ikisite
11144 aegir     20   0  161m  16m 6608 R  22,7  0,2   0:00.05 php
 2649 liquidso  20   0  222m  54m 5516 S   5,7  0,7  40:40.61 liquidsoap
11149 anarcat   20   0 24448 1552 1072 R   5,7  0,0   0:00.01 top
    1 root      20   0 10644  784  652 S   0,0  0,0   0:01.10 init
    2 root      20   0     0    0    0 S   0,0  0,0   0:00.01 kthreadd
    3 root      20   0     0    0    0 S   0,0  0,0   2:10.09 ksoftirqd/0
    6 root      rt   0     0    0    0 S   0,0  0,0   0:00.22 migration/0
    7 root      rt   0     0    0    0 S   0,0  0,0   0:00.59 watchdog/0
    8 root      rt   0     0    0    0 S   0,0  0,0   0:00.25 migration/1
   10 root      20   0     0    0    0 S   0,0  0,0   2:11.10 ksoftirqd/1
   12 root      rt   0     0    0    0 S   0,0  0,0   0:00.17 watchdog/1
   13 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 cpuset
   14 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 khelper
   15 root      20   0     0    0    0 S   0,0  0,0   0:00.00 kdevtmpfs
   16 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 netns
   17 root      20   0     0    0    0 S   0,0  0,0   0:00.17 sync_supers
   18 root      20   0     0    0    0 S   0,0  0,0   0:00.00 bdi-default
   19 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kintegrityd
   20 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kblockd
   22 root      20   0     0    0    0 S   0,0  0,0   0:00.06 khungtaskd
   23 root      20   0     0    0    0 S   0,0  0,0   3:04.23 kswapd0
   24 root      25   5     0    0    0 S   0,0  0,0   0:00.00 ksmd
   25 root      39  19     0    0    0 S   0,0  0,0   0:00.00 khugepaged
   26 root      20   0     0    0    0 S   0,0  0,0   0:00.00 fsnotify_mark
   27 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 crypto
  113 root      20   0     0    0    0 S   0,0  0,0   0:00.01 khubd
  133 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 ata_sff
  153 root      20   0     0    0    0 S   0,0  0,0   0:05.68 scsi_eh_0
  158 root      20   0     0    0    0 S   0,0  0,0   0:00.00 scsi_eh_1
  160 root      20   0     0    0    0 S   0,0  0,0   0:00.09 kworker/u:2
  161 root      20   0     0    0    0 S   0,0  0,0   0:00.00 scsi_eh_2
  162 root      20   0     0    0    0 S   0,0  0,0   0:00.00 scsi_eh_3
  163 root      20   0     0    0    0 S   0,0  0,0   0:01.43 kworker/u:3
  294 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kdmflush
  295 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kcryptd_io
  296 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kcryptd
  305 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kdmflush
  307 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kdmflush
  311 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kdmflush
  313 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kdmflush
  341 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kdmflush
  342 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kcryptd_io
  343 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kcryptd
  352 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kdmflush
  354 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kdmflush
  356 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kdmflush
  358 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kdmflush
  361 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kdmflush
  400 root      20   0     0    0    0 S   0,0  0,0   0:00.00 jbd2/dm-1-8
  401 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 ext4-dio-unwrit
  468 root      20   0     0    0    0 S   0,0  0,0   0:02.19 kworker/1:2
  538 root      20   0 21804 1804  796 S   0,0  0,0   0:00.10 udevd
  705 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kpsmoused
  878 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 hd-audio0
  890 postfix   20   0 44216 2460 1912 S   0,0  0,0   0:00.00 pickup
  939 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kvm-irqfd-clean
 1150 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 firewire
 1719 root      20   0     0    0    0 S   0,0  0,0   0:03.40 flush-254:3
 1786 root      20   0     0    0    0 S   0,0  0,0   0:00.17 jbd2/dm-2-8
 1787 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 ext4-dio-unwrit
 1788 root      20   0     0    0    0 S   0,0  0,0   0:06.60 jbd2/dm-3-8
 1789 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 ext4-dio-unwrit
 1790 root      20   0     0    0    0 S   0,0  0,0   0:02.56 jbd2/dm-4-8
 1791 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 ext4-dio-unwrit
 1792 root      20   0     0    0    0 S   0,0  0,0   0:01.19 jbd2/dm-8-8
 1793 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 ext4-dio-unwrit
 2004 anarcat   20   0     0    0    0 Z   0,0  0,0   0:17.57 chromium
 2379 root      20   0  115m 1796  928 S   0,0  0,0   0:02.16 rsyslogd
 2435 icecast2  20   0  148m 4440 2780 S   0,0  0,1   0:56.15 icecast2
 2488 root      20   0  4112  644  500 S   0,0  0,0   0:00.66 acpid
 2513 daemon    20   0 16668  156    0 S   0,0  0,0   0:00.00 atd
 2536 proxy     20   0  6884  344  196 S   0,0  0,0   0:00.00 polipo
 2573 messageb  20   0 31488 2548  884 S   0,0  0,0   0:00.79 dbus-daemon
 2626 bind      20   0  111m  15m 2276 S   0,0  0,2   0:00.46 named
 2694 root      20   0 82308 3112 2520 S   0,0  0,0   0:00.03 gdm3
 2729 root      20   0 99,3m 4652 3608 S   0,0  0,1   0:00.02 gdm-simple-slav
 2731 root      20   0  178m  27m 9908 S   0,0  0,3  40:30.80 Xorg
 2738 root      20   0  282m  14m 6576 S   0,0  0,2   0:02.98 apache2
 2763 root      20   0     0    0    0 S   0,0  0,0   0:00.92 flush-254:4
 2779 root      20   0 21000 1164  904 S   0,0  0,0   0:00.00 bluetoothd
 2784 avahi     20   0 34236 1876 1460 S   0,0  0,0   0:00.15 avahi-daemon
 2786 avahi     20   0 34032  440  184 S   0,0  0,0   0:00.00 avahi-daemon
 2792 root      10 -10     0    0    0 S   0,0  0,0   0:00.00 krfcommd
 2829 bitlbee   20   0 45216 1136  712 S   0,0  0,0   0:00.00 bitlbee
 2973 root      20   0  136m 3572 2892 S   0,0  0,0   0:00.37 accounts-daemon
 2974 clamav    20   0 45432 2172 1260 S   0,0  0,0   0:26.77 freshclam
 2997 root      20   0  141m 4552 3072 S   0,0  0,1   0:00.21 polkitd
 3021 root      20   0 18876  984  740 S   0,0  0,0   0:00.48 cron
 3062 root      20   0 75948 3436 2304 S   0,0  0,0   0:00.03 cupsd
 3064 colord    20   0  152m 8672 3344 S   0,0  0,1   0:00.10 colord
 3069 root      20   0  126m 4116 2964 S   0,0  0,1   0:00.16 console-kit-dae
 3071 colord    20   0  300m  12m 7932 S   0,0  0,2   0:01.23 colord-sane
 3171 nobody    10 -10 44328 1672 1048 S   0,0  0,0   0:00.88 gpsd
 3178 root      20   0  9204 3508  380 S   0,0  0,0   0:39.47 haveged
 3192 ikiwiki-  20   0 13772  864  728 S   0,0  0,0   0:00.00 git-daemon
 3310 anarcat   20   0 91856  27m 2820 S   0,0  0,3   0:03.30 xterm
 3314 anarcat   20   0 24004 5452 1876 S   0,0  0,1   0:00.54 bash
 3336 root      20   0  152m 4172 3300 S   0,0  0,1   0:00.07 upowerd
 3437 root      20   0     0    0    0 S   0,0  0,0   0:00.00 scsi_eh_6
 3438 root      20   0     0    0    0 S   0,0  0,0   6:52.70 usb-storage
 3470 anarcat   20   0 61648 2516 1912 S   0,0  0,0   0:00.01 gvfs-gphoto2-vo
 3474 anarcat   20   0 78024 2564 2048 S   0,0  0,0   0:02.04 gvfs-afc-volume
 3530 root      20   0 52400  10m 2064 S   0,0  0,1   0:03.34 munin-node
 3599 rtkit     21   1 39632 1344 1112 S   0,0  0,0   0:00.30 rtkit-daemon
 3721 root      20   0  4176  700  560 S   0,0  0,0   0:00.01 mysqld_safe
 3899 anarcat   20   0 85216  21m 2872 S   0,0  0,3   0:00.76 xterm
 3903 anarcat   20   0 23216 4688 1804 S   0,0  0,1   0:00.58 bash
 4108 mysql     30  10  434m 137m 4396 S   0,0  1,7   1:53.14 mysqld
 4109 root      20   0  4084  628  524 S   0,0  0,0   0:00.00 logger
 4209 anarcat   20   0  129m  23m  18m S   0,0  0,3   1:06.16 git-annex
 4226 root      20   0  183m 4888 3504 S   0,0  0,1   0:00.08 gdm-session-wor
 4240 anarcat   20   0 21360 2572 2188 S   0,0  0,0   0:00.34 git
 4431 root      20   0     0    0    0 S   0,0  0,0   0:00.00 kauditd
 4447 anarcat   20   0  4176  664  548 S   0,0  0,0   0:00.03 sh
 4530 ntp       20   0 41052 2284 1644 S   0,0  0,0   0:02.71 ntpd
 4593 postgres  20   0  102m 9008 7680 S   0,0  0,1   0:01.04 postgres
 4667 anarcat   20   0 12384 1072  704 S   0,0  0,0   0:00.11 ssh-agent
 4668 anarcat   20   0 20164  372  156 S   0,0  0,0   0:01.03 gpg-agent
 4671 anarcat   20   0 24184  552  304 S   0,0  0,0   0:00.00 dbus-launch
 4672 anarcat   20   0 30668 1592  736 S   0,0  0,0   0:01.10 dbus-daemon
 4704 anarcat   20   0 49456 2740 2216 S   0,0  0,0   0:00.13 xfconfd
 4711 anarcat   20   0 63740 3124 2312 S   0,0  0,0   0:02.61 xscreensaver
 4713 anarcat   20   0  156m 8232 5856 S   0,0  0,1   0:06.14 xfce4-session
 4714 postgres  20   0  102m 1952  616 S   0,0  0,0   0:06.79 postgres
 4715 postgres  20   0  102m 1704  376 S   0,0  0,0   0:04.95 postgres
 4716 postgres  20   0  102m 3068 1128 S   0,0  0,0   0:01.32 postgres
 4717 postgres  20   0 72884 1800  416 S   0,0  0,0   0:00.99 postgres
 4735 postgrey  20   0 66452  14m 3136 S   0,0  0,2   0:00.40 /usr/sbin/postg
 4753 anarcat   20   0  124m 4060 2916 S   0,0  0,0   0:00.02 xfsettingsd
 4754 anarcat   20   0  166m 6496 5204 S   0,0  0,1   0:00.07 Thunar
 4778 anarcat   20   0 63364 2564 2096 S   0,0  0,0   0:00.03 gvfsd
 4783 anarcat   20   0  305m  14m 9568 S   0,0  0,2   0:52.46 xfce4-panel
 4790 anarcat   20   0  156m 7968 6232 S   0,0  0,1   0:00.23 xfdesktop
 4791 root      20   0 21516  804  496 S   0,0  0,0   0:00.00 dovecot
 4837 anarcat   20   0  148m 4296 2768 S   0,0  0,1   0:11.87 xfce4-settings-
 4856 anarcat   20   0 26620 1248  988 S   0,0  0,0   1:19.60 xcompmgr
 4873 anarcat   20   0  115m 8004 3872 S   0,0  0,1   0:07.42 awesome
 4887 anarcat   20   0  267m  12m 8664 S   0,0  0,2   0:02.51 update-notifier
 4899 anarcat   20   0  183m 5868 4708 S   0,0  0,1   0:00.18 polkit-gnome-au
 4905 anarcat   20   0 56320 4180 2040 S   0,0  0,1   0:00.10 gconfd-2
 4908 anarcat    9 -11  300m 6912 4184 S   0,0  0,1   0:08.00 pulseaudio
 4916 root      20   0  126m 4208 3056 S   0,0  0,1   0:02.46 udisks-daemon
 4917 root      20   0 47408  776  428 S   0,0  0,0   0:06.61 udisks-daemon
 4921 anarcat   20   0  113m 3200 2448 S   0,0  0,0   0:00.00 gconf-helper
 4972 anarcat   20   0  145m 7432 5880 S   0,0  0,1   0:00.09 panel-16-systra
 4974 anarcat   20   0  148m 9328 7348 S   0,0  0,1   2:40.37 xfce4-weather-p
 4975 anarcat   20   0  174m  11m 8312 S   0,0  0,1   0:42.26 xfce4-xkb-plugi
 4978 mpd       20   0  503m  29m 2920 S   0,0  0,4   1:24.97 mpd
 5029 mpd       20   0  256m 6412 4008 S   0,0  0,1   0:11.74 pulseaudio
 5030 dovecot   20   0 13044  768  592 S   0,0  0,0   0:00.00 anvil
 5031 root      20   0 13172  944  628 S   0,0  0,0   0:00.00 log
 5034 root      20   0 23156 2536  768 S   0,0  0,0   0:00.00 config
 5040 mpd       20   0  113m 3192 2448 S   0,0  0,0   0:00.00 gconf-helper
 5234 root      20   0     0    0    0 S   0,0  0,0   0:01.19 kworker/0:1
 5745 root      20   0 37788 2472 1912 S   0,0  0,0   0:00.68 master
 5756 postfix   20   0 44368 2700 2100 S   0,0  0,0   0:00.27 qmgr
 5807 root      20   0 49848 1216  608 S   0,0  0,0   0:00.05 sshd
 5908 debian-t  20   0 56008  21m 2988 S   0,0  0,3   0:21.87 tor
 5928 debian-t  30  10  190m  31m 2560 S   0,0  0,4   7:35.56 transmission-da
 5948 anarcat   20   0 71776 8204 2860 S   0,0  0,1   0:05.76 xterm
 5954 anarcat   20   0 22956 4540 1772 S   0,0  0,1   0:00.10 bash
 6019 spampd    20   0  147m  62m 1844 S   0,0  0,8   0:00.55 spampd
 6049 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 iprt
 6085 root      20   0  3948   84    0 S   0,0  0,0   0:00.00 minissdpd
 6095 root      20   0 17404  908  744 S   0,0  0,0   0:00.00 getty
 6096 root      20   0 17404  908  744 S   0,0  0,0   0:00.00 getty
 6097 root      20   0 17404  904  744 S   0,0  0,0   0:00.00 getty
 6098 root      20   0 17404  896  744 S   0,0  0,0   0:00.00 getty
 6099 root      20   0 17404  900  744 S   0,0  0,0   0:00.00 getty
 6100 root      20   0 17404  904  744 S   0,0  0,0   0:00.00 getty
 6108 anarcat   20   0 24248 1040  816 S   0,0  0,0   0:00.08 screen
 6109 anarcat   20   0 25864 2640  796 S   0,0  0,0   0:05.37 screen
 6110 anarcat   20   0 22868 4220 1624 S   0,0  0,1   0:00.06 bash
 6153 root      20   0     0    0    0 S   0,0  0,0   0:00.00 flush-254:8
 6174 anarcat   20   0 22872 4224 1624 S   0,0  0,1   0:00.07 bash
 6238 anarcat   20   0  103m  12m 5232 S   0,0  0,2   0:20.80 irssi
 6252 anarcat   20   0  212m  40m  13m S   0,0  0,5   1:09.74 emacs23
 6534 anarcat   20   0 22876 4400 1716 S   0,0  0,1   0:00.07 bash
 6598 anarcat   20   0  302m 163m 3864 S   0,0  2,0  17:19.63 offlineimap
 6616 anarcat   20   0 22408 2648 2228 S   0,0  0,0   0:05.04 git
 6662 root      20   0     0    0    0 S   0,0  0,0   0:01.25 kworker/1:1
 6666 root      20   0     0    0    0 S   0,0  0,0   0:00.00 flush-254:2
 6897 anarcat   20   0 1011m 159m  47m S   0,0  2,0   4:54.69 chromium
 6902 postfix   20   0 46744 3396 2400 S   0,0  0,0   0:00.09 tlsmgr
 6903 anarcat   20   0  283m 7952 2788 S   0,0  0,1   0:05.03 chromium
 6904 anarcat   20   0  6240  356  272 S   0,0  0,0   0:00.00 chrome-sandbox
 6905 anarcat   20   0  304m  17m  12m S   0,0  0,2   0:00.02 chromium
 6908 anarcat   20   0  320m 5800  764 S   0,0  0,1   0:00.16 chromium
 6950 anarcat   20   0  955m  31m  16m S   0,0  0,4   0:01.68 chromium
 6955 anarcat   20   0  953m  33m  17m S   0,0  0,4   0:01.25 chromium
 6960 anarcat   20   0  959m  36m  15m S   0,0  0,5   0:08.32 chromium
 6965 anarcat   20   0  957m  34m  14m S   0,0  0,4   0:10.25 chromium
 6970 anarcat   20   0 1029m 114m  17m S   0,0  1,4   0:46.99 chromium
 6975 anarcat   20   0  953m  26m  14m S   0,0  0,3   0:00.53 chromium
 6980 anarcat   20   0  959m  35m  16m S   0,0  0,4   0:14.68 chromium
 6985 anarcat   20   0  963m  45m  17m S   0,0  0,6   0:22.59 chromium
 6990 anarcat   20   0  952m  25m  13m S   0,0  0,3   0:00.70 chromium
 6995 anarcat   20   0  960m  36m  18m S   0,0  0,5   0:27.21 chromium
 7040 anarcat   20   0  979m  69m  22m S   0,0  0,9   0:05.41 chromium
 7055 anarcat   20   0  966m  51m  20m S   0,0  0,6   0:04.88 chromium
 7063 anarcat   20   0  967m  42m  18m S   0,0  0,5   0:04.46 chromium
 7072 anarcat   20   0  967m  49m  19m S   0,0  0,6   0:05.17 chromium
 7078 anarcat   20   0  971m  53m  19m S   0,0  0,7   0:05.68 chromium
 7087 anarcat   20   0  963m  51m  19m S   0,0  0,7   0:04.75 chromium
 7096 anarcat   20   0  961m  48m  18m S   0,0  0,6   0:04.75 chromium
 7107 anarcat   20   0  161m 4440 3252 S   0,0  0,1   0:02.90 gvfs-gdu-volume
 7127 anarcat   20   0  967m  55m  22m S   0,0  0,7   1:07.92 chromium
 7183 anarcat   20   0  912m  55m  20m S   0,0  0,7   0:04.97 chromium
 7213 anarcat   20   0  964m  47m  19m S   0,0  0,6   0:04.69 chromium
 7235 anarcat   20   0  965m  65m  21m S   0,0  0,8   0:05.26 chromium
 7258 anarcat   20   0  961m  57m  20m S   0,0  0,7   0:05.39 chromium
 7285 anarcat   20   0 18652 2484 1000 S   0,0  0,0   0:01.25 git
 7434 anarcat   39  19 18124 1960  964 S   0,0  0,0   0:00.29 git
 7445 www-data  20   0  141m  24m 5620 S   0,0  0,3   0:00.71 munin-cgi-graph
 7452 anarcat   20   0  980m  63m  22m S   0,0  0,8   0:02.50 chromium
 7492 anarcat   20   0  978m  64m  21m S   0,0  0,8   0:03.74 chromium
 7498 anarcat   20   0  978m  61m  21m S   0,0  0,8   0:01.42 chromium
 7518 anarcat   20   0  984m  63m  22m S   0,0  0,8   0:04.38 chromium
 7532 anarcat   20   0  973m  56m  21m S   0,0  0,7   0:02.28 chromium
 7813 root      20   0     0    0    0 S   0,0  0,0   0:01.22 kworker/0:2
 7899 root      20   0     0    0    0 S   0,0  0,0   0:00.96 kworker/1:0
 8058 anarcat   20   0 24488 1460 1104 S   0,0  0,0   0:28.34 git
 8107 anarcat   20   0 17412 1384 1100 S   0,0  0,0   0:00.00 git
 8130 anarcat   20   0 23792 7572 1004 S   0,0  0,1   0:41.63 git
 8236 anarcat   39  19 21152 4940  964 S   0,0  0,1   0:00.71 git
 8405 anarcat   20   0 17368 1364 1108 S   0,0  0,0   0:00.54 git
 8407 anarcat   20   0 17464 1336  964 S   0,0  0,0   0:00.03 git
 8610 anarcat   20   0 78228  14m 2860 S   0,0  0,2   0:00.37 xterm
 8614 anarcat   20   0 23084 4548 1796 S   0,0  0,1   0:00.28 bash
 8689 anarcat   20   0 72992 9224 2860 S   0,0  0,1   0:00.38 xterm
 8693 anarcat   20   0 22876 4380 1696 S   0,0  0,1   0:00.07 bash
 8735 root      20   0 99,3m 4616 3576 S   0,0  0,1   0:00.02 gdm-simple-slav
 8736 root      20   0  142m  15m 6664 S   0,0  0,2   0:00.56 Xorg
 8755 Debian-g  20   0 30196 1184  736 S   0,0  0,0   0:00.01 dbus-daemon
 8756 root      20   0  119m 4548 3432 S   0,0  0,1   0:00.08 gdm-session-wor
 8757 root      20   0 55872 2116 1572 S   0,0  0,0   0:00.00 sudo
 8759 aegir     20   0 20640 2248 1660 S   0,0  0,0   0:00.07 bash
 8769 Debian-g  20   0  266m  10m 7720 S   0,0  0,1   0:00.07 gnome-session
 8773 Debian-g  20   0  502m  24m  10m S   0,0  0,3   0:00.85 gnome-settings-
 8780 Debian-g  20   0 51980 2236 1836 S   0,0  0,0   0:00.00 gvfsd
 8784 Debian-g   9 -11  237m 6044 3644 S   0,0  0,1   0:00.07 pulseaudio
 8785 Debian-g  20   0  113m 3196 2448 S   0,0  0,0   0:00.00 gconf-helper
 8787 Debian-g  20   0 52956 3756 1932 S   0,0  0,0   0:00.06 gconfd-2
 8790 Debian-g  20   0  279m 9124 6632 S   0,0  0,1   0:00.12 metacity
 8795 Debian-g  20   0  171m 9328 7056 S   0,0  0,1   0:00.04 polkit-gnome-au
 8796 Debian-g  20   0  362m  16m  11m S   0,0  0,2   0:00.68 gdm-simple-gree
 8812 root      20   0 72360 3916 2948 S   0,0  0,0   0:00.08 gdm-session-wor
 9129 root      20   0     0    0    0 S   0,0  0,0   0:00.74 kworker/0:0
 9311 www-data  20   0  288m  44m  33m S   0,0  0,6   0:07.55 apache2
10405 www-data  20   0  287m  42m  32m S   0,0  0,5   0:06.75 apache2
10406 www-data  20   0  287m  44m  34m S   0,0  0,6   0:07.02 apache2
10409 www-data  20   0  287m  41m  31m S   0,0  0,5   0:06.51 apache2
10412 www-data  20   0  287m  42m  32m S   0,0  0,5   0:06.82 apache2
10413 www-data  20   0  291m  40m  27m S   0,0  0,5   0:06.52 apache2
10415 www-data  20   0  287m  39m  29m S   0,0  0,5   0:06.67 apache2
10416 www-data  20   0  288m  42m  31m S   0,0  0,5   0:06.28 apache2
10417 www-data  20   0  290m  50m  38m S   0,0  0,6   0:07.43 apache2
10878 root      20   0     0    0    0 S   0,0  0,0   0:00.02 kworker/0:3
11037 anarcat   20   0  4180  584  488 S   0,0  0,0   0:00.00 sh
11038 anarcat   20   0 62760 4964 3244 S   0,0  0,1   0:00.48 notmuch
11040 anarcat   20   0  4180  652  544 S   0,0  0,0   0:00.00 post-new
11133 root      20   0 52576 1468 1052 S   0,0  0,0   0:00.00 cron
11134 root      20   0 52576 1464 1052 S   0,0  0,0   0:00.00 cron
11135 root      20   0 52576 1468 1052 S   0,0  0,0   0:00.00 cron
11138 aegir     20   0  4176  576  484 S   0,0  0,0   0:00.00 sh
11139 root      20   0  4176  580  484 S   0,0  0,0   0:00.00 sh
11141 munin     20   0  4176  576  480 S   0,0  0,0   0:00.00 sh
11143 root      20   0  4240  324  248 S   0,0  0,0   0:00.00 xargs
11145 munin     20   0  4176  576  484 S   0,0  0,0   0:00.00 munin-cron
11166 anarcat   20   0 61364 2988 2468 R   0,0  0,0   0:00.00 notmuch
11167 anarcat   20   0 61364  752  212 R   0,0  0,0   0:00.00 notmuch
12004 www-data  20   0  292m  52m  37m S   0,0  0,7   0:09.05 apache2
12172 anarcat   20   0  4092  580  480 S   0,0  0,0   0:00.00 emacsclient
12465 spampd    20   0  149m  65m 2840 S   0,0  0,8   0:01.41 spampd
12468 spampd    20   0  150m  67m 2840 S   0,0  0,8   0:01.07 spampd
12895 anarcat   20   0  970m  51m  21m S   0,0  0,6   0:01.80 chromium
13761 anarcat   20   0     0    0    0 Z   0,0  0,0   0:00.00 git
16056 root      20   0 21800 1464  452 S   0,0  0,0   0:00.00 udevd
16057 root      20   0 21800 1340  332 S   0,0  0,0   0:00.00 udevd
16655 www-data  20   0  147m 8232  700 S   0,0  0,1   0:00.04 apache2
17006 anarcat   20   0  526m  29m  17m S   0,0  0,4   0:19.95 gmpc
17959 www-data  20   0 70828  24m 2172 S   0,0  0,3   1:22.37 munin-cgi-html
18082 anarcat   20   0 71776 8164 2820 S   0,0  0,1   0:00.12 xterm
18086 anarcat   20   0 23048 4664 1804 S   0,0  0,1   0:00.18 bash
18571 anarcat   20   0  258m  11m 8296 S   0,0  0,1   0:00.11 xfrun4
19692 spampd    20   0  148m  64m 2832 S   0,0  0,8   0:00.63 spampd
25574 anarcat   20   0  952m  31m  15m S   0,0  0,4   0:00.71 chromium
25590 anarcat   20   0     0    0    0 Z   0,0  0,0   0:41.75 chromium
"""]]

And here's the process tree:

[[!format sh """
$ ps -O start xf | grep git-annex
 8610 13:03:21 S ?        00:00:00 xterm -class UXTerm -title uxterm -u8
 8614 13:03:21 S pts/4    00:00:00  \_ bash
12172 13:13:15 S pts/4    00:00:00      \_ emacsclient --alternate-editor=vim -c install.hostmaster.inc
 8099 13:00:20 S ?        07:40:31 /usr/local/bin/git-annex assistant --startdelay=5s
 8107 13:00:20 S ?        00:00:00  \_ git --git-dir=/srv/books/.git --work-tree=/srv/books -c core.bare=false cat-file --batch
 8405 13:00:28 S ?        00:00:00  \_ git --git-dir=/srv/books/.git --work-tree=/srv/books -c core.bare=false cat-file --batch
 8407 13:00:28 S ?        00:00:00  \_ git --git-dir=/srv/books/.git --work-tree=/srv/books -c core.bare=false check-ignore -z --stdin --verbose --non-matching
 8045 13:00:11 S ?        07:35:36 /usr/local/bin/git-annex assistant --startdelay=5s
 8058 13:00:11 S ?        00:00:28  \_ git --git-dir=/srv/mp3/.git --work-tree=/srv/mp3 cat-file --batch
 8130 13:00:21 S ?        00:00:41  \_ git --git-dir=/srv/mp3/.git --work-tree=/srv/mp3 check-ignore -z --stdin --verbose --non-matching
 8236 13:00:24 S ?        00:00:00  \_ git --git-dir=/srv/mp3/.git --work-tree=/srv/mp3 check-attr -z --stdin annex.backend annex.numcopies --
 7107 12:58:40 S ?        00:00:02 /usr/lib/gvfs/gvfs-gdu-volume-monitor
 6525 12:57:25 S ?        07:29:13 /usr/local/bin/git-annex assistant --startdelay=5s
 6616 12:57:34 S ?        00:00:05  \_ git --git-dir=/srv/video/.git --work-tree=/srv/video cat-file --batch
 7285 12:58:45 S ?        00:00:01  \_ git --git-dir=/srv/video/.git --work-tree=/srv/video check-ignore -z --stdin --verbose --non-matching
 7434 12:58:51 S ?        00:00:00  \_ git --git-dir=/srv/video/.git --work-tree=/srv/video check-attr -z --stdin annex.backend annex.numcopies --
 4209 19:30:17 S ?        00:01:06  \_ /usr/local/bin/git-annex transferkeys --readfd 25 --writefd 24
 4240 19:30:17 S ?        00:00:00      \_ git --git-dir=/srv/video/.git --work-tree=/srv/video cat-file --batch
13761 23:56:38 Z ?        00:00:00      \_ [git] <defunct>
 6252 12:56:59 S ?        00:01:09 /usr/bin/emacs23
"""]]

#### This bug is fixed

> [[fixed|done]]. This was a Cronner bug, triggered when you had a
> scheduled fsck job that runs monthly at any time, and the last time it ran was on a day of a
> month > 12. Workaround: Disable scheduled fsck jobs, or change them to
> run on a specific day of the month. Or upgrade. --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.43"
 subject="comment 10"
 date="2014-01-15T21:14:07Z"
 content="""
I was unable to reproduce the problem with a Debian stable VM running in virtualbox.
"""]]

repo_name: kakafarm/git

license: unknown


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.43"
 subject="comment 11"
 date="2014-01-15T21:30:36Z"
 content="""
In case the problem is fixed in a newer ghc than the 7.4.1 used to build the debian backport and the standalone builds, I have made a standalone build with 7.6.3, which is temporarily here: <http://tmp.kitenet.net/git-annex-standalone-i386.tar.gz> ff3e8a3c35301c8a625747f98435fe427be17f47345540f537ff47f6fc97bdb6

(That's a i386 build, which will also run on amd64. That could also be a difference as AFAICS it's only been reported happening with amd64 builds (although I was not able to reproduce the problem on amd64))
"""]]


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="2001:1928:1:9::1"
 subject="comment 12"
 date="2014-01-16T17:04:42Z"
 content="""
Again, I can reproduce with the standalone build, so there's clearly something going on here... Oddly enough, git-annex doesn't appear in top, but I see:

      PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
    21064 anarcat   20   0  111m  23m  14m S  83,0  0,3   9:59.91 ld-linux.so.2
    20829 anarcat   20   0  133m  38m  15m S  76,4  0,5   8:54.88 ld-linux.so.2
    21133 anarcat   20   0  130m  32m  14m S  14,9  0,4   5:00.55 ld-linux.so.2

ld-linux?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.43"
 subject="comment 13"
 date="2014-01-16T17:07:05Z"
 content="""
It's not clear from your comment if you tried my ghc 7.6.3 build.

The hacks used to make the standalone build run everywhere cause ugly process names.
"""]]


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="2001:1928:1:9::1"
 subject="comment 14"
 date="2014-01-16T17:43:33Z"
 content="""
Sorry yes, I tried the 7.6.3 build:

    http://tmp.kitenet.net/git-annex-standalone-i386.tar.gz ff3e8a3c35301c8a625747f98435fe427be17f47345540f537ff47f6fc97bdb6

    $ ps x -o user,pid,start,time,vsz,c,comm | grep ld
    anarcat  20818 11:27:32 00:00:00   6640  0 ld-linux.so.2
    anarcat  20829 11:27:32 00:30:23 216608 40 ld-linux.so.2
    anarcat  20845 11:27:33 00:00:00  13804  0 ld-linux.so.2
    anarcat  20873 11:27:33 00:00:22  13804  0 ld-linux.so.2
    anarcat  20934 11:27:45 00:00:40  12060  0 ld-linux.so.2
    anarcat  20954 11:27:46 00:00:00   9944  0 ld-linux.so.2
    anarcat  20955 11:27:48 00:00:00   9948  0 ld-linux.so.2
    anarcat  21064 11:28:09 00:49:42 114204 67 ld-linux.so.2
    anarcat  21073 11:28:09 00:00:00   6772  0 ld-linux.so.2
    anarcat  21099 11:28:12 00:00:00   6684  0 ld-linux.so.2
    anarcat  21101 11:28:12 00:00:00   6780  0 ld-linux.so.2
    anarcat  21133 11:28:20 00:45:39 128544 62 ld-linux.so.2
    anarcat  21146 11:28:20 00:00:03  11172  0 ld-linux.so.2
    anarcat  21200 11:28:26 00:00:01   7704  0 ld-linux.so.2
    anarcat  21204 11:28:28 00:00:00   7308  0 ld-linux.so.2
    $ /home/anarcat/Downloads/git-annex.linux/git-annex version
    git-annex version: 5.20140115-g7a86704
    build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP Feeds Quvi TDFA CryptoHash
    key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
    remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier hook external
    local repository version: unknown
    supported repository version: 5
    upgrade supported from repository versions: 0 1 2 4

"""]]


code:
[[!comment format=mdwn
 username="stp"
 ip="188.193.209.7"
 subject="Probably confirming this bug"
 date="2014-01-16T19:59:56Z"
 content="""
I am trying around with the assistant and webapp for a few months now.
I have rather large git annex repos and even switched from direct ones to indirect repos to increase performance.

The issue I'm seeing is that when using the assistant or webapp it does the initial startupscan, but only consumes 99-100% of one of my cores and doesn't use significant io. It's stuck in this state for hours (12+) or even days. The only thing standing out in the logs is that it puts a lot of files into the queue. (I suspected a not yet synced backup remote to be the issue) I stopped syncing with it, but that didn't speed up the startupScan at all.

[2014-01-16 19:34:26 CET] TransferScanner: queued Upload NoUUID db/2012-07/1406469990005618750.meta Nothing : expensive scan found missing object
[2014-01-16 19:34:27 CET] TransferScanner: queued Upload NoUUID db/2012-07/1406471293790598760.eml.gz Nothing : expensive scan found missing object

Some additional information:
Total file count: ~100000
Total size: ~500mb
SSD + i7 Quadcore running Ubuntu 13.10
Latest build from 2014-01-16 as standalone built.

Running git annex add && git annex sync from the cli returns within seconds 0.1s (no new files) and 10s (500-1000 new files).
Not understanding the main difference git annex add and git annex assistant (startupScan function) really have.

Thanks for your help.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.43"
 subject="comment 16"
 date="2014-01-17T19:07:10Z"
 content="""
This bug is about cpu usage when git-annex assistant should be sitting idle, not about the statup scan.
"""]]


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="2001:1928:1:9::1"
 subject="comment 17"
 date="2014-01-30T04:57:24Z"
 content="""
@joeyh - any other ideas? I still can't really use the assistant...
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm3ayIrWKe5SqLYomXiCL-l2CDpREvA-IE"
 nickname="myownlittl"
 subject="Seconding this Bug."
 date="2014-02-19T02:04:34Z"
 content="""
Hi Joey, I have also experienced this bug.  I put my \"~/my\" folder under gitannex's management, and I've since had gitannex's assistant consume a CPU (100% of one core) when starting my computer for 1 - 12 hours (at which point I get impatient and kill annex).  If stack traces would help, I can email you those.  Unfortunately, my \"my\" folder includes every important document I've created, coded, or needed, in the last six years.  It has just under 30,000 files, and probably isn't using gitannex the way it's intended to be used.  You can see the distribution of files and file sizes in the tables below.

If gitannex isn't designed for this sort of use case, can you recommend any free software tools that might be able to help opportunistically keep files in sync at this sort of scale, like GA?

Thanks for your time,

Nick

----

[[!table data=\"\"\"
Size<= |      Bytes | Count |
0B     |          0 |  1108 |
1B     |          1 |   264 |
16B    |         16 |   179 |
256B   |        256 |  3414 |
4K     |       4096 | 12771 |
65K    |      65536 |  7402 |
1M     |    1048576 |  2975 |
16M    |   16777216 |   703 |
256M   |  268435456 |   126 |
4G     | 4294967296 |     5 |
\"\"\"]]

    | Size<= |      Bytes | Count |
    | 0B     |          0 |  1108 |
    | 1B     |          1 |   264 |
    | 16B    |         16 |   179 |
    | 256B   |        256 |  3414 |
    | 4K     |       4096 | 12771 |
    | 65K    |      65536 |  7402 |
    | 1M     |    1048576 |  2975 |
    | 16M    |   16777216 |   703 |
    | 256M   |  268435456 |   126 |
    | 4G     | 4294967296 |     5 |
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawntVnR-Z5ghYInvsElbDeADPSuCsF18iTY"
 nickname="Thomas"
 subject="XMPP something to do with it?"
 date="2014-03-02T13:48:22Z"
 content="""
High (between 30 and 100%) processor load in my case seems to be related to lots of these logfile entries:

    [2014-03-02 14:36:54 CET] XMPPClient: sending: Pushing \"s34\" (SendPackOutput 75 \"<elided>\")
    [2014-03-02 14:36:54 CET] XMPPClient: to client: s6/a303a6cfb26b8e08
    [2014-03-02 14:36:54 CET] XMPPClient: sending: Pushing \"s34\" (SendPackOutput 76 \"<elided>\")
    [2014-03-02 14:36:54 CET] XMPPClient: received: [\"Ignorable Presence from s6/941d024438f7cd11 Just (Element {elementName = Name {nameLocalName = \\"git-annex\\", nameNamespace = Just \\"git-annex\\", namePrefix = Nothing}, elementAttributes = [], elementNodes = []})\"]
    [2014-03-02 14:36:54 CET] XMPPClient: to client: s6/a303a6cfb26b8e08
    [2014-03-02 14:36:54 CET] XMPPClient: received: [\"Pushing \\"s34\\" (CanPush (UUID \\"005cc6c3-a4de-44d5-847e-58b5bc4e4c26\\") [Ref \\"d39375b2da76a05c13a8fdc2197a58af0a9812fc\\",Ref \\"5c0108dcad557ab48bfe335e533f13f29777ca7b\\"])\"]
    [2014-03-02 14:36:54 CET] XMPPClient: sending: Pushing \"s34\" (SendPackOutput 77 \"<elided>\")
    [2014-03-02 14:36:54 CET] XMPPClient: to client: s6/a303a6cfb26b8e08
    [2014-03-02 14:36:54 CET] XMPPClient: sending: Pushing \"s34\" (SendPackOutput 78 \"<elided>\")
    [2014-03-02 14:36:54 CET] XMPPClient: to client: s6/a303a6cfb26b8e08
    [2014-03-02 14:36:55 CET] XMPPClient: sending: Pushing \"s34\" (SendPackOutput 79 \"<elided>\")
    [2014-03-02 14:36:55 CET] XMPPClient: to client: s6/a303a6cfb26b8e08
    [2014-03-02 14:36:55 CET] XMPPClient: sending: Pushing \"s34\" (SendPackOutput 80 \"<elided>\")
    [2014-03-02 14:36:55 CET] XMPPClient: to client: s6/a303a6cfb26b8e08
    [2014-03-02 14:36:55 CET] XMPPClient: sending: Pushing \"s34\" (SendPackOutput 81 \"<elided>\")
    [2014-03-02 14:36:55 CET] XMPPClient: to client: s6/a303a6cfb26b8e08
    [2014-03-02 14:36:55 CET] XMPPClient: sending: Pushing \"s34\" (SendPackOutput 82 \"<elided>\")
    [2014-03-02 14:36:55 CET] XMPPClient: to client: s6/a303a6cfb26b8e08
    [2014-03-02 14:36:56 CET] XMPPClient: sending: Pushing \"s34\" (SendPackOutput 83 \"<elided>\")
    [2014-03-02 14:36:56 CET] XMPPClient: to client: s6/a303a6cfb26b8e08
    [2014-03-02 14:36:56 CET] XMPPClient: sending: Pushing \"s34\" (SendPackOutput 84 \"<elided>\")
    [2014-03-02 14:36:56 CET] XMPPClient: to client: s6/a303a6cfb26b8e08
    [2014-03-02 14:36:56 CET] XMPPClient: received: [\"Presence from s6/a303a6cfb26b8e08 Just (Element {elementName = Name {nameLocalName = \\"git-annex\\", nameNamespace = Just \\"git-annex\\", namePrefix = Nothing}, elementAttributes = [(Name {nameLocalName = \\"query\\", nameNamespace = Nothing, namePrefix = Nothing},[ContentText \\"\\"])], elementNodes = []})\",\"QueryPresence\"]
    [2014-03-02 14:36:56 CET] XMPPClient: sending: Pushing \"s34\" (SendPackOutput 85 \"<elided>\")
    [2014-03-02 14:36:56 CET] XMPPClient: received: [\"Pushing \\"s34\\" (CanPush (UUID \\"005cc6c3-a4de-44d5-847e-58b5bc4e4c26\\") [Ref \\"d39375b2da76a05c13a8fdc2197a58af0a9812fc\\",Ref \\"f43c777aed404e1e98dcbf9b6499f4d056b6662a\\"])\"]
    [2014-03-02 14:36:56 CET] XMPPClient: to client: s6/a303a6cfb26b8e08
    [2014-03-02 14:36:56 CET] XMPPClient: sending: Pushing \"s34\" (SendPackOutput 86 \"<elided>\")
    [2014-03-02 14:36:56 CET] XMPPClient: to client: s6/a303a6cfb26b8e08
    [2014-03-02 14:36:57 CET] XMPPClient: sending: Pushing \"s34\" (SendPackOutput 87 \"<elided>\")
    [2014-03-02 14:36:57 CET] XMPPClient: received: [\"Ignorable Presence from s6/941d024438f7cd11 Just (Element {elementName = Name {nameLocalName = \\"git-annex\\", nameNamespace = Just \\"git-annex\\", namePrefix = Nothing}, elementAttributes = [], elementNodes = []})\"]
    [2014-03-02 14:36:57 CET] XMPPClient: to client: s6/a303a6cfb26b8e08
    [2014-03-02 14:36:57 CET] XMPPClient: received: [\"Pushing \\"s34\\" (CanPush (UUID \\"005cc6c3-a4de-44d5-847e-58b5bc4e4c26\\") [Ref \\"d39375b2da76a05c13a8fdc2197a58af0a9812fc\\",Ref \\"f43c777aed404e1e98dcbf9b6499f4d056b6662a\\"])\"]
    [2014-03-02 14:36:57 CET] XMPPClient: sending: Pushing \"s34\" (SendPackOutput 88 \"<elided>\")
    [2014-03-02 14:36:57 CET] XMPPClient: to client: s6/a303a6cfb26b8e08
    [2014-03-02 14:36:57 CET] XMPPClient: sending: Pushing \"s34\" (SendPackOutput 89 \"<elided>\")
    [2014-03-02 14:36:57 CET] XMPPClient: to client: s6/a303a6cfb26b8e08
    [2014-03-02 14:36:57 CET] XMPPClient: sending: Pushing \"s34\" (SendPackOutput 90 \"<elided>\")
    [2014-03-02 14:36:57 CET] XMPPClient: to client: s6/a303a6cfb26b8e08

This is after everything is uploaded to a ssh server (in the cloud, used for transfer) and while another computer is downloading from there (?).

git-annex version: 5.20140227
build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier hook external

Debian testing using jabber.org for xmpp
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 1"
 date="2014-01-01T20:13:04Z"
 content="""
This strongly resembles ghc RTS bugs such as <http://bugs.debian.org/677096> (except that one got fixed, and affected the non-threaded RTS, while this is using the threaded RTS).

The assistant and the webapp are the same process. The only difference is whether a web browser is talking to it. Unless the other bug is regarding the CPU used by the web browser, that other bug is a duplicate of this one.
"""]]


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="2001:1928:1:9::1"
 subject="comment 20"
 date="2014-03-02T16:33:57Z"
 content="""
This is still a problem in Debian jessie and version `5.20140210` from the debian packages.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.146"
 subject="strace or it didn't happen"
 date="2014-03-05T21:02:17Z"
 content="""
This bug is about a particular sequence of system calls. It is not about every possible way that git-annex can use CPU! If you think you have reproduced this bug, you need to include a strace demonstrating the problem, or you're just adding noise.

@Thomas your log shows git-annex doing a git pull over XMPP. This will take CPU, as will any git pull. It is not related to this bug.

@myownlittl the assistant has to scan your repository when it's started to find any files you have modified since the last time it was run. That is not related to this bug. However, I will probably be adding a config knob to disable the startup scan.

@Oliver, you said you had seen this same bug on Ubuntu, but did not show a strace, so I don't know if you are really experiencing the same bug.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.146"
 subject="comment 22"
 date="2014-03-05T21:08:34Z"
 content="""
@anarcat, it seems to me that you need to find a minumal way to reproduce this.

What architecture, what specfic model of CPU?

Can you reproduce it by running the assistant in a new, empty git annex repository? If not, find the difference between that repository and the repository it happens in. If so, it would probably make sense to build git-annex from source with most features disabled to see if one of them causes it, and if not, try to get it down to a minimal test case suitable to being submitted as a GHC bug.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.191"
 subject="comment 23"
 date="2014-04-11T16:32:47Z"
 content="""
<maxy> the clock_gettime(0x2 and clock_gettime(0x3 are consistent with getCurrentTime and getTimeZone of nextTime

So, that strongly points to the Cronner thread, and I doubt this is specific to stable at all.

Please run git-annex vicfg, and paste all the \"schedule\" lines, from a repository that has the problem. That should allow me to reproduce and fix this bug.
"""]]


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="2001:1928:1:9::1"
 subject="comment 2"
 date="2014-01-01T20:29:59Z"
 content="""
Humm... it doesn't look like #677096 is fixed in wheezy, are you sure this isn't the problem I am seeing?

I am considering upgrading this machine to jessie soon, would that help diagnosing the problem?

Regarding the other bug, if you are refering to [[webapp takes 100% of the cpu]], i do believe it's a different issue because the CPU usage is lower in the assistant when it's *not* transfering lots of small files.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 3"
 date="2014-01-01T20:45:19Z"
 content="""
Note that I have installed the backports and autobuilds on systems and run the assistant without seeing this problem, so it may be particular to the system's configuration in some way.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 4"
 date="2014-01-02T19:04:26Z"
 content="""
I've tested with the current release's Linux standalone amd64 build, and once the webapp shows no activity, strace shows it sitting there waiting for a futex, which causes no cpu load at all.
"""]]


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="2001:1928:1:9::1"
 subject="comment 5"
 date="2014-01-02T22:21:34Z"
 content="""
I rebuilt using the current git head, i was expecting 20131230 but I got 5.20140102-gd93f946. The problem is still there:

[[!format txt \"\"\"
anarcat@marcos:mp3$ top -b -n 1 | head
top - 17:18:23 up 2 days,  4:24,  9 users,  load average: 4,01, 3,91, 3,80
Tasks: 295 total,   1 running, 294 sleeping,   0 stopped,   0 zombie
%Cpu(s): 44,0 us,  4,9 sy,  0,6 ni, 49,7 id,  0,6 wa,  0,0 hi,  0,3 si,  0,0 st
KiB Mem:   8162960 total,  6778608 used,  1384352 free,   397144 buffers
KiB Swap:  4022268 total,    45752 used,  3976516 free,  2750720 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
28910 anarcat   20   0  158m  37m  21m S  90,2  0,5  42:49.44 git-annex
29550 anarcat   20   0  173m  44m  21m S  72,2  0,6  36:49.59 git-annex
29673 anarcat   20   0  304m 175m  21m S  12,0  2,2  34:09.70 git-annex
\"\"\"]]

This doesn't seem to be proportional to the repository size or anything. pid 28910 is /srv/books, a moderately small repository. 29550 is my video repository and 29673 is my music.

I suspect this is because I build git-annex myself. I am guessing that the ghc version I am using doesn't build with the proper thread patches and so on. I will try the backport to see if it fixes the issue.
"""]]


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="2001:1928:1:9::1"
 subject="comment 6"
 date="2014-01-02T23:30:38Z"
 content="""
bad news: i can reproduce this with the backport:

Version: 4.20131106~bpo70+1
Build flags: Assistant Webapp Pairing Testsuite S3 Inotify DBus Feeds Quvi TDFA

oddly enough, one of the repository saw some files be committed by the assistant during the startup scan that were not found by the 5.x version I was running earlier.

but processor usage is still maxed out.

i would be happy to provide you a shell here if you send me a OpenPGP-signed SSH public-key (or if you use monkeysphere).
"""]]


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="2001:1928:1:9::1"
 subject="comment 7"
 date="2014-01-09T23:51:02Z"
 content="""
Okay, so the current situation is this: joey can't reproduce this, but I can reliably can, with the backport. So there has to be something different in our setups.

We are suspecting the kernel, so I'm trying to upgrade to the backport, we'll see.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkmBRpgz-elUViGzECtWiO9cRLUpxIuMmg"
 nickname="Oliver"
 subject="comment 8"
 date="2014-01-11T19:31:35Z"
 content="""
I have two Ubuntu 13.10 boxes, both running 5.20131221 from the fmarier PPA.  They both have a repository with my XBMC library, and these repositories are synced together (they run on the same network).  Interestingly, one machine exhibits this behaviour and one doesn't.  The only obvious relevant difference is that the kernel versions are slightly different: the one that behaves properly is running

3.11.0-15-generic #23-Ubuntu SMP Mon Dec 9 18:17:04 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

The one where the assistant is running at full CPU is on

3.11.0-14-generic #21-Ubuntu SMP Tue Nov 12 17:04:55 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

I'm not sure if this is significant or why the kernels are different versions (both machines are fully updated).

I would like to try and help debug this problem if possible.  Please let me know if there is any diagnostic output I can provide that will help.

"""]]


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="2001:1928:1:9::1"
 subject="comment 9"
 date="2014-01-15T04:29:55Z"
 content="""
I have upgraded to the 3.10 backport, and there was no change in this behavior. I still see unreasonable CPU usage.

    Linux marcos 3.10-0.bpo.3-amd64 #1 SMP Debian 3.10.11-1~bpo70+1 (2013-09-24) x86_64 GNU/Linux

"""]]


code:
### Please describe the problem.
Summary: The assistant pulls files from remote repos although they are stored on trusted repos (remote machine may be up or down).

After starting the assistant via "git annex webapp --listen #ip# --verbose --debug" it starts to check for pending tasks and starts to queue lots of files (could be all of them). I checked the daemon.log and found the following line:

[[!format sh """
[2014-06-03 21:39:04 CEST] TransferScanner: queued Download UUID "541d2f88-16c3-11e2-aa7e-8f1a2c8e14c5" apps/Apache_OpenOffice_4.0.0_MacOS_x86_install_en-US.dmg Nothing : expensive scan found missing object
"""]]

The file is available in a remote repo:

[[!format sh """
$ git annex whereis apps/Apache_OpenOffice_4.0.0_MacOS_x86_install_en-US.dmg
whereis apps/Apache_OpenOffice_4.0.0_MacOS_x86_install_en-US.dmg (1 copy)
  	541d2f88-16c3-11e2-aa7e-8f1a2c8e14c5 -- dump
ok
"""]]

The remote "dump" is trusted and is configured as "backup" repo. The repo running the assistant is configured as "manual". The configuration settings for wanted, required and scheduled remained untouched.

Num copies is unset and defaults to 1:

[[!format sh """
$ git annex numcopies
global numcopies is not set
(default is 1)
"""]]

When calling "git annex find --want-get" no files are listed as in my case, all are stored in the "dump" repo which is trusted.

[[!format sh """
$ git annex find --want-get
$
"""]]

### What steps will reproduce the problem?
Start the assistant and wait for it to finish startup checking. The assistant adds lots (all?) of the files due to “expensive scan found missing object”.

### What version of git-annex are you using? On what operating system?
Snapshot build from the annex branchable webpage running on an OSX 10.9.3 machine:
[[!format sh """
$ git annex version
git-annex version: 5.20140411-g2503f43
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV FsEvents XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier hook external
local repository version: 5
supported repository version: 5
upgrade supported from repository versions: 0 1 2 4
"""]]

Note: Updated to 5.20140529-g68a56a6 but same behavior is observed.

### Please provide any additional information below.

The backup node was down while the assistant starts up. However also files from another trusted node that is online 24/7 is pulled. Repo was upgraded from version 4 to 5. Is there a way to further check why a file was added e.g. to dump all variables of the evaluated term (approxlackingcopies, copies, etc)?

[[!format sh """
[2014-06-03 21:39:04 CEST] TransferScanner: starting scan of [Remote { name ="origin" },Remote { name ="dump" }]
[2014-06-03 21:39:04 CEST] read: git ["--git-dir=/Volumes/DATA/annex/images/.git","--work-tree=/Volumes/DATA/annex/images","ls-files","--cached","-z","--"]
[2014-06-03 21:39:04 CEST] chat: git ["--git-dir=/Volumes/DATA/annex/images/.git","--work-tree=/Volumes/DATA/annex/images","check-attr","-z","--stdin","annex.backend","annex.numcopies","--"]
[2014-06-03 21:39:04 CEST] TransferScanner: queued Download UUID "541d2f88-16c3-11e2-aa7e-8f1a2c8e14c5" apps/Apache_OpenOffice_4.0.0_MacOS_x86_install_en-US.dmg Nothing : expensive scan found missing object
[2014-06-03 21:39:04 CEST] Transferrer: Transferring: Download UUID "541d2f88-16c3-11e2-aa7e-8f1a2c8e14c5" apps/Apache_OpenOffice_4.0.0_MacOS_x86_install_en-US.dmg Nothing
[2014-06-03 21:39:04 CEST] TransferScanner: queued Download UUID "541d2f88-16c3-11e2-aa7e-8f1a2c8e14c5" apps/Apache_OpenOffice_4.0.1_MacOS_x86_install_en-US.dmg Nothing : expensive scan found missing object
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 1"
 date="2014-06-12T20:07:30Z"
 content="""
This seems to me to be likely to be some kind of configuration problem. In particular, what is the \"wanted\" setting for the repo running the assistant? If it's not \"standard\" then the fact that it's in the \"manual\" group won't affect the assistant's behavior.

If that's not it, it would probably help to paste the entire vicfg display, so I can replicate the config (or share the repo).
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmXSkgjC_ypUQafVwvHTLsStrkiXH8CfHU"
 nickname="Matthias"
 subject="comment 2"
 date="2014-06-13T18:30:25Z"
 content="""
Thanks for your fast reply, Joey. I appended the shortened config below. Wanted and groupwanted were untouched and therefore still commented out. I removed the hash from wanted to explicitly set it standard. Now the assistant behaves as expected. However I’m wondering:

* Shouldn’t ‘standard’ be the default rule to apply for wanted/groupwanted in case both are not explicitly set in the config? I assumed this to be the case.
* Shouldn’t “find --want-get” list the identical set of files that is also requested by the assistant? Especially this one confused me.

[[!format sh \"\"\"
# Repository trust configuration
# (for macbook)
#trust 3938865a-ce40-4166-a918-2012078846c7 = semitrusted

# Repository groups
# (for macbook)
group 3938865a-ce40-4166-a918-2012078846c7 = manual

# Repository preferred contents
# (for macbook)
#wanted 3938865a-ce40-4166-a918-2012078846c7 = standard

# Group preferred contents
# (Used by repositories with \"groupwanted\" in their preferred contents)
(config defaults - all commented out)

# Standard preferred contents
(config defaults - all commented out)

# Repository required contents
# (for macbook)
#required 3938865a-ce40-4166-a918-2012078846c7 =

# Scheduled activities
# (for macbook)
#schedule 3938865a-ce40-4166-a918-2012078846c7 =
\"\"\"]]
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 3"
 date="2014-06-13T18:35:53Z"
 content="""
I need to look into why --want-get didn't behave as expected.

It would probably make sense to make standard be the default. There could be upgrade concerns though.
"""]]


code:
What steps will reproduce the problem?

Use git-annex and add file to repo.

What is the expected output? What do you see instead?

File is committed and added to annex.

Instead a command like this is run "git --git-dir=/home/jchu/annex/.git --work-tree=/home/jchu/annex commit --allow-empty-message -m  --quiet" and for some reason git decides to open a COMMIT_EDITMSG file in my editor of choice "vim .git/COMMIT_EDITMSG".  Since git annex doesn't give me access to the terminal, I don't see it.

Depending on when it happens, the web interface could refuse to start (if the commit hang happens on startup) or if it happens during regular operation, then everything looks fine but the web interface doesn't update after that.

What version of git-annex are you using? On what operating system?

git-annex (binary release) 4.20130405
git 1.8.2.1

On Arch Linux.

Please provide any additional information below.

I'm not sure if the git interface has changed, but I do see that --allow-empty-message does still exist.  If I run the git commit command (with a '' after the -m), it does indeed start up vim for me.  Would we benefit from just making a custom commit message ("Commit from date YYYYMMDDTHHMMSSZ")?

> [[fixed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkZktNHFhxC1kYA9KKdKpYJO4clq9WDsjE"
 nickname="Jason"
 subject="comment 1"
 date="2013-04-15T15:58:04Z"
 content="""
Or if that solution isn't acceptable, it seems that git commit has a --no-edit option that will guarantee that an editor isn't started.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-04-16T19:04:34Z"
 content="""
Proving a commit message with -m, which git-annex already does, should be enough to prevent git opening an editor. You appear to have some broken version of git there if that is not the case.

I see that you're using Arch. Where exactly did you download git-annex from? Have you tried the standalone tarball?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkZktNHFhxC1kYA9KKdKpYJO4clq9WDsjE"
 nickname="Jason"
 subject="comment 3"
 date="2013-04-17T02:19:02Z"
 content="""
I absolutely agree that the -m should stop git from requiring a commit message.  I don't know if anything has changed in a newer version, but that's what I'm seeing.

I'm using the git-annex-bin package from the AUR so it's already the standalone binary: https://aur.archlinux.org/packages/git-annex-bin/

As you can see by the PKGBUILD, it just copies the git-annex binary to /usr/bin/git-annex and creates a symlink for git-annex-shell.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 4"
 date="2013-04-17T05:09:42Z"
 content="""
AUR git-annex-bin is a very different thing from the standalone tarball. The tarball carefully puts together a program and a set of libraries and utilities it will work with. The git-annex-bin in AUR yanks that binary out and puts it in an entirely foreign system.

So, try the standalone tarball.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkZktNHFhxC1kYA9KKdKpYJO4clq9WDsjE"
 nickname="Jason"
 subject="comment 5"
 date="2013-04-17T16:01:56Z"
 content="""
Using the git annex standalone binary has exactly the same action.  It seems that the git annex standalone uses the system git instead of the one that's included.

The git command \"git --git-dir=/home/jchu/annex/.git --work-tree=/home/jchu/annex commit --allow-empty-message -m  --quiet\" hangs because it's waiting for a commit message \"vim .git/COMMIT_EDITMSG\".

Having not looked too closely at the git source, I think I may have found what happened.  The change happened in git 1.8.2 development.  a24a41ea9a928ccde2db074ab0835c4817223c9d changed the code to only add \"\n\n\" to a commit message if it's entirely necessary, which means a completely blank commit message looks to the code on line 978 of commit.c \"if (logfile || message.len || use_message || fixup_message)\" as if it were actually empty instead of being considered \"\n\n\".

Since this change only happened in 1.8.2, that's why when you run this code, it doesn't happen (any version before 1.8.2 shouldn't have this regression) but when I run it it does open the commit message editor.  The --no-edit option has existed since 1.7.8 (ca1ba201).  I don't know if you want to detect this case somehow or what, but I don't know if/when it's going to be fixed in git.

As it stands, the git annex assistant is useless to me because it keeps hanging when trying to commit (the thing that I really wish I could trust it on!).
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 6"
 date="2013-04-18T20:30:10Z"
 content="""
Thank you for that investigation. Have you confirmed that --no-edit actually fixes it?  I've made it pass that parameter.

(BTW, the standalone tarball contains a `runshell` script that you use to set up an environment before running git-annex. This includes putting its bundled version of git before the system one in PATH.)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkZktNHFhxC1kYA9KKdKpYJO4clq9WDsjE"
 nickname="Jason"
 subject="comment 7"
 date="2013-04-18T23:22:51Z"
 content="""
Yes, in my initial testing, I found that --no-edit fixes the problem.
"""]]


code:
What steps will reproduce the problem?

1. have an existing directory with a bunch of files
2. create a `.gitignore` file that matches some files (*.log *.aux *~ etc.)
3. `git init .`
4. `git annex init work`
5. `git remote add server server:Blabla`
6. `ssh server`
7. `@server $ mkdir Blabla`
8. `@server $ cd Blabla`
9. `@server $ git init .`
10. `@server $ git annex init server`
11. `@server $ exit`
12. `git annex webapp`

What is the expected output? What do you see instead?

I expect that ingored files stay ignored,
I see instead that all the files (including the ignored) are transfered to the server

What version of git-annex are you using? On what operating system?

3.20130124, debian sid (on both machines)

> As noted in [[design/assistant/inotify]]'s TODO list, this
> needs an efficient gitignore query interface in git (DNE)
> or a gitignore parser. --[[Joey]]

[[!tag /design/assistant]]

> [[fixed|done]]; with git 1.8.4 the assistant honors .gitignore --[[Joey]]


code:
[[!comment format=mdwn
 username="Johannes"
 ip="153.109.130.54"
 subject="comment 1"
 date="2013-07-30T11:34:15Z"
 content="""
Any news on this or this or any other way to exclude unwanted files? (e.g. all the temporary files created by latex)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnNqLKszWk9EoD4CDCqNXJRIklKFBCN1Ao"
 nickname="maurizio"
 subject="git-annex from wheezy-backports should depend on a more recent git version on debian wheezy (7.2) "
 date="2013-11-12T18:57:52Z"
 content="""
It is correct that the bug is solved on the git-annex package found on wheezy-backports, but this package does not force an update of git to a more recent version. Therefore the bug still affects wheezy users. The way to solve it is to install git also from wheezy-backports.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.246"
 subject="comment 3"
 date="2013-11-12T19:10:48Z"
 content="""
Agreed, I've made the changes so the next update of the backport will do so.
"""]]


code:
### Please describe the problem.

When the assistant is running, using "git annex unlock" on the commandline may lead to unexpected results.

### What steps will reproduce the problem?

Make sure the assitant is running on a repository.

`git annex unlock somefiles`, try to edit them, wait a while, try to edit.

If you're not lucky, the assistant will notice the unlocked files as new and will add them back, locking them in the process, and you won't be able to save them.

### What version of git-annex are you using? On what operating system?

debian wheezy.

[[!format sh """
git-annex version: 5.20131109-gf2cb5b9
build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav glacier hook
local repository version: 3
default repository version: 3
supported repository versions: 3 5
upgrade supported from repository versions: 0 1 2 4
"""]]

### Please provide any additional information below.

[[!tag confirmed]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 1"
 date="2013-12-05T20:46:30Z"
 content="""
Right -- you can't use the assistant at the same time as git-annex commands that get/drop/unlock files, unless you set the assistant to run in manual mode (which prevents it undoing a get/drop) and/or disable syncing in the assistant entirely (which prevents it adding back unlocked files).
"""]]


code:
What steps will reproduce the problem?

In one terminal, I created a new annex and started the assistant watching it.

In another, I added a file file1; assistant noticed it and added it to the annex.

I moved file1 to a directory directory1; the link broke and assistant did not fix it.

I created a file called file2 inside directory 2; assistant noticed it and added it to the annex.

I moved file2 back up to the annex root directory; the link broke and assistant did not fix it.

I created a file file3 in the annex root directory; assistant noticed it and added it to the annex.

Here is the content of the first terminal, where I created the annex and ran assistant:

    ~$ mkdir testannex
    ~$ cd testannex/
    testannex$ git init .
    Initialized empty Git repository in /Users/ed/testannex/.git/
    testannex$ git annex init "test annex"
    init test annex ok
    (Recording state in git...)
    testannex$ git annex assistant --foreground
    assistant . (scanning...) (started...) add file1 (checksum...) ok
    (Recording state in git...)
    (Recording state in git...)
    add directory1/file2 (checksum...) ok
    (Recording state in git...)
    (Recording state in git...)
    add file3 (checksum...)

here is the content of the second terminal, where I created and moved files:

    ~$ cd testannex
    testannex$ echo "file1 content" > file1
    testannex$ mkdir directory1
    testannex$ ls -l file1
    lrwxr-xr-x  1 ed  staff  180 Dec 13 15:40 file1 -> .git/annex/objects/FX/51/SHA256E-s14--edac79763e630b1b77aefb6c284bcb0362dea71c0548be0e793ffa8fd5907b80/SHA256E-s14--edac79763e630b1b77aefb6c284bcb0362dea71c0548be0e793ffa8fd5907b80
    testannex$ mv file1 directory1/
    testannex$ cd directory1/
    directory1$ cat file1
    cat: file1: No such file or directory
    directory1$ echo "file2 content" > file2
    directory1$ cat file2
    file2 content
    directory1$ cat file1
    cat: file1: No such file or directory
    directory1$ mv file2 ../
    directory1$ cd ..
    testannex$ echo "file3 content" > file3
    testannex$

What is the expected output? What do you see instead?

The links do not break when moved to another directory.

What version of git-annex are you using? On what operating system?

One compiled using cabal from checkout 739c937

Please provide any additional information below.

> [[fixed|done]]; this turned out to be an kqueue specific bug,
> the kqueue code statted new files, but that files for a broken symlink.
> Using lstat instead fixed this. --[[Joey]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 1"
 date="2012-12-13T20:50:18Z"
 content="""
(Operating system is OS X Lion)

"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 2"
 date="2012-12-15T05:47:25Z"
 content="""
thanks, fix worked for me!  Now the links are fixed virtually instantaneously when the file is moved.

"""]]


code:
git remotes that have git-annex and git "installed" using the linux
standalone tarball (or the OSX dmg) may not have git-receive-pack in PATH.

To make `git annex sync` and `git push` etc work on such a remote,
it needs to have a dedicated ssh key that runs git-annex-shell and uses it
to run the command.

The assistant sets that up when it sets up a dedicated ssh key. However, if
the server can be sshed into without a password using an existing ssh key,
the assistant re-uses that key.

The assistant should either always set up a dedicated ssh key, even when
not otherwise needed, or it should probe to see if git-receive-pack and
git-send-pack can be run as part of its probing, and set up a dedicated
ssh key when they cannot be. --[[Joey]]

> After recent sshpassword changes, the webapp sets up a dedicated ssh key
> by default. If the user chooses to use an existing ssh key, it will be
> used. So this is less likely to be a problem. --[[Joey]]

[[!tag confirmed]]


code:
### Please describe the problem.

I migrated a repo I set up by hand on my Android phone to use the assistant and webapp. That repo has two remotes, both full git-annex repos over SSH. I still want to control when I synchronize (cellular data is precious), so the vast majority of the time I disable syncing on both remotes. However, when I return to the webapp after it's been running for "a while," it shows a status message indicating it has synced with the remotes.

It looks like it might be related to the NetWatcherFallback, because a NetWatcherFallback entry appears in the log approximately every hour, followed by the usual repo sync output.

### What steps will reproduce the problem?

1. Set up a repo with a git-annex over SSH remote, and set annex-sync on that remote to false.
2. Run the webapp.
3. Observe after a period of time that the webapp shows that the repo has been synced.

### What version of git-annex are you using? On what operating system?

git-annex 20130709 on Android (4.2)

### Please provide any additional information below.

[[!format sh """
[2013-07-22 22:50:31 MST] NetWatcherFallback: Syncing with sigsegv, sigusr1
Everything up-to-date
Everything up-to-date
[2013-07-22 23:52:57 MST] NetWatcherFallback: Syncing with sigsegv, sigusr1
Everything up-to-date
Everything up-to-date
[2013-07-23 00:54:15 MST] NetWatcherFallback: Syncing with sigsegv, sigusr1
Everything up-to-date
Everything up-to-date
"""]]

> Excellent bug report! Especially useful that you tracked it
> down to the NetWatcher. Indeed it was not honoring the
> annex-sync setting. [[fixed|done]] --[[Joey]]


code:
[[!meta title="windows assistant fails to set up rsync.net authorized keys"]]

### Please describe the problem.
Having set up a key pair, the assistant is unable to use it to log in. The console only shows a single attempt to log in, when it's adding the key to the remote machine's authorized_keys file. This is evidently failing, as the key is not actually added to authorized_keys, possibly because the shell script it tries to execute is malformed (crazy quoting, I can't untangle it), or possibly because it failed to ask for a password (I expected it to, but it didn't prompt for one). Also, once I do add the key to authorized_keys, I can ssh in using it but asking the assistant to retry still fails.

### What steps will reproduce the problem?
Set up a remote repository.

### What version of git-annex are you using? On what operating system?
Windows 7, Version: 5.20131230-g192d991

### Please provide any additional information below.

    Failed to make repository
    Something went wrong setting up the repository on the remote server.
    Transcript:
    Permission denied, please try again.
    Permission denied, please try again.
    Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[[!format sh """
[2014-01-01 03:28:05 Eastern Standard Time] call: ssh-keygen ["-P","","-f","C:\\
Users\\FAMILY~1\\AppData\\Local\\Temp\\git-annex-keygen.5\\key"]
Generating public/private rsa key pair.
cygwin warning:
  MS-DOS style path detected: C:\Users\FAMILY~1\AppData\Local\Temp\git-annex-key
gen.5\key
  Preferred POSIX equivalent is: /cygdrive/c/Users/FAMILY~1/AppData/Local/Temp/g
it-annex-keygen.5/key
  CYGWIN environment variable option "nodosfilewarning" turns off this warning.
  Consult the user's guide for more details about POSIX paths:
    http://cygwin.com/cygwin-ug-net/using.html#using-pathnames
Your identification has been saved in C:\Users\FAMILY~1\AppData\Local\Temp\git-a
nnex-keygen.5\key.
Your public key has been saved in C:\Users\FAMILY~1\AppData\Local\Temp\git-annex
-keygen.5\key.pub.
The key fingerprint is:
c7:0f:81:19:c8:39:cc:e1:d3:6a:62:9a:d5:d2:9f:c1 Familyroom@Familyroom-PC
The key's randomart image is:
+--[ RSA 2048]----+
|     +.+.        |
|     .B. +       |
|      o.+ .      |
|     o + . .     |
|    = = E +      |
|   = + . + o     |
|  o     o   .    |
|                 |
|                 |
+-----------------+
[2014-01-01 03:28:07 Eastern Standard Time] chat: ssh ["-p","22","rlbrooks@celeb
dil.db48x.net","sh -c 'mkdir -p '\"'\"'annex/test'\"'\"'&&cd '\"'\"'annex/test'\
"'\"'&&if [ ! -d .git ]; then git init --bare --shared && git config receive.den
yNonFastforwards false; fi&&git annex init&&mkdir -p ~/.ssh&&if [ ! -e ~/.ssh/gi
t-annex-shell ]; then (echo '\"'\"'#!/bin/sh'\"'\"';echo '\"'\"'set -e'\"'\"';ec
ho '\"'\"'if [ \"x$SSH_ORIGINAL_COMMAND\" != \"x\" ]; then'\"'\"';echo '\"'\"'ex
ec git-annex-shell -c \"$SSH_ORIGINAL_COMMAND\"'\"'\"';echo '\"'\"'else'\"'\"';e
cho '\"'\"'exec git-annex-shell -c \"$@\"'\"'\"';echo '\"'\"'fi'\"'\"') > ~/.ssh
/git-annex-shell; fi&&chmod 700 ~/.ssh/git-annex-shell&&touch ~/.ssh/authorized_
keys&&chmod 600 ~/.ssh/authorized_keys&&echo '\"'\"'command=\"GIT_ANNEX_SHELL_DI
RECTORY='\"'\"'\"'\"'\"'\"'\"'\"'annex/test'\"'\"'\"'\"'\"'\"'\"'\"' ~/.ssh/git-
annex-shell\",no-agent-forwarding,no-port-forwarding,no-X11-forwarding ssh-rsa A
AAAB3NzaC1yc2EAAAADAQABAAABAQDeB64i+DhYBCq0mdet6sBr1JCtYR5zNc0E9NYaZbBVRBcpZDCRs
AS1z8mjyEnb49y7AnNgsx/f7xAWtyFHFr8AbrYnaHmJYicix0R6inDXZ42WJTusHVW47/1ysErHmEusZ
KWXNzN5bu8bOoz0FqCh4Ymo3QjgMzk6+PoWqSCmwewuX++RkB6q1IN1VTMqJguPtdQpTp3hCIV6GENf6
HvnLUQ59JKg1sgt5UA88zDAMlXicmUeuHRSW7nVlVIBHFDJW8f5CyF66wiuGTLdhhMlaOqpUqIEHubPF
ZE4iTOC/uuOdWjGgySX681ub4seKyz7h8uDePPo0uz0BXJpt9vT Familyroom@Familyroom-PC\n'\
"'\"' >>~/.ssh/authorized_keys'"]
"""]]

[[!format sh """
$ ssh rlbrooks@git-annex-celebdil.db48x.net-rlbrooks_annex
git-annex-shell: bad parameters

Usage: git-annex-shell [-c] command [parameters ...] [option ..]

Options:
           --force         allow actions that may lose annexed data
  -F       --fast          avoid slow operations
  -a       --auto          automatic mode
  -q       --quiet         avoid verbose output
etc
"""]]

> [[fixed|done]]; both for regular remote ssh servers, and for rsync.net --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 1"
 date="2014-01-01T19:56:32Z"
 content="""
I can reproduce this bug. Seems the problem is that sshTranscript is called with a string that is fed in on stdin. But I guess that on Windows, ssh password prompting is also done using stdin (on Linux it would use /dev/tty), and so this prevents it from prompting.

I've fixed this for the remote ssh server setup in the webapp, which did not actually need to feed anything in on stdin. Confirmed ssh adding works!

However, for rsync.net, the ssh pubkey is fed into dd using stdin, and so I was unable to fix it there yet. Since rsync.net does not support redirection to files in its shell, the only approach that would seem to work is to first download the authorized_keys file, add the line, and rsync in the new file. Until I get a chance to do that, I am leaving this bug report open.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 2"
 date="2014-01-01T19:59:11Z"
 content="""
Also, fixing rsync.net that way would mean two ssh connections to set up authorized_keys, so 2 password prompts. suboptimal..
"""]]


code:
### Please describe the problem.
The assistant (or rather the programs it's running) appears to be using unix-style paths instead of cygwin paths, causing several types of errors. A selection of items from the terminal:

[[!format sh """
Launching web browser on file://C:\Users\Familyroom\annex\.git\annex\webapp.html
[2013-12-31 01:14:05 Eastern Standard Time] read: ssh-keygen ["-F","celebdil.db4
8x.net"]
ssh-keygen: /home/Familyroom/.ssh/known_hosts: No such file or directory
[2013-12-31 01:14:05 Eastern Standard Time] read: ssh ["-oNumberOfPasswordPrompt
s=0","-oStrictHostKeyChecking=no","-n","-p","22","rlbrooks@celebdil.db48x.net","
sh -c 'echo git-annex-probe loggedin;if which git-annex-shell; then echo git-ann
ex-probe git-annex-shell; fi;if which git; then echo git-annex-probe git; fi;if
which rsync; then echo git-annex-probe rsync; fi;if which ~/.ssh/git-annex-shell
; then echo git-annex-probe ~/.ssh/git-annex-shell; fi;cd '\"'\"'annex'\"'\"' &&
 git config --list'"]
[2013-12-31 01:14:08 Eastern Standard Time] read: ssh-keygen ["-F","celebdil.db4
8x.net"]
ssh-keygen: /home/Familyroom/.ssh/known_hosts: No such file or directory
[2013-12-31 01:14:08 Eastern Standard Time] read: ssh ["-oStrictHostKeyChecking=
no","-n","-p","22","rlbrooks@celebdil.db48x.net","sh -c 'echo git-annex-probe lo
ggedin;if which git-annex-shell; then echo git-annex-probe git-annex-shell; fi;i
f which git; then echo git-annex-probe git; fi;if which rsync; then echo git-ann
ex-probe rsync; fi;if which ~/.ssh/git-annex-shell; then echo git-annex-probe ~/
.ssh/git-annex-shell; fi;cd '\"'\"'annex'\"'\"' && git config --list'"]
rlbrooks@celebdil.db48x.net's password:
[2013-12-31 01:15:03 Eastern Standard Time] read: gpg ["--quiet","--trust-model"
,"always","--with-colons","--list-secret-keys","--fixed-list-mode"]
gpg: WARNING: using insecure memory!
gpg: please see http://www.gnupg.org/documentation/faqs.html for more informatio
n
gpg: fatal: can't create directory `/home/Familyroom/.gnupg': No such file or di
rectory
secmem usage: 0/0 bytes in 0/0 blocks of pool 0/65536
"""]]

Naturally, I don't have a /home/Familyroom, although I do have a /c/Users/Familyroom.

### What steps will reproduce the problem?
Start creating a remote repository.

### What version of git-annex are you using? On what operating system?
Windows 7, git-annex version 5.20131230-g192d991

> [[fixed|done]]; git-annex now ensures HOME is set when running cygwin
> commands that require it. --[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5WyknJirJJridJjiPNgrlYxGG9xrZBvA"
 nickname="Daniel"
 subject="comment 1"
 date="2013-12-31T20:48:03Z"
 content="""
I neglected to mention that the assistant is failing when gpg fails, and that it worked last week in 5.20131221-gf8c928d (or at least it got past this point).
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 2"
 date="2013-12-31T21:17:32Z"
 content="""
It seems that both ssh-keygen and gpg are for some reason acting as if HOME is set to /home/Familyroom.
git-annex does not do anything to override the default home directory these programs use, and AFAIK, nothing has changed in the Windows build between the working and broken versions that could have anything to do with this.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5WyknJirJJridJjiPNgrlYxGG9xrZBvA"
 nickname="Daniel"
 subject="comment 3"
 date="2013-12-31T23:47:43Z"
 content="""
Yea, the only relevant change I noticed was 5c834daf5, and it's not particularly relevant.

However, its occurred to me that I started the webapp differently this time than last; I used the start menu shortcut this time rather than typing git annex webapp in the git shell. It's not surprising that the environment variables would be quite different between the two.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5WyknJirJJridJjiPNgrlYxGG9xrZBvA"
 nickname="Daniel"
 subject="comment 4"
 date="2013-12-31T23:56:24Z"
 content="""
Yea, if I start it from the Git Bash command line, it continues past this to a completely different error.
"""]]


code:
I am running 3.20121112 and this bug appeared when I upgraded to that from 3.20121017.

After performing the startup scan after I login, git-annex gets GPG to sends me a pinentry pop-up asking for my GPG passphrase.  However, I know that there is nothing to be dropped or copied to the encrypted SSH remote that I am being asked to provide access to.  So I can't see any good reason why I would be asked for the passphrase.

> [[fixed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.6.49"
 subject="comment 1"
 date="2012-11-29T20:13:53Z"
 content="""
Hmm, is this an encrypted rsync remote?

I have not been able to reproduce the problem so fat.

If you start the assistant with --debug, and look in .git/annex/daemon.log, you should find where it runs gpg, and the lines showing what it's doing just before/after that should provide a hint what it's doing.
"""]]


code:
[[!comment format=mdwn
 username="spwhitton"
 ip="163.1.167.50"
 subject="comment 2"
 date="2012-12-01T18:34:10Z"
 content="""
Thanks for the log advice.  I looked and it wants to drop a file `ttmik/TTMIK-Lesson-L1L1.mp3` from the encrypted rsync remote ma.  So I did `git annex whereis ttmik/TTMIK-Lesson-L1L1.mp3` and learnt that the file is not on ma.  I tried `git annex drop --from ma ttmik` to be sure, and the command was successful, but it still tries to drop the file from ma on startup.  Presumably it would try all the other files in the ttmik directory if I gave it the chance to try to drop this first one.  The only thing special about the ttmik directory is that every file in there was added using addurl, so I guess the problem has something to do with that.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.6.49"
 subject="comment 3"
 date="2012-12-01T18:59:45Z"
 content="""
How is this rsync remote configured? Is it configured as a transfer remote? If not, the assistant's default behavior is to sync all files to it, and since you say the file is not there, it'd make sense it would start transferring it to there on startup.

OTOH, this could be a complete red herring. You haven't shown me the log file. Perhaps the drop you're seeing in the log is before the operation that is asking for the GPG passphrase. I can't tell until you show me the log.
"""]]


code:
[[!comment format=mdwn
 username="spwhitton"
 ip="163.1.167.50"
 subject="comment 4"
 date="2012-12-02T18:27:09Z"
 content="""
The rsync remote is in the archive group but it has the preferred content string `exclude=video/* and exclude=ttmik/*`.  So it ought to be dropping the files, but not over and over again.

Here is the log file, showing me hitting Escape on the pinentry dialog:

    (scanning...) Already up-to-date.
    Already up-to-date.
    Already up-to-date.
    drop ma ttmik/TTMIK-Lesson-L1L1.mp3 (gpg) gpg: cancelled by user
    gpg: decryption failed: secret key not available
    TransferScanner crashed: user error (gpg [\"--batch\",\"--no-tty\",\"--use-agent\",\"--
    quiet\",\"--trust-model\",\"always\",\"--decrypt\"] exited 2)
    (started...)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.72"
 subject="comment 5"
 date="2012-12-05T16:34:24Z"
 content="""
Aha, I see the bug now. It is indeed trying to drop from every remote, even ones without the data.

(Also, the thread should not crash here if the drop fails.. I've fixed that.)
"""]]


code:
Found this out the hard way.  See the comment in the below post for what happens.

[[/forum/git_annex_add_crash_and_subsequent_recovery/]]

> [[fixed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2011-12-06T16:49:32Z"
 content="""
This only happens with the WORM backend (or possibly with SHA1E if the file's extension has a newline).

The problem is not the newline in the file, but the newline in the key generated for the file. It's probably best to just disallow such keys being created.
"""]]


code:
In our family repository, we had an event where git-annex committed a bad
merge commit. This had the effect of seeming to delete all the files
in the repository. However, it is completely recoverable with no file loss.

Other users have reported apparently the same problem.
--[[Joey]]

> This appears to be a race, and I have put in a fix for that.
> It will be released in git-annex 5.20140709.
>
> It's a little bit difficult to say for sure that the bug is fixed, since
> I don't have a good way to reproduce it. However, everything I am seeing
> is consistent with my analysis, at least as far as the people who
> reported this happening with versions 5.20140613-5.20140707.
> Calling this [[done]]. --[[Joey]]

## recovery

Look through the git log for the bad commit. It will be the one
that deletes a lot of files. Since the bad commit is a merge commit,
you need to include the -c switch:

	git log -c --stat

Once the bad commit sha is found, revert it. Since it's a merge commit,
you will need to pass -m 1 to git revert. If the repository you're working
is uses direct mode, you will first need to switch it to indirect mode.

	git annex indirect
	git revert -m 1 a36077cf1234eeb755fec8f699d3cbaaee817bac

(It's possible, but I think unlikely that this will pick the wrong
branch of the merge to revert. If it did, "git reset --hard HEAD@{1}" and
redo, passing `-m 2` instead.)

Once the revert is done, all your files should be back. You can switch
the repository back to direct mode if desired (`git annex direct`)
and can `git annex sync` to push the fix out to all other clones.
Everywhere the fix lands should restore the deleted files.

Note that it's possible that some repositories may have pruned the deleted
files as unused. This is most likely to happen in the repository that made
the bad merge commit in the first place.

If you have any problems following these instructions, please contact
<urgent2014@joeyh.name> for support.

[[!tag confirmed urgent]]

## analysis

In the one case I have of this happening, the commit was made by the
git-annex assistant, in direct mode. It was git-annex version 5.20140613.
This version had a bug that caused it to do unncessary merge commits.
That bug has been fixed in 5.20140707.
That is apparently related, in that it caused the assistant to do much more
work than normal.

But, I don't think that bug is fully responsible for the
problem. I think a few users have run into this bug before, although I have
never succeeded in getting a full problem description from anyone who might
have. I think that the unncessary commit bug made it more likely to happen.
(This suggests a race may be involved.)

The bad merge commit looked like this:

<pre>
commit a36077cf1234eeb755fec8f699d3cbaaee817bac
Merge: 52ecff2 9d8bfd4
Author: xxx
Date:   Mon Jul 7 19:58:18 2014 -0400

    merge refs/heads/synced/master
</pre>

Both parent trees were good and contained all files. In fact, the 2
parent trees contained the same set of files. But the merge commit
"resolved" the merge by deleting all the files.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.55"
 subject="comment 1"
 date="2014-07-09T18:20:13Z"
 content="""
mergeDirect locks .git/index, so no other git command should use it while the merge is in progress.

However, if the index file is deleted, and stageMerge runs, it stages deletion of all files. So that looks likely to be part of the story.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.55"
 subject="comment 2"
 date="2014-07-09T18:25:08Z"
 content="""
Also pointing in the direction of a race: The bad commit happened 3 seconds after another commit did. The other commit was an empty commit, so caused by the bug in 5.20140613.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.55"
 subject="theory"
 date="2014-07-09T18:31:07Z"
 content="""
Suppose we have 2 mergeDirect's A and B running at the same time somehow, with B around 3 seconds behind A, and this sequence of events occurs:

1. A copies index file to index.lock
2. A stages an empty commit
3. B copies index file to index.lock
4. A finishes its commit, and so renames index.lock back to index.
5. B runs stageMerge

So, B is now running stageMerge with `GIT_INDEX_FILE` pointing to an index.lock that DNE. As noted above, this causes a deletion of all files to get staged by B.

If this is the problem, git-annex could fix it using a real lock file. (The index.lock only prevents other git commands from manipulating the index during the merge.)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.55"
 subject="comment 4"
 date="2014-07-09T18:36:36Z"
 content="""
The above race was first introduced in version 5.20140613.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.55"
 subject="comment 6"
 date="2014-07-09T19:01:33Z"
 content="""
Note that [[Assistant_merge_loop]], which contributes to this bug, is not actually fixed in 5.20140707, it seems.
"""]]


code:
[[!comment format=mdwn
 username="http://jamesjustjames.wordpress.com/"
 nickname="purpleidea"
 subject="The guy with this bug."
 date="2014-07-09T18:50:36Z"
 content="""
* Sure enough looking though the git log, I found one that removed 81
files. It was useful to grep for 'deletions'.

* It turned out there were actually two, one from each of the two
git-annex-assistant machines. I had to do the revert separately on each
machine for the corresponding commit that it had made.

* I typically run the latest version that the webapp offers, within
about a week or so of it prompting me (unless I'm away for a while).
This means that maybe you can track down the approximate version where
this occured by looking at the two commit dates of the \"bad\" merges:

commit bbe43fb638d101d5b9bfa65a4023e7f9d90e72e6
Merge: 351cca8 95935ce
Author: James <james@computer1>
Date:   Tue Jul 8 02:15:57 2014 -0400

commit 1902a3af25b1ce3723c38877c7f25907885040e4
Merge: 125b984 efcece7
Author: James <james@computer2>
Date:   Tue Jul 8 02:15:27 2014 -0400

* I use the command line for pretty much most of what I do. Funny enough
though I only really use git-annex through the webapp.

* Bug wise I don't know if this is relevant, but I have done the
following:
> added/changed/removed files to ~/annex on computer one while git-annex
wasn't running
> added/changed/removed files to ~/annex on computer two while it wasn't
running
> started git-annex on computer one
> started git-annex on computer two
> and similar combinations of the above...
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.55"
 subject="more confirmation of race theory"
 date="2014-07-09T19:26:55Z"
 content="""
@purpleidea showed me his git log, and as I had expected, there were previous empty commits immediately before the 2 commits that removes all of his files (which happened independenctly on 2 different machines running the assistant). This is consistent with my suspicion it's a race. Note that the assistant's git ref watcher can easily run multiple merges concurrently if multiple refs change.
"""]]