code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.63"
 subject="comment 3"
 date="2013-08-24T16:35:33Z"
 content="""
Seems to me that this could easily be dealt with by installing git-annex on the webserver, making the directory there a git repository, and using either a cron job or `git annex watch` to commit files as they were changed there.

Then you can make a direct mode, indirect mode, or even a bare clone on your local machine and use git-annex to get the files.

Maybe you have good reasons for not wanting to go that route. And rsync on a direct mode repository should work, provided to tell it to not delete `.git`. :P I don't see any way to make rsync work in an indirect mode repository. As for trying to make git-annex handle this import over rsync itself in a way that would work in an indirect mode repository, let alone a bare repository -- I don't see a good way to do it and it seems quite special case and likely to get quite complicated to implement.

In the meantime, I did implement `git annex mirror`, which I think is a much more interesting and generally useful tool to have. And could even be used in my recommended solution above.
"""]]


code:
[[forum/tips:_special__95__remotes__47__hook_with_tahoe-lafs]] is a good
start, but Zooko points out that using Tahoe's directory translation layer
incurs O(N^2) overhead as the number of objects grows. Also, making
hash subdirectories in Tahoe is expensive. Instead it would be good to use
it as a key/value store directly. The catch is that doing so involves
sending the content to Tahoe, and getting back a key identifier.

This would be fairly easy to do as a [[backend|backends]], which can assign its
own key names (although typically done before data is stored in it),
but a tahoe-lafs special remote would be more flexible.

To support a special remote, a mapping is needed from git-annex keys to
Tahoe keys, stored in the git-annex branch.

> This is now done, however, there are 3 known
> problems:
>
> * tahoe start run unncessarily <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2149>
> * web.port can conflict <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2147>
> * Nothing renews leases, which is a problem on grids that expire.
>   <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2212>

> --[[Joey]]

repo_name: kakafarm/git

license: unknown


code:
[[!comment format=mdwn
 username="zooko"
 ip="97.118.97.117"
 subject="performance"
 date="2011-05-17T19:20:39Z"
 content="""
Hm... O(N^2)? I think it just takes O(N). To read an entry out of a directory you have to download the entire directory (and store it in RAM and parse it). The constants are basically \"too big to be good but not big enough to be prohibitive\", I think. jctang has reported that his special remote hook performs well enough to use, but it would be nice if it were faster.

The Tahoe-LAFS folks are working on speeding up mutable files, by the way, after which we would be able to speed up directories.
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 2"
 date="2011-05-17T19:57:33Z"
 content="""
Whoops! You'd only told me O(N) twice before..

So this is not too high priority. I think I would like to get the per-remote storage sorted out anyway, since probably it will be the thing needed to convert the URL backend into a special remote, which would then allow ripping out the otherwise unused pluggable backend infrastructure.

Update: Per-remote storage is now sorted out, so this could be implemented
if it actually made sense to do so.
"""]]


code:
It should be possible to union mount annexes. So if multiple drives have
content, an annex mounting them both would have available all the
content from all the drives.

This could be done by just making .git/annex/KEY link to the actual content
on the mounted annex.

(Need to make sure the [[copy_tracking|copies]] code does not
confused and think the symlink is a copy of the content.. Also need to make
sure that code that writes to .git/annex does not follow symlinks.))


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="comment 1"
 date="2013-03-01T01:26:36Z"
 content="""
This would indeed be very helpful when remotely mounting a photo/video collection over samba.
"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 2"
 date="2013-03-01T04:50:28Z"
 content="""
+1 this would be sweet as hell

"""]]


code:
[[!comment format=mdwn
 username="http://ypid.wordpress.com/"
 nickname="ypid"
 subject="Please add this ;)"
 date="2014-03-13T19:10:17Z"
 content="""
+1 This would be so great. For me the only thing which is missing in this awesome project.
"""]]


code:
Is it possible to freeze or peg repositories at a particular version, or to prevent automatic repository version upgrades?  Is it possible to "downgrade" a repository?

### Please describe the problem.

We have a number of repositories on a shared file server.  These repositories are accessed by multiple machines.  Some of these repositories appear to have gotten upgraded and are now unusable on machines running older versions of git-annex.

We're getting this message:
[[!format sh """
user@system:/path/to/repository$ git annex status
git-annex: Repository version 5 is not supported. Upgrade git-annex.
"""]]

The machine experiencing the problem is running Debian Wheezy (Stable).
[[!format sh """
user@system:/path/to/repository$ git version
git version 1.7.10.4
user@system:/path/to/repository$ git annex version
git-annex version: 3.20120629
local repository version: 5
default repository version: 3
supported repository versions: 3
upgrade supported from repository versions: 0 1 2
"""]]

I'm guessing that one of the machines with access to this repository was running a newer version of git-annex, and that the repository was upgraded in the course of some action.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 1"
 date="2014-06-04T18:14:19Z"
 content="""
If your repository is not using direct mode, it's completely safe to edit .git/config and set the version back to 3. There is no change between 3 and 5 for indirect mode repositories.

Unfortunately, using git-annex version 5 will automatically upgrade the repository to 5 again. In general, I only want git-annex to support one version at a time, to avoid complicating the code. I did try leaving the indirect mode repositories at v3, but that didn't work out (some details in [[!commit b1d7474c1d713a5b422948178abb4e5f39e85096]]).

I kind of think that part of the problem is that you're using git-annex repositories accessed via a file server. If your server had git-annex installed on it and the clients talked to it only by sshing in and running git-annex-shell, it would not matter if the clients had a newer version, because they'd never access the central repository directly.
"""]]


code:
Having the option of choosing for every file if we want it in our repository or not would be a great feature. It is currently possible using the wanted expression, but it is not very flexible, or it becomes unmaintainable.

I tried with two repositories a and b, with the following wanted expressions :

* for a: `not metadata=unwanted=<uuid-of-a>`
* for b: `not metadata=unwanted=<uuid-of-b>`

I think those expressions should be included in standard wanted expressions.

Also, to improbe the feature, it should be possible to set (or remove) metadata in directories, and those should automatically affect their content.

And we could imagine a `git annex unwant` command that would add the unwanted metadata to a file, copy it to other repositories, and attempt to drop it.


code:
It seems like it is possible to achieve this now on later versions of Windows (7 and above).

The mklink tool creates a symlink that works on windows.

There would be some work required so that a linux symlink and a windows symlink are considered the same, a user has recommend 'git update-index --assume-unchanged' for this.

There is a good write up of someone using this approach on vanilla git here: http://stackoverflow.com/questions/5917249/git-symlinks-in-windows


code:
Just gave git annex a quick try for a few minutes and I must admit it's pretty great.

A must have feature for me is to be able to view git annex log in the web app as "git annex log" doesn't got BIDI support (RTL scripts like Arabic, Farsi, Hebrew).
Adding Bi-directional text support would be too much to ask from a developer that don't know these languages thus the solution is to use the already the web browser to handle that.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmHp5oVW8Z9v_vHs5FRtlYj80TYMQWVTS0"
 nickname="dhead"
 subject="comment 1"
 date="2014-05-30T00:13:27Z"
 content="""
...the already built in BIDI support the web browser to handle that.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.176"
 subject="comment 2"
 date="2014-05-30T00:32:25Z"
 content="""
I don't understand this request at all

* There are plenty of console emulators with bidi support. If you need bidi support, surely you have already found and are using one of them?
* `git annex log` does not output much that seems likely to need bidi support. At least no more than any other git-annex command, specifically names of remotes, and names of files.
* `git annex log` is a low-level tool, almost a debugging tool. Users of the webapp might be interested in which repositories a file has gotten to, but surely not the history of past locations of files.
"""]]


code:
When the webapp is used to set up a ssh repository, it should detect if
there is already a repository, and sshing in to the server forces running
git-annex-shell (perhaps by it being set to the user's login shell).

In this case it should just use the already set up repository. Note that
it would not be possible to add a new repository on the server using only
git-annex-shell. --[[Joey]]


code:
## git-annex as service on windows

Use nssm to run git-annex as a service. Will need to include it in the
git-annex bundle.

Problem: nssm runs git-annex as a service as a LocalService user. (Or some
similar user.) This leads to permission problems, when the normal user
tries to write to its directory.

Solution: Make `git-annex mkservice $repo` command (only avilable on
Windows) that does:

1. git -c core.sharedRepository=true init $repo
2. cd $repo; git annex init
4. chmod 777 -R $repo
5. Add $repo to C:\.config\git-annex\autostart
6. If git-annex service does not yet exist in nssm, set it up and start it.

**Problem**: With 2 users writing to one repository, files and subdirs
will end up owned by git-annex or by the desktop user, and the other user
won't be able to eg, edit a file or remove a file from a directory.

Make git-annex read `C:\.config\git-annex\autostart`
on Windows, in addition to the one in $HOME. This way, `git annex assistant
--autostart` and `git annex webapp` will use it, no matter which user.

WIP git branch: `winservice`

> I am calling this [[done]], it's not done using a service, but it works.
> --[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm9ocq1Kb0WL-cz-LPpvd2Xm-q8tIQvqXA"
 nickname="Dominik"
 subject="running as service"
 date="2014-05-13T17:40:57Z"
 content="""
Hey, just tried git-annex on windows. Love the idea, but the windows port still seems to be a work in progress. I thought I'd check in here and see if i could help. Maybe this is a start:

You can setup any executable file as a windows service using Sc.exe included in the Windows Resource Kit.
https://support.microsoft.com/kb/251192?SmcNavTabIndex=1
Its pretty easy and straightforward

Another option would be installing the service using nsis, here is one possible plugin that should do the trick: http://nsis.sourceforge.net/NSIS_Simple_Service_Plugin
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 11"
 date="2014-06-16T23:47:36Z"
 content="""
I am stuck with apparently intractable permissions problems, if the git-annex service doesn't run as the same user who wants to use the git-annex repository.

nssm is supposed to let the user the service runs as be set on the Log on tab, but I cannot get it to accept any user/password there, on my XP test VM. If anyone gets that tab in nssm to work, let me know.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm9ocq1Kb0WL-cz-LPpvd2Xm-q8tIQvqXA"
 nickname="Dominik"
 subject="comment 12"
 date="2014-06-17T00:01:35Z"
 content="""
To get rid of permission problems, you may find setacl useful. Chmod and chown from msys dont't seem to work correctly with windows 8
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 12"
 date="2014-05-15T20:36:09Z"
 content="""
@Dominik, thanks for the links. Now that the webapp handles prompting for ssh passwords, the console is entirely vestigial and I'd like to get rid of it. The sc commands seems possible to use; there are also haskell libraries for building windows services.

The tricky part is that multiple git-annex assistant processes can be running, if there are multiple local repositories. This seems to be hard to do as a service.
"""]]


code:
[[!comment format=mdwn
 username="http://johan.kiviniemi.name/"
 nickname="Johan"
 subject="comment 13"
 date="2014-05-16T02:27:51Z"
 content="""
“The tricky part is that multiple git-annex assistant processes can be running, if there are multiple local repositories.”

This would probably mean a lot of work, but it might be nice if there was just a single git-annex assistant per user which could take care of multiple local repos. The UI would be nicer, too, since the status of and notifications from every repository could be shown in the same place.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 13"
 date="2014-06-17T16:52:55Z"
 content="""
I have messed with the windows ACLs some yesterday, but I don't know how or if it's possible to set ACLs on a directory, such that everything created inside it will be writable by two different users. Certainly this is doable on POSIX; if it's doable on Windows, I'll revisit services.

----

For now, it seems that a better option may be to not run git-annex as a service, but use various dos-window hiding technologies.

<http://stackoverflow.com/questions/21031171/how-to-run-a-command-on-the-background-on-windows/21031281#21031281>

I have successfully gotten this to work using nircmd. make a git-annex-webapp.bat, containing:

<pre>
title GitAnnex
nircmd.exe win hide ititle \"GitAnnex\"
git annex webapp
</pre>

This works, although the DOS box flashes onscreen for maybe 1/10th of a second before nircmd hides it. A git-annex-assistant.bat could run git-annex assistant --autostart, and would be suitable to be setup to run on startup.

(It seems that it's possible to write a VBScript or C# program that sets up a hidden WScript.Shell and runs a command in it. That might avoid the window flash. However, it seems hard to get VBScript to run, and I have not investigated C#.)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm9ocq1Kb0WL-cz-LPpvd2Xm-q8tIQvqXA"
 nickname="Dominik"
 subject="multiple git-annex assistant processes"
 date="2014-05-16T23:05:28Z"
 content="""
shouldn't be a problem though if all processes are spawned from the same service though, right? Or what problems did you have in mind?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 14"
 date="2014-06-17T16:56:32Z"
 content="""
Note that nircmd is not free software; but can be distributed free of charge, provided all the files in the zip are included and unmodified. Sucks, but it's windows..
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 15"
 date="2014-06-17T17:10:28Z"
 content="""
<http://www.ntwind.com/software/hstart.html> is another option.

How to run VB script:

<http://stackoverflow.com/questions/289498/running-batch-file-in-background-when-windows-boots-up>
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 15"
 date="2014-06-04T20:49:26Z"
 content="""
`sc` can only be used by the administrator. That is problimatic.

Also, in order for sc to be used, the program has to have support for it. It cannot be any arbitrary program. win32-service adds such support. However, I have not been able to get programs using win32-service to build: <https://github.com/mikesteele81/Win32-services/issues/3>
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 16"
 date="2014-06-17T17:18:34Z"
 content="""
Using wscript to run a file containing this will start the webapp w/o any console flicker:

<pre>
Set objshell=CreateObject(\"Wscript.Shell\")
objShell.Run(\"git-annex webapp\"), 0, False
</pre>
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="service not needed"
 date="2014-06-04T21:16:06Z"
 content="""
Seems that all I need to do is pass -optl-mwindows when building git-annex; this will produce a binary that does not open a console window when started from the start menu, but still runs in the background. So it will work as the webapp.

**However**, it can't be used at the command line. So, I think I should make a git-annex-webapp shim that's built that way and starts the real webapp.

But, some work needs to be done, because when run this way, git-annex can't write to stdout or stderr, which it tries to do, and so crashes on startup.

Also, when git-annex is built that way and tries to run git.. windows helpfully opens a command.exe to display any output git might have. So we get lots of flickering command.exe windows.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm9ocq1Kb0WL-cz-LPpvd2Xm-q8tIQvqXA"
 nickname="Dominik"
 subject="comment 17"
 date="2014-06-05T16:42:02Z"
 content="""
Actually you can create a service from any program or batch file using srvany.exe from the windows resource kit: http://support.microsoft.com/kb/137890. An alternative would be the Non-Sucking Service Manager http://nssm.cc/
Also nsis has some plugins for creating services, but i havent tried any of them: http://nsis.sourceforge.net/How_do_I_start/stop/create/remove/check_a_service

Upsides:
1) You don't have to touch your code or change anything in the compilation just to create the service
2) git-annex will still work fine from the console
Downside: You'd have to include an extra binary or plugin in your installer for creating the service

But I'm pretty sure that none of the commandline calls would pop up in a new window that way, but if you need the verbose mode, you could still start the webapp from console instead as service and everything will show as expected
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="nssm notes"
 date="2014-06-10T18:16:00Z"
 content="""
* Have to install and run nssm as an administrator. (It may be possible to get it to run as a non-administrator user if the admin sets it up. I haven't succeeded.)
* `nssm install git-annex`
* Set path to git-annex, and set Arguments to: `assistant --autostart`
* In Exit tab, change Restart to \"No action\"
* In Process tab, uncheck \"Console window\"
* Repositories to start up have to be listed in `C:\Documents and Settings\LocalService\.config\git-annex\autostart`
  (rather than the normal user home directory)

After all that it works! Even opening git-annex webapp from the menu avoids the console window (it appears briefly but then goes away).

Most of this setup could be boiled down to a command line invocation, which git-annex could do at install time. However, it would still need to be run by the admin. Luckily the git-annex installation process already only works as admin (and I guess I could close the bug about that if it gets a legitimate reason to need admin..)

Some changes in git-annex would improve this.

* Maybe have a way to specify the user that git-annex is running on behalf of, and look in that user's home directory, rather than LocalService. (Other parts of the webapp UI, like adding a new repository, also use LocalService as the home directory, which is confusing).
* Starting the webapp for the first time to create a repository still opens a console window, so find a workaround for this.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm9ocq1Kb0WL-cz-LPpvd2Xm-q8tIQvqXA"
 nickname="Dominik"
 subject="comment 19"
 date="2014-06-10T18:24:22Z"
 content="""
That sounds great! I'll try it out on windows 8 aswell once i get around to it.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 20"
 date="2014-06-10T19:11:57Z"
 content="""
This seems pretty close to a command line to install the service with nssm.

<pre>
nssm install git-annex 'C:\Program Files\Git\cmd\git-annex.exe' assistant --autostart
nssm set git-annex AppExit 0 Ignore
nssm set git-annex AppExit 1 Ignore
nssm set git-annex AppDirectory C:\
nssm set git-annex AppNoConsole 1
nssm start git-annex
</pre>
"""]]


code:
The git-annex Windows port is beta, but rapidly becoming polished and
usable!

## status

* XMPP library not yet built. (See below.)

* Local pairing seems to fail, after acking on Linux box, it stalls.
  (Also, of course, the Windows box is unlikely to have a ssh server,
  so only pairing with a !Windows box will work.)

* gcrypt is not ported to windows (and as a shell script, may need
  to be rewritten)

* Incremental fsck sets the sticky bit to record when a file is fscked,
  and this is not done on windows, so fsck doesn't behave incrementally
  there.

* Deleting a git repository from inside the webapp fails "RemoveDirectory
  permision denied ... file is being used by another process"

## potential encoding problems

[[bugs/Unicode_file_names_ignored_on_Windows]] is fixed, but some potential
problems remain, since the FileSystemEncoding that git-annex relies on
seems unreliable/broken on Windows.

* When git-annex displays a filename that it's acting on, there
  can be mojibake on Windows. For example, "háčky.txt" displays
  the accented characters as instead the pairs of bytes making
  up the utf-8. Tried doing various things to the stdout handle
  to avoid this, but only ended up with encoding crashes, or worse
  mojibake than this.

* `md5FilePath` still uses the filesystem encoding, and so may produce the
  wrong value on Windows. This would impact keys that contain problem characters
  (probably coming from the filename extension), and might cause
  interoperability problems when git-annex generates the hash directories of a
  remote, for example a rsync remote.

* `encodeW8` is used in Git.UnionMerge, and while I fixed the other calls to
  encodeW8, which all involved ByteStrings reading from git and so can just
  treat it as utf-8 on Windows (via `decodeBS`), in the union merge case,
  the ByteString has no defined encoding. It may have been written on Unix
  and contain keys with invalid unicode in them. On windows, the union
  merge code should probably check if it's valid utf-8, and if not,
  abort the merge.

* If interoperating with a git-annex repository from a unix system, it's
  possible for a key to contain some invalid utf-8, which means its filename
  cannot even be represented on Windows, so who knows what will happen in that
  case -- probably it will fail in some way when adding the object file
  to the Windows repo.

* If data from the git repo does not have a unicode encoding, it will be
  mangled in various places on Windows, which can lead to undefined behavior.

## minor problems

* rsync special remotes with a rsyncurl of a local directory are known
  buggy. (git-annex tells rsync C:foo and it thinks it means a remote host
  named C...)
* webapp lets user choose to encrypt repo, and generate gpg key,
  before checking that gcrypt is not installed
* Ssh connection caching does not work on Windows, so `git annex get`
  has to connect twice to the remote system over ssh per file, which
  is much slower than on systems supporting connection caching.
* glacier-cli is not easily available (probably)
* user feedback: "Git on windows installer provides openssh 4.6. git-annex installer
  provides openssh 6.2 . This seems to create problems regarding how
  `known_hosts` file path is setup. Setting `GIT_SSH=` to the git-annex
  openssh version fixes the problem."
  However, I don't know how to determine what that location is after
  it's been installed. Maybe look for ssh.exe in same directory as
  git-annex.exe? --[[Joey]]

## stuff needing testing

* test that adding a repo on a removable drive works; that git is synced to
  it and files can be transferred to it and back
* Does stopping in progress transfers work in the webapp?

## trying to build XMPP

Lots of library deps:

1. gsasl-$LATEST.zip from <http://josefsson.org/gnutls4win/> (includes
   gnuidn and gnutls)
2. pkg-config from
   <http://sourceforge.net/projects/pkgconfiglite/files/latest/download?source=files>
3. libxml2 from mingw:
   <http://sourceforge.net/projects/mingw/files/MSYS/Extension/libxml2/libxml2-2.7.6-1/>
   both the -dll and the -dev
3. Extract all the above into the Haskell platform's mingw directory. Note
   that pkg-config needs to be moved out of a named subdirectory.
4. Run in DOS prompt (not cygwin!): cabal install network-protocol-xmpp

Current FAIL:

<pre>
Loading package gnutls-0.1.5 ... ghc.exe: internal error: Misaligned section: 18206e5b
    (GHC version 7.6.3 for i386_unknown_mingw32)
        Please report this as a GHC bug:
	http://www.haskell.org/ghc/reportabug
</pre>

<https://ghc.haskell.org/trac/ghc/ticket/8830>

Note: This only happens in the TH link stage. So building w/o the webapp
works with XMPP.

Options:

1. Use EvilSplicer, building first without XMPP library, but with its UI,
   and a second time without TH, but with the XMPP library. Partially done
   on the `winsplicehack` branch, but requires building patched versions
   of lots of yesod dependency chain to export modules referenced by TH
   splices, like had to be done on Android. Horrible pain. Ugly as hell.
2. Make a helper program with the XMPP support in it, that does not use TH.
3. Swich to a different XMPP client library, like
   <http://hackage.haskell.org/package/pontarius-xmpp>


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.145"
 subject="comment 10"
 date="2013-08-04T18:23:00Z"
 content="""
Encryption is now working on Windows.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 11"
 date="2014-06-16T23:48:03Z"
 content="""
service stuff moved to [[todo/windows_git-annex_service]]
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkRITTYYsN0TFKN7G5sZ6BWGZOTQ88Pz4s"
 nickname="Zoltán"
 subject="cygwin"
 date="2012-05-15T00:14:08Z"
 content="""
What about [Cygwin](http://cygwin.com/)? It emulates POSIX fairly well under Windows (including signals, forking, fs (also things like /dev/null, /proc), unix file permissions), has all standard gnu utilities. It also emulates symlinks, but they are unfortunately incompatible with NTFS symlinks introduced in Vista [due to some stupid restrictions on Windows](http://cygwin.com/ml/cygwin/2009-10/msg00756.html).

If git-annex could be modified to not require symlinks to work, the it would be a pretty neat solution (and you get a real shell, not some command.com on drugs (aka cmd.exe))
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawk5cj-itfFHq_yhJHdzk3QOPp-PNW_MjPU"
 nickname="Michael"
 subject="+1 Cygwin"
 date="2012-05-23T19:30:21Z"
 content="""
Windows support is a must. In my experience, binary file means proprietary editor, which means Windows.

Unfortunately, there's not much overlap between people who use graphical editors in Windows all day vs. people who are willing to tolerate Cygwin's setup.exe, compile a Haskell program, learn git and git-annex's 90-odd subcommands, and use a mintty terminal to manage their repository, especially now that there's a sexy GitHub app for Windows.

That aside, I think Windows-based content producers are still *the* audience for git-annex. First Windows support, then a GUI, then the world.
"""]]


code:
[[!comment format=mdwn
 username="http://xolus.net/openid/max"
 nickname="B0FH"
 subject="What about NTFS support ?"
 date="2012-08-02T17:45:10Z"
 content="""
Has git-annex been tested with an NTFS-formatted disk under Linux ? NTFS is supposed to be case-sensitive and to allow symlinks, and these are supposed to work with ntfs3g.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlc1og3PqIGudOMkFNrCCNg66vB7s-jLpc"
 nickname="Paul"
 subject="Re: What about NTFS support?"
 date="2012-08-16T19:30:38Z"
 content="""
I successfully use git-annex on an NTFS formatted external USB drive, so yes, it is possible and works well.
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/dASECLNzvckz4VwqUGYsvthiplY.#d2c27"
 nickname="A. D. Sicks"
 subject="comment 5"
 date="2012-09-09T23:48:21Z"
 content="""
Haskell has C++ binding, so it should be possible to port it to .net/Mono with a VB GUI for Windows users.  Windows has a primitive form of symlinks called shortcuts.  Perhaps shortcut support in Windows could replace the use of symlinks.  I've used shortcuts since XP to put my home Windows directory on another partition and never had a hitch...

If anyone is interested in working on this, hit me up.  I would like to use this in my XP vbox to have access to files on my host OS...I have a student edition of Visual Studio 2005 to do an open source port...
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnlwEMhiNYv__mEUABW4scn83yMraC3hqE"
 nickname="Sean"
 subject="NTFS symlinks"
 date="2013-01-11T21:44:21Z"
 content="""
It seems that NTFS (from Vista forward) has full POSIX support for symlinks. At least, Wikipedia [seems to think so.](http://en.wikipedia.org/wiki/NTFS_symbolic_link). What about doing like GitHub and using MinGW for compatibility? Cygwin absolutely blows in terms of installation size and compatability with the rest of Windows.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlpSOjMH7Iaz56v6Pr9KCFSpbvMXvg-y9o"
 nickname="Dominik"
 subject="So close :-)"
 date="2013-06-30T12:46:40Z"
 content="""
I was fighting my way forward until I read here that special remote with ssh+rsync and encryption doesn't work. Interestingly I got everything working so far, ssh login is keybased, gpg -k works and the remote setup also correctly cooperated with gpg... but it just didn't sync. Any ideas how complex it is to get this last missing piece moving?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.193"
 subject="comment 8"
 date="2013-06-30T17:58:08Z"
 content="""
It should be easy to fix whatever's wrong the the rsync special remote. Just a matter of debugging that.

Adding encryption support on Windows is stuck at a roadblock I don't know the way around. To drive gpg, git-annex uses the `--passphrase-fd` option, and sends the \"passphrase\" (really a big block of binary foo!) over a file descriptor of a pipe that it set up.

Windows, AFAIK, doesn't have file descriptors, or at least there is no equivilant to them that I have access to in the Haskell POSIX compatability layer for Windows. I am reluctant to fall back to using `--passphrase-file` on Windows, since that would be a massive security hole (as would passing the passphrase as a parameter via `--passphrase=`).
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlpSOjMH7Iaz56v6Pr9KCFSpbvMXvg-y9o"
 nickname="Dominik"
 subject="comment 9"
 date="2013-07-31T10:29:51Z"
 content="""
The tradeoff for me is a \"local\" security hole (where I can secure my own laptop) vs. a remotely exploitable thing... If it needs to go through a file, so be it -- it would however be good if that file would be overwritten with garbage before being deleted :-)
"""]]


code:
`git annex find` currently makes for a great way to find which files are already local, and don't need to get `git annex get` gotten; obviously `ls` just shows me all the files in a given directory, disregarding git-annex (and without recursing to subdirectories). I think that adding a '--maxdepth' option to `git annex find` would make it much easier to use at directories high up in the directory structure, since currently `git annex find` recurses all subdirectories necessarily, when I really just want to see whether or not there are git-annex files present from a given directory.

Obviously, since directories themselves are not git-annex objects, there is no way to say whether or not they are "present", but perhaps the most intuitive would be to say whether or not any git-annex files under a given directory are present.

For example, if I have:
./
+-- subdir0/
|  +-- file0 (present in local git-annex repo)
|  +-- file1 (present in local git-annex repo)
+-- subdir1/
|  +-- file0 (not present in local git-annex repo)
|  +-- file1 (not present in local git-annex repo)
+-- file2 (present in local git-annex repo)

and I type `git annex find --maxdepth 1 .`, the output might look something like:
subdir0/
file2

rather than:
subdir0/file0
subdir0/file1
file2


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 1"
 date="2014-07-21T17:28:53Z"
 content="""
`find --maxdepth` is a nice optimisation because it can short-circuit when it gets deep in the tree. However, `git annex find` is built on top of `git ls-files --cached`, which has no equivilant way to short-circuit. I am not sure if the format of the index makes it practical for it to get a --maxdepth option (it may need to traverse the whole index, or might be able to short-circuit).

I don't see any point in adding a --matdepth to git-annex if it doesn't actually make it any faster, so getting such a thing into `git ls-files` would be the first step. So, suggest filing a feature request on git.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlScsufvQF7s8TVTwPd-h_QiP5Hn_i-hrs"
 nickname="Jason"
 subject="comment 2"
 date="2014-07-21T18:37:19Z"
 content="""
I see your point, `git ls-files` may still have to walk the whole tree, precluding a speed advantage.
But I guess the point of what I was saying was more that a way summarize from a high level what is here and what is not would be nice.
I certainly understand if this is not something you see as worthwhile, but if someone were inclined to write a patch (if ever I find the time) that would add a `--maxdepth` option that would merely summarize the results of `git annex find` would it be something you would be inclined to include in the main repo (providing, of course, that you find the behavior sensible)?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 3"
 date="2014-07-21T18:40:00Z"
 content="""
I think that --maxdepth has a well-defined meaning and this summary option would need to be named something else.

I don't object to the idea of implementing it. However, I don't know that it would be very easy to implement either.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlScsufvQF7s8TVTwPd-h_QiP5Hn_i-hrs"
 nickname="Jason"
 subject="good point"
 date="2014-07-21T19:22:30Z"
 content="""
You make another good point `--maxdepth` is vague in this context...
I guess if we were to decide to come up with a summary option, it will have be named something else, like `--summary-depth`, where the default would be to represent all files of whatever depth, and specifying the option would take the output that would otherwise get from `git annex find <opts>`, truncate the paths to a certain depth, and then make a set thereof (to remove the many dups), that way any directory that had any files that would have been output by `git annex find <opts>`, that would also be at or above a certain depth, would be listed.

I think if I get a chance I'll try to implement something like this.

"""]]


code:
This option is related to `--time-limit`, but stops after processing X
bytes. For example, I often need to free up a certain amount of space on
a disk and want `git-annex move` to stop after it has copied a specified
amount of data:

    git annex move --to otherdisk --byte-limit 3G
    git annex drop --auto --byte-limit 500m
    git annex get --byte-limit 500m

I've been using some `annex.diskreserve` trickery now and then to
accomplish this, but it's a bit cumbersome and also not a very precise
way to specify the amount I want to copy/move/get. The last example
would also be a useful command to limit the traffic when I'm connecting
via mobile – get as much as possible, but don't blow the mobile quota.


code:
It would be very nice with an "advanced settings" for jabber and webdav support.

Currently XMPP fails if you use a google apps account. Since the domain provided in the email is not the same as the XMPP server.

Same goes for webdav support. If i have my own webdav server somewhere on the internet there is no way to set it up in the assistant.

[[!tag /design/assistant]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmWg4VvDTer9f49Y3z-R0AH16P4d1ygotA"
 nickname="Tobias"
 subject="Hooks too"
 date="2013-05-22T10:40:33Z"
 content="""
It would actually be very nice if this could be done with hooks too.

Especially with the new hook method.

Take this hook

	mega-hook = /usr/bin/python2 ~/sources/megaannex/megaannex.py

git-annex could make a request with either the parameter(or environment variable) \"getsettingsobject\" that could return. {\"username\": \"\", \"password\": \"\", \"folder\": \"\"}.

The point being git-annex can request from the hooks program what settings it takes, and give a web interface to set it. Then store the information in the creds folder(ew ew, that folder is unencrypted, oh well) and pass it to the hook on run.

The advantage being that users wouldn't have to edit a settings file manually (this is currently also the case for the IMAP special remote, which also requires a settings file).
"""]]


code: As suggested during the first Gitify BoF during DebConf13: Adding a way to have on-demand dropping of content in a given remote would allow a user to quickly free up disk space on demand while still heeding numcopies etc.


code:
A big part of my online use is done via a low-speed connection over my mobile phone, this is limited to 16KB/sec because I always use up my 500MB quota the very first day of the month. `;-/` So when I need to download big files, I first download them to my online server, then transfer the files to my laptop with git-annex. If I'm connected via GSM, this occupies all the bandwidth and everything else moves like a heavily sedated slug. So if I want to work via VNC or SSH, I have to terminate ongoing transfers with Ctrl-C and then hopefully remember to restart it when I work locally. I know git-annex is robust enough to handle this gracefully, but it would be really nice to have a continuous connection going on in the background, limited to a value I choose.

rsync(1) has a `--bwlimit` (bandwidth limit) where you can specify max download/upload speed in kilobytes/sec. It would be great if a similar option was integrated into git-annex. Thanks in advance.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
 nickname="Justin"
 subject="trickle"
 date="2013-01-22T22:44:52Z"
 content="""
not exactly integrated, but you can easily use trickle for this.

    trickle -d 50 git annex ...
"""]]


code:
[[!comment format=mdwn
 username="http://sunny256.sunbase.org/"
 nickname="sunny256"
 subject="Yay, trickle works"
 date="2013-01-23T01:36:21Z"
 content="""
Justin, thanks a lot! trickle(1) works great. I didn't know about this program, but I'm not surprised that such a program is available. It never ceases to amaze me what's possible in a *NIX environment.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 3"
 date="2013-01-24T01:55:10Z"
 content="""
In addition to trickle, the git-annex man page has examples of how to make rsync use --bwlimit

Something like trickle is needed to limit rates for remotes not using rsync, however.
"""]]


code:
I have data that has accompanying parity files.  This is supposed to add some
security to file integrity; however, it only works as long as the files are
available unencrypted.  In case of encrypted special remotes the existing parity files
won't be of any use if the encrypted versions of files get corrupted in the remote location.

Would it be worthwhile for git-annex to generate its own
parity files for the encrypted data in encrypted special remotes?


code:
I would like to use the automated AWS lifecycle rules to move the git annex files store on S3 to Glacier after a bit of time. Git annex need must support this kind of S3 files explicitly in order for it to work.

This is different from the adding a Glacier remote to git annex because of the reasons explained in <http://aws.typepad.com/aws/2012/11/archive-s3-to-glacier.html>.

Basically, the files moved by AWS from S3 to Glacier are not available under the normal Glacier API. In fact, the moved S3 files are listed as available but under the `GLACIER` storage class and need a RESTORE request before they can be GET like other S3 files. Trying to GET an S3 file that has been moved to Glacier will not restore it from Glacier and will result in an 403 error.

I suppose DELETE needs special care as well.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkAUMhKOSkh9JaBA6xst3XxQIIsDEq5Zd4"
 nickname="Ovidiu"
 subject="Keen to see this happen too!"
 date="2014-05-08T21:44:30Z"
 content="""
Or alternatively support for S3 Lifecycle Rules.
I'm not exactly sure what git-annex assistant does if it backs up into a S3 bucket with i.e. a rule to archive everything 1 month later into Glacier. Can it restore/access?
"""]]


code:
We're using git-annex to manage large files as part of a team.

We have a central repository of large files that everyone grabs from.

We would like to be able to get the files without updating the `git-annex` branch.  This way it doesn't pollute the history with essentially always unreachable locations.

I think the easiest way would be to just add an option to not update the `git-annex` branch when `annex get` is executed.

Thoughts?

> See [[untracked_remotes]] for a todo item that will probably
> be useful in this sitation. Since that describes better what
> this bug report seems to be asking for, I am closing this one.
> [[closed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.6.48"
 subject="comment 1"
 date="2013-07-17T23:23:50Z"
 content="""
I'm interested to hear that your team is using git-annex.

Have you tried `git config annex.alwayscommit false`? This will avoid committing, and just store the info in a local journal -- so even `git annex fsck` will still work.

Hmm, perhaps you want to update the branch when running `git annex copy` to put files onto the server, but not when getting them? A switch to disable updating the branch would then make sense. Any use of fsck would notice the inconsistency though, and commit a fix to the git-annex branch -- unless you also used the new switch when running fsck.

But what happens if someone makes a change, pushes to the server, but forgets to `git annex copy` the file? Everyone would then be left with a missing file that git annex doesn't know where it is. One of the reasons it tracks the locations is so that if necessary you know which repository such a misplaced fail is stored in. And can go track down that person's laptop and apply a cluebat. ;) Do you do something on the server to prevent this scenario?
"""]]


code:
[[!comment format=mdwn
 username="http://caust1c.myopenid.com/"
 nickname="asbraithwaite"
 subject="comment 2"
 date="2013-07-17T23:42:25Z"
 content="""
What would happen if we wanted to copy a file to the server
with that set? or even when running `git annex add`
You understood me exactly though.
We'd like to be able to get the files without a commit, but copy them
with a commit of the changes.
The way we operate, if somebody makes a change with regards to
largefiles, they should also add a test for it.
Ideally, the test suite would catch it.  That or people would
realize that theres a new big file and send around an email to
see who was trying to a largefile when they couldn't reach it.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.6.48"
 subject="comment 3"
 date="2013-07-18T00:12:39Z"
 content="""
Actually, when a file is sent to a git repository on the server, it will update the location log on that side. So even if the client has alwayscommit=false, other clients will learn the file has reached the server.

So that might just work for you.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="comment 4"
 date="2013-07-18T18:03:00Z"
 content="""
If you can have developers only add large files on the central server, avoiding using git annex sync and only pulling from git repo should work, I think.
"""]]


code:
[[!comment format=mdwn
 username="http://caust1c.myopenid.com/"
 nickname="asbraithwaite"
 subject="comment 5"
 date="2013-07-18T21:17:45Z"
 content="""
Wow.  This worked better than expected.  I'm still trying to understand the implementation (I'm not familiar with Haskell), but there is some magic going on.

To explain:

I didn't expect that when I added a file (Locally and to the annex remote) that when somebody else did a pull, git-annex would recognize this and update the working copy to download and include that file.

I'm not sure if this is intended behavior or not (since I thought all commands had to go through git-annex) but I like it nonetheless.

Thanks again for the tips!
"""]]


code:
[[!comment format=mdwn
 username="http://caust1c.myopenid.com/"
 nickname="asbraithwaite"
 subject="comment 6"
 date="2013-07-18T21:23:58Z"
 content="""
Disregard that.  I was testing it with a particular file that didn't exactly play nice.

Basically, to test this before production I used `dd bs=1024 count=100000 if=/dev/zero of=bigfile`.

When I did a `git pull`, it showed the symlink as being valid.

When I changed the command to `dd bs=1024 count=100000 if=/dev/urandom of=bigfile` then it showed the symlink as bad after pulling.

Weird!
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="comment 7"
 date="2013-07-18T21:48:06Z"
 content="""
If you wanted to auto-get files on git pull, you could trying putting git annex get into .git/hooks/post-merge (needs to be marked as executable).
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.140"
 subject="comment 8"
 date="2013-07-20T20:07:53Z"
 content="""
`dd bs=1024 count=100000 if=/dev/zero of=bigfile` obviously creates the same data each time, so if that same size empty file has previously been in a repository, and the content is still present there, the symlink will automatically point to it when it gets added. In other words, git-annex performs automatic deduplication of file contents, and so it was able to save you a download in this case.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 9"
 date="2014-01-02T17:25:50Z"
 content="""
I have now implemented a way to mark a remote as readonly. This prevents git-annex from pushing anything to the remote, or modifying it in other ways.

It seems that in your use case, you might want to avoid git-annex sync pushing the git-annex branch, but still push a branch like master. So far, it's up to you to decide which branches to manually push; readonly disables all pushing.
"""]]


code:
During the campaign adding a chunking feature to obscure filesize for encrypted files was added to the roadmap.  But there is still one potentially valuable set* of data that git-annex can help obscure: when you access your remotes.

This data can be used to determine when a user is actively using a remote, but if a remote is always accessed at the same time that data becomes less useful.  Somebody could still monitor total traffic over a long period and figure out that a remote was more active in a given week or month, but scheduling reduces the resolution of your access times and their data.  Maybe this isn't the most important feature to add, but it would be nice to have, and could possibly be built on top of the existing git-annex scheduler.  It could work by queuing inter-remote transactions ('get', 'copy', 'sync', etc.), so that jobs start at the same time every day, or even the same time and day every week.

Possible syntax examples:
###Setting up the schedule:
git annex queue schedule Monday:1730 (starts every monday at 5:30PM)

git annex queue schedule 1400 (starts every day at 2PM)

###Queuing git-annex commands:
git annex queue prepend sync (pretends 'sync' to the very front of the queue)

git annex queue append get file.ISO (appends to queue file.ISO for retrieval from a repo)

###Viewing/editing queue:
git annex queue view (view the current queue, jobs displayed with corresponding numbers)

git annex queue rm 20 (removes job 20 from queue)

\*The four I can think of are:

* File contents (solved by crypto)

* File size (solved on the remote by chunking, but total traffic usage can't be helped)

* User IP/Remote IP (solved by VPN - outside scope of git-annex, unless someone writes a plugin)

* Access times (obscured by a queue and scheduling)


code:
I've looked for this feature in the webapp but can't find it...

I mainly use the webapp and have been wondering 'how many copies of file X are there' and 'where are the copies of file Y'?
This is available in the commandline interface but it would be nice to have this in the webapp too.


code:
Also suggested during the first Gitify BoF during DebConf13:

`git annex drop` deletes immediately. In some situations a mechanism to tell git-annex "I would like to hold onto this data if possible, but if you need the space, please delete it" could be nice.

An obvious question would be how to do cleanups. With the assistant, that's easy. On CLI, at the very least `git annex fsck` should list, and optionally delete, that data.


code:
[[!comment format=mdwn
 username="http://cstork.org/"
 nickname="Chris Stork"
 subject="How should this interact with the trust model and location tracking?"
 date="2013-10-04T11:13:11Z"
 content="""
This could become complicated.  AFAIU, right now git-annex keeps track of files as either present or absent.  With this feature it's tempting to introduce a third state 'potentially dropped' (or 'dropped in a relaxed fashion') but do you then treat them as if they were dropped depending in wether they are on a trusted or untrusted repo?  Or maybe a potentially dropped file in a trusted repo is treated as a file in a semitrusted repo?  This becomes convoluted.  You also need a command to undrop a file in case you decide that you really want to keep it and in order to do this you need a command to see which files are up for relaxed dropping....

As an alternative approach maybe it makes sense to extend [[preferred content]] expressions to take file sizes and disk usage into account.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.243"
 subject="comment 2"
 date="2013-10-04T20:17:07Z"
 content="""
I don't think that a third state would be necessary. Actually dropping the file when it happens would need to do the same numcopies verification that `git annex drop` does now.

I agree it might be simpler to first improve the power of preferred content expressions. Unfortunately one thing that cannot be put in them is anything that probes the current state of the system. This is because repo A on machine X needs to be able to calculate the preferred content of repo B on machine Y.
But I could certainly add file size as a preferred content term, since that info is known throughout the network.
"""]]


code:
[[!comment format=mdwn
 username="erics"
 ip="76.10.136.8"
 subject="comment 3"
 date="2014-05-04T00:48:55Z"
 content="""
I think things would be simpler if a \"drop --relaxed\" file were to look to
the outside world just like one that was dropped without \"--relaxed\".
In particular, even if a file is dropped with \"--relaxed\":

  - the file's work-tree symlink should be broken synchronously by the
   \"drop --relaxed\" command (as opposed to only being broken later,
   if and when the file physically goes away)

  - other repos should no longer see the file as available from this repo

Basically, the idea is to add a third state, but **not** a user-visible one.
Rather, it should be a well hidden implementation detail, which doesn't
affect the conceptual model (very much like git's own distinction between
loose and packed objects).  Thus, *logically dropped* would be a
better name than *potentially dropped*.

Corollaries:

  - A logically (but not physically) dropped file should *not* count towards
    satisfying the numcopies limit, i.e. if some other repo has been asked
    to drop the file too

  - That in turn means that \"git annex drop --relaxed\" needs to satisfy a
    numcopies check at the time the user runs it; it's not enough to only do
    the check later, at physical-deletion time.  (At that point, there should probably be
    a second numcopies check.  I don't know whether the model requires it,
    but even if not, paranoia is good :-) )

  - If the user wants to use the file again, they have to \"git annex get\" it
    again, just like usual -- but if the file hasn't been physically deleted yet,
    the \"get\" will be nearly instantaneous, since the data won't have to be copied

One possible implementation would be to have \"drop --relaxed\" behave almost identically
to a non-relaxed drop -- do all the same safety checks, bookkeeping, etc.  The only
difference would be to have it rename the file at the end, rather
than deleting it outright.  (Logically dropped files could stay in their same
directory, but with a distinguishing filename, or they could be moved to a
parallel tree, e.g. *.git/annex/dropped*.  I don't have an opinion on that choice;
I've just picked one arbitrarily to keep talking about.)

\"get\" would simply search .git/annex/dropped before going off to remote
repos, and if the file is found there, would move (not copy) it back into
.git/annex/objects.

An alternative might be to set some kind of *logically dropped* flag, but
that would probably be a much more intrusive change; a lot of places in the
code would have to check the flag.  Doing it as a file rename would make for a much more
localized change; most of git-annex would completely ignore .git/annex/dropped, and just go
about its business as it has always done.

(It might be tempting to think of (or even implement) .git/annex/dropped as a
very low-cost remote, but that's not accurate; the semantics are different.)

I'm just starting to experiment with git-annex, so I can only hope that what
I'm saying isn't completely silly...

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="216.145.95.162"
 subject="comment 4"
 date="2014-05-19T16:56:15Z"
 content="""
erics, that all makes a lot of sense, except I don't know if there's actually a use case for a git-annex that behaves that way. It doesn't seem to solve the original use case.

I'd be inclinded to instead use the new metadata support. A file could have a tag that indicates it's not strongly wanted, and if git-annex get doesn't have enough space it could seek out and drop such files.
"""]]


code:
[[!comment format=mdwn
 username="erics"
 ip="76.10.136.8"
 subject="comment 5"
 date="2014-06-03T17:49:10Z"
 content="""
> It doesn't seem to solve the original use case.

It doesn't?  The OP requested:

> I would like to hold onto this data if possible, but if you need the space, please delete it

It looks to me as though my suggestion does just that -- or am I misunderstanding what they asked for?
"""]]


code:
[[!comment format=mdwn
 username="erics"
 ip="76.10.136.8"
 subject="Metadata vs "drop --relaxed""
 date="2014-06-03T18:20:50Z"
 content="""
[This isn't as much about my suggested implementation for \"drop --relaxed\" as about whether the feature is worth providing in the first place.  I'm not arguing strongly for it, actually; just continuing the discussion.]

> I'd be inclinded to instead use the new metadata support.

I see metadata as more for static attributes of a given file -- this thing is \"a picture\", \"related to project X\", \"from Mary\".  Thus, the combination of metadata plus preferred-content settings seems to me more suitable for static preferences (likely ones that implement some kind of policy, however informal); e.g. \"this repo wants pictures but not mp3s\", or \"Mary's stuff but not Alex's\".

\"drop --relaxed\", on the other hand, would be good for more ad-hoc usage: \"disk space is getting tight; hmm, I'm not using *foo* today, so git-annex, please delete my local copy of *${myrepo}/foo* -- but only as much as you have to, because I'm going to want it again tomorrow\".

One reason not to want to use metadata and preferred-content settings for such short-term, ad-hoc needs is that you then have to remember to go undo the changes later.  That's even worse if you had to add ad-hoc metadata, and now have to go delete it all again.  Undoing a \"drop --relaxed\", on the other hand, consists of a simple \"git annex get\".

"""]]


code:
Similar to how

    git commit -m 'foo'

works, if I run

    git annex sync -m 'my hovercraft is full of eels'

git annex should use that commit message instead of the default ones. That way, I could use [[sync]] directly and not be forced to commit prior to syncing just to make sure I have a useful commit message.


code:
I would like to have way, ideally a per-repo-cloud setting which syncs around automatically, to not have `git annex sync` autocommit staged additions.

I often have quite complex additions with a mix of `git add` and `git annex add` in various stages of completion; running `git annex sync` regularly to see what state the other repos are in should not autocommit if possible.

Richard


code:
[[!comment format=mdwn
 username="zardoz"
 ip="92.227.51.179"
 subject="comment 1"
 date="2014-05-13T20:23:11Z"
 content="""
This could be particularly useful for direct-mode repositories.
"""]]


code:
[[!comment format=mdwn
 username="zardoz"
 ip="92.227.51.179"
 subject="comment 2"
 date="2014-05-13T20:27:38Z"
 content="""
I filed a bug today, which boils down to a related issue.

http://git-annex.branchable.com/bugs/git_annex_list__47__whereis_and_uncommited_local_changes/
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 3"
 date="2014-05-16T19:02:08Z"
 content="""
Well, you could: git fetch --all; git annex merge; git push origin
"""]]


code: Make 'git annex import' for each imported file leave a symlink behind. One may consider this a bit nasty as this introduces symlinks out of the annex. There are also some things careful to consider, link to the annexed symlinks or into the .git/annex/objects store? the first breaks views, the second relies on implementation details. Shall these be absolute (i'd say yes) or relative (won't harm) links etc. But after all it gives a easier migration and possibly even some new usage to manage files outside of the annex. A sister-command for 'export' comes in mind, drop a symlink anywhere. Anyways, when you feel this is a good idea, keep it, otherwise just delete this idea. Thanks ;)


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 1"
 date="2014-07-15T18:53:47Z"
 content="""
Well, it's easy enough to make symlinks to content in a git-annex repository yourself if you want to. I don't see why this belongs in `git annex import`, which is too complicated already.
"""]]


code:
The commit messages made by git-annex are quite spartan, especially in direct mode where one cannot enter its own commit messages. This means that all that the messages say is "branch created", "git-annex automatic sync", "update", "merging" or little more.

It would be nice if git-annex could add at least the name of the repository/remote to the commit message. This would make the log a lot more clear, especially when dealing with problems or bugs.


code:
It could be useful for Linux users and package maintainers to have systemd .service file samples to launch the assistant and the webapp.

For multi-user systems, we could have a git-annex-webapp@username.service .

See [systemd documentation](http://www.freedesktop.org/software/systemd/man/systemd.service.html) for informations about those files.
See [archlinux wiki](https://wiki.archlinux.org/index.php/Systemd/Services) for .service examples.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="64.134.31.139"
 subject="comment 1"
 date="2013-10-19T15:33:18Z"
 content="""
Wouldn't this involve running it as root or a dedicated user? Neither is ideal. git-annex already uses .desktop files to auto-start when the user who is using it logs in.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmVE2R20dyfPQIzdi6urVUUAXtD6eeBsr0"
 nickname="Benoît"
 subject="comment 2"
 date="2013-10-19T15:43:36Z"
 content="""
 Not necessarily. I have a systemd instance handling my userland services. See [archlinux wiki systemd/User](https://wiki.archlinux.org/index.php/Systemd/User).
"""]]


code:
It is now relatively easy to build your own S3-compatible storage system with software such as Ceph radosgw or Openstack Swift.

So a way for users to specify their own S3 url would come pretty handy in the webapp's "add S3 remote" page.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 1"
 date="2013-12-29T20:16:43Z"
 content="""
Such S3 clones are supported if you make the repository at the command line. See [[S3_special_remote_documentation|special_remotes/S3]].

I think the question comes down to, is it worth adding UI in the webapp for this, or is it likely that someone who is using a S3 clone knows the endpoint url to use? Similarly, the webapp only offers box.com as a WebDAV remote, although there are many other DAV servers. What I'm trying to do with the UI is guide the user through finding a place to store their data, and covering every step as easily as possible.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkgy5Qf4bivpTfYPw_yh92OcDlesrVTREg"
 nickname="François"
 subject="comment 2"
 date="2014-01-12T20:31:44Z"
 content="""
I understand your concerns about keeping the UI straightforward.

It makes sense however to allow users to add a custom backend without using the CLI. What about a new \"Custom backend (expert only)\" option which directly asks for a string with the arguments to 'git annex initremote'?
"""]]


code:
A replacement for a web-browser's downloads menu that uses git-annex internally ([[`addurl`|tips/using the web as a special remote]] command for the download, and `drop` or `git rm` for the clean up) would be quite helpful:

say, when working on a topic, writing a paper or similar things, I usually have a Git repo with my text, all relevant data and processing code, and possibly other background information. It's nice to store the literature you needed at the same place where you work. (So that it is easy to catch up with what I was doing and thinking over when I left this work aside for a while, perhaps after cloning the repo from another location.)

When I find an interesting literature, I save the file to the directory with my work, and read it. Then I might return to it to re-read it. There might be references to this document from my work, so I'd like them to work as links (perhaps pointing at the local file, but also at the source URL for the case when my document is read by someone else not on my system).

I need to keep track of the source URLs for the documents I have saved which I read and use.

That's a task that fits well git-annex.

Note that doing the dull work of copying and pasting the URL and the downloading it and then opening it for reading is a pain to do every time I'm interested in a document I have found on the web. (Of course, I would need to fill out the bibliogrphic information for this document if I want to refer to it, but that can be done later. Initially, I wish I just don't lose the source URL of a document at the moment when I get interested in it and start reading.)

So, I could be assisted by a replacement of the "downloads" menu of, say, Firefox: whenever I want to open a file for viewing (like a PDF), it should ask me where to save it, and I'd choose the directory with my work, then it should register it with git-annex (so that the source URL is saved, and perhaps it should also write down the referring page's URL somewhere nearby automatically), download it, and open with a viewer for reading.

Then I'll have the interesting literature there when I'm offline; the source URLs would be saved, so that they can be put into the references. Also, if I distribute this work with Git, at another location git-annex can be used to easily get all the literature again.

(Hmmm... probably, the browser that this will be simplest for me to implement for is emacs-w3m; simply, some functions calling git-annex...)

> This seems fairly doable to implement since the git-annex [[design/assistant]]
> already has a webapp. So a javascript toolbar thing could be made that
> submits the current url (or maybe links dragged into it?) to the webapp
> for adding to the annex.
>
> The only wrinkle is that the webapp runs under a new url each time
> it starts, due to using a high port and embedding some auth token in the
> url. --[[Joey]]

[[!tag /design/assistant]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
 nickname="Justin"
 subject="comment 1"
 date="2012-09-25T23:55:44Z"
 content="""
You know about `git annex addurl` right?  It doesn't help with the browser integration (though I bet there are existing download manager extensions you could re-use for this) but it takes care of the other use cases.
"""]]


code:
[[!comment format=mdwn
 username="http://m-f-k.myopenid.com/"
 ip="93.207.162.28"
 subject="+1"
 date="2012-10-05T20:00:31Z"
 content="""
Copy+Paste+Open is to much for lazy people like me;)  I like the idea of a direct browser download manager integration.  This would make it so much easier to find find the original source of a downloaded file when you're to lazy to write it down somewhere in the first place …
"""]]


code:
[[!comment format=mdwn
 username="http://lj.rossia.org/users/imz/"
 ip="79.165.56.162"
 subject="the scheme to follow to use the addurl command is longer"
 date="2012-09-26T00:07:11Z"
 content="""
Justin, yes, I know. To use [[addurl|tips/using the web as a special remote]], I should force myself not to click on a link to view it, but rather to copy it, paste into the command line with addurl (then it will be downloaded, and I can run a command to view it...). Not terrible, probably learnable.
"""]]


code:
[[!comment format=mdwn
 username="andy"
 ip="99.48.75.171"
 subject="I've just written a script that does this (with a Firefox download manager)"
 date="2013-04-09T02:51:26Z"
 content="""
So I've just written a two-line shell script that makes [FlashGot](http://flashgot.net/) able to do this. It's available on GitHub at <https://gist.github.com/andyg0808/5342434>.

This could be even neater if Git-annex had the ability to set the download manager... Wishlist created: [[wishlist:_allow_configuration_of_downloader_for_addurl]]
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 5"
 date="2013-04-09T03:36:29Z"
 content="""
Wishlist implemented ;)

A full example of doing this would make a nice addition to [[tips]] ...
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 6"
 date="2013-04-11T15:12:00Z"
 content="""
See [[tips/Using_Git-annex_as_a_web_browsing_assistant]].

I am leaving this bug open for now, as some javascript that passes the url off to the webapp is still a nice idea to build.

A solution to the webapp's url always changing could be to have a file:/// url that the webapp creates, and javascript submits to, that then passes the request off to the webapp, probably by using additional javadcript in the html file. Assuming javascript security allows this.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 7"
 date="2013-04-11T15:41:16Z"
 content="""
A less round-about method would be to have the webapp listen for links dropped into its page. You could then have the webapp open in a tab, and just drag urls there to add them to the annex.

I'm not sure if it's possible to intercept drags into a tab. Default browser behavior is certainly to redirect the tab to the url dropped into it.

If someone finds out how to do this, I will build it.
"""]]


code: The `annex.largefiles` feature is very nice to mix annexed files with normal git managed files. I'd like to be able to configure this setting on the webapp and that the configuration directive would be synchronized accross all remotes.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 1"
 date="2013-11-05T16:46:28Z"
 content="""
It might make sense to sync this across remotes and have it edited with `git annex vicfg`

Putting it in the webapp would need some nice interface for constructing the underlying expression. Might be doable for at least simple filtering (ie, files larger than XX or with extensions .A, .B, .C). I tend to think of this as a setting for people comfortable with the command line though.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawne_amN4fko4p5cRY_9EYwaYuJKNn7LRio"
 nickname="Tobias"
 subject="feedback"
 date="2013-11-05T21:23:11Z"
 content="""
> It might make sense to sync this across remotes and have it edited with git annex vicfg

That would be great!

> Putting it in the webapp would need some nice interface for constructing the underlying expression. Might be doable for at least simple filtering (ie, files larger than XX or with extensions .A, .B, .C). I tend to think of this as a setting for people comfortable with the command line though.

I could live with a simple filtering interface without too many fancy stuff. The fancy stuff could be done on the command line if needed...
"""]]


code: It would be nice to have mimetype support on the `annex.largefiles` configuration directive. F.e. `git config annex.largefiles "not mimetype=text/plain"`


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlhE8Xar6m4x3JSILXdm-Y5KhwHTH5qzKQ"
 nickname="bruno"
 subject="+1"
 date="2013-10-14T14:43:11Z"
 content="""
Big +1 for this feature: this would really help configuring annex.largefiles
"""]]


code: An interesting feature, when an archived file cannot be removed from all clients because of the minimum number of copies required, would be to remove it from the repositories with the smallest amount of free space available.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.14.105"
 subject="comment 1"
 date="2013-09-19T21:01:30Z"
 content="""
That might be useful in some cases. git-annex can only tell how much free space is available on remotes that are mounted to the local filesystem. Did you use case involve removable drives?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmOsy6nbvPyXLd--qqjPMLnVIzxgZwtKlQ"
 nickname="Nicolas"
 subject="comment 2"
 date="2013-09-20T19:03:13Z"
 content="""
I was not thinking of removable drives, but only of \"client\" repositories. Ideally, git-annex would query the remaining space of all connected client repositories to choose on which repositories to drop a copy.
"""]]


code:
I handle some video and music files with git annex and it would be awesome if git annex supported tracking of relations between files.

Example:
I have music in flac on my htpc, but i think flac is way overkill to have on my Android device, so i transcode them to ogg/mp3 on my htpc and then sync them to my Android device.

Transcoding is a good example for this type of feature, but there might be other uses too.

The basic thing is to know a file is a "copy" of another file, but its a generated result, so the actual file is not important, but the source is.


code:
When using the [[/assistant]] on some of my repositories, I would like to
retain manual control over the granularity and contents of the commit
history.  Some motivating reasons:

* manually specified commit messages makes the history easier to follow
* make a series of minor changes to a file over a period of a few hours would result in a single commit rather than capturing intermediate incomplete edits

* manual choice of which files to annex (based on predicted usage) could be useful, e.g. a repo might contain a 4MB PDF which you want available in *every* remote even without `git annex get`, and also some 2MB images which are only required in some remotes

> This particular case is now catered to by the ["manual" repository group](/preferred_content/standard_groups/)
> in [[preferred content]] settings. --[[Joey]]

Obviously this needs to be configurable at least per repository, and
ideally perhaps even per remote, since usage habits can vary from machine
to machine (e.g. I could choose to commit manually from my desktop machine
which has a nice comfy keyboard and large screen, but this would be too
much pain to do from my tiny netbook).

In fact, this is vaguely related to [[design/assistant/partial_content]],
since the usefulness of the commit history depends on the context of the
data being manipulated, which in turn depends on which subdirectories are
being touched.  So any mechanism for disabling sync per directory could
potentially be reused for disabling auto-commit per directory.

According to Joey, it should be easy to arrange for the watcher thread not
to run, but would need some more work for the assistant to notice manual
commits in order to sync them; however the assistant already does some
crazy inotify watching of git refs, in order to detect incoming pushes, so
detecting manual commits wouldn't be a stretch.

[[!tag design/assistant]]

> You can do this now by pausing committing via the webapp,
> or setting `annex.autocommit=false`.
>
> The assistant probably doesn't push such commits yet.


code: It would be nice to have an indication of the status of the remotes in the webapp, for example with a field showing "In Sync", "Syncing", or the date of the last successful synchronization for unreachable remotes.


code:
[[!meta title="mass reinject of any known content from a directory"]]

Right now `git annex import DIR/*` will import all the files in DIR, both those that are already known to git-annex and those that are new. Using the option `--skip-duplicates` one can import only new files that are in DIR but unknown to git-annex.

It would be nice if there were an opposite `--only-duplicates` option that could be used to import only the files that are already known to git, ignoring the new files in DIR.

PS: it would also be nice to have aliases like `--only-new-files` and `--skip-new-files` for `--skip-duplicates` and `--only-duplicates`.


code:
[[!comment format=mdwn
 username="http://ypid.wordpress.com/"
 ip="213.153.84.215"
 subject="gitignore"
 date="2014-07-12T17:59:42Z"
 content="""
Hi

The gitignore file is probably what you are looking for which is also honored by git-annex. Some documentation:

* [On git-scm](http://git-scm.com/docs/gitignore)
* [On gitready](http://de.gitready.com/beginner/2009/01/19/ignoring-files.html)
* [On github](https://help.github.com/articles/ignoring-files)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 2"
 date="2014-07-14T18:30:38Z"
 content="""
You can use --clean-duplicates unless your goal is for some reason to add the duplicate files to your repository a second time.
"""]]


code:
[[!comment format=mdwn
 username="http://svario.it/gioele"
 nickname="gioele"
 subject="comment 3"
 date="2014-07-15T06:54:40Z"
 content="""
>  You can use --clean-duplicates unless your goal is for some reason to add the duplicate files to your repository a second time.

My use case is that I clone an existing remote on a PC where there are already some of the annexed files (+ others).

My workflow would be:

* clone git-annex server:~/Documents.git
* git annex init \"other pc\"
* git annex import --skip-new (or --only-duplicates) ~/Dump/*

~/Dump contains many other files in addition to those found in the Documents repository.

In this case --clean-duplicate would not be the correct solution.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 4"
 date="2014-07-15T19:01:24Z"
 content="""
So the goal is to inject any known objects from the dump into the local annex to avoid needing to re-transfer them.

It seems to me that in this case, you would not even want to create new symlinks in the git repository.

`git annex reinject` might be a better place to put code to handle this than `git annex import`.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 5"
 date="2014-07-15T19:13:05Z"
 content="""
A fundamental problem with this idea is that git-annex's keys can use any of many checksumming backends. So, which checksum should it try? Running every possible checksum on a file is going to re-read it repeatedly and be expensive.

`git annex import` avoids this problem by using whatever the default backend is configured to be for the filename it's importing. This is good enough to make repeated runs of `git annex import` work ok, but when we get into trying to reinject whole directory trees like this, I don't think that's good enough.
"""]]


code:
Given that git/config will have information on remotes and maybe costs, it might be a good idea to do a simple round robin selection of remotes to download files where the costs are the same.

This of course assumes that we like the idea of "parallel" launching and running of curl/rsync processes...

This wish item is probably only useful for the paranoid people who store more than 1 copy of their data.


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2011-04-03T16:39:35Z"
 content="""
I dunno about parrallel downloads -- eek! -- but there is at least room for improvement of what \"git annex get\" does when there are multiple remotes that have a file, and the one it decides to use is not available, or very slow, or whatever.
"""]]


code: It would be great to be able to do **private encrypted git remote on hosting site** and **multiuser encrypted git remote on hosting site** as explained in [[tips/fully encrypted git repositories with gcrypt]] through the webapp. I think it's a pretty common usecase that can be very useful for people not owning a proper server.


code:
### Current setup

ATM git-annex has

remote.<name>.annex-cost
remote.<name>.annex-cost-command  # command is not provided cmdline options by annex

to set the cost for a given remote.  That requires setting up one of those variables per each host, and possibly hardcoding options for the annex-cost-command providing e.g. the remote name.

### Suggestion

wouldn't it be more general and thus more flexible to have a repository-wide

annex.cost-command

which could take options %remote, %file and assessed accordingly per each file upon '--get' request to allow maximal flexibility: e.g. some files might better be fetched from remotes supporting transfer compression, some from the web, etc.  Also it might be worth providing %remote_kind ("special" vs "git") to disambiguate %remote's?


code:
git diff is not very helpful for annexed files.

How about a git annex diff command that allows to compare two versions of an annexed file?

Should be relatively simple, only there would have to be a way to deal with the situation where not both versions are present in the repository. Either abort with a message showing the command you need to run to get the missing version(s). Or even interactively volunteer to get it automatically, asking the user for confirmation.

Of course you wouldn't want to diff two large files, but with git annex assistant, all files are annexed by default (right?), so this would be useful.

There might already be a way to easily diff two versions of an annexed file which I'm missing -- in that case please point me to it! :)


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.154"
 subject="comment 1"
 date="2014-03-18T20:00:16Z"
 content="""
`git diff` is quite flexible; it can use external diff drivers to perform the diff. Someone could write a diff driver that knows about git-annex symlinks, and shows some kind of diff of the file contents (since the files are probably binary, this gets into how to display a diff of different file types..)
"""]]


code: The stats produced by `git annex info .` are nice but I often find myself separately looking up the actual numcopies set value.  Can this also be included in the report please?


code:
All repos contain some level of information about all other tracked repositories.
It would be nice if I could see that info, preferably with a timestamp telling me when the last sync happened.

`git annex info UUID/name` would be suggestion.

Thanks,
Richard


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 1"
 date="2014-01-02T02:16:10Z"
 content="""
git-annex vicfg shows everything (except location tracking info and some remote.log stuff that it would be hard to present in any useful way).

It would be good to make `git annex info` segment its list of repos by group. The only other information is location tracking info and scheduling, which I think vicfg gives a good overview of.
"""]]


code:
One piece of information that's sometimes useful, but not always, is to get a count of keys present in another remote plus the size of the remote.

Thus, I could verify that some repos are empty, archive repos have every single file, etc etc.

I still think that info is best suited for `git annex info name/UUID` as it's more volatile than what `git annex vicfg` displays.

Richard


code:
Hi,

I would love to have a page of "history" or "events" in the webapp. Similar to how Dropbox or Box show it.
I've been using git-annex for my personal files for a few months now, and I feel like this is the only feature missing to start using it in a production multi-user environment.

Thanks

[[!tag design/assistant]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlJEI45rGczFAnuM7gRSj4C6s9AS9yPZDc"
 nickname="Kevin"
 subject="I second this wishlist item"
 date="2013-07-24T16:48:59Z"
 content="""
I too would like to see a recent history of events in the assistant webapp.  A simple listing of what changed when and by whom would be sufficient.
"""]]


code:
### Please describe the problem.

`git annex reinject` refuses to work while in direct mode.

When in direct mode git annex reinject could simply perform `rm $symlink; mv $file_copy .; git annex add $file`. I prefer having git annex doing that so I am sure I am not messing up (mistakenly adding new files for instance) and everything is properly managed.

### What version of git-annex are you using? On what operating system?

git-annex 4.20130516.1

~~~~
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 12.04.2 LTS
Release:	12.04
Codename:	precise
~~~~

> [[fixed|done]]. Why did I take so long to do this, it was a trivial 1
> word change! --[[Joey]]


code:
Imagine this situation:
You have a laptop and a NAS.
On your laptop you want to consume a large media file located on the NAS.
So you type:

    git annex get --from nas mediafile

But now you have to wait for the download to complete, unless either

* rsync is pointed directly to the file in the object storage ("--inplace")
or
* the symlink temporarily points to the partial file during a transfer

which would allow you instantaneous consumption of your media.
It might make sense to make this behavior configurable, because not everyone might agree with having partial content (that mismatches its key) around.

So what do you say?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.154"
 subject="comment 2"
 date="2014-03-18T20:08:13Z"
 content="""
There's now an easy way to do this:

        git annex find --include=* --format='.git/annex/tmp/${hashdirmixed}${key}/${key}\n'

Pass it the file or files you're interested in to get their partially transferred contents.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.194"
 subject="comment 3"
 date="2012-11-04T19:58:28Z"
 content="""
I'm not at all comfortable with either idea. Temporarily repointing the symlink could lead to accidentially git committing a bad symlink. Or the user accidentially doing something with a partially transferred file. Running rsync in place would break lots of things that assume that, once the file is present, it can be assumed to be the full and correct file. (Obviously fsck doesn't assume that, but checks made by `git annex drop` do, for example.)

However, you can access partially transferred files by key in `.git/annex/tmp`. It would be easy to write some hack that looks at the symlink to get the key, and then spits out the name of the partial file in `.git/annex/tmp.`.
"""]]


code:
This is probably an extension of [[wishlist: more info in the standard commit message of `sync`]]:

It would also help debugging if the default commit messages listed, e.g., the name of all the files modified by that commit (or merge).

> No, it would not help debugging to put redundant info in commit
> messages. It will only make your repository take up more disk space.
> git log --stat will already show you the files changes by
> any commit.  [[wontfix|done]] --[[Joey]]


code:
Could you include the REPO and UUID information in the "automatic sync" commit message?

This would make troubleshooting easier. (not there was much trouble with git-annex!)


code:
[[!comment format=mdwn
 username="http://olivier.mehani.name/"
 nickname="olivier-mehani"
 subject="comment 1"
 date="2014-01-15T02:26:31Z"
 content="""
See also [[wishlist: more info in commit messages in general]] for a similar but more generic wish.
"""]]


code:
Right now the assistant can have a huge list of pending transfers for certain hosts if its data is a bit outdated, or a host hasn't been synced lately. When starting up it will then attempt each transfer to said host (which will in turn fail, but at times take time to time out), possibly before doing other stuff like attempting to download new files, or copy files to online hosts.

I suggest that if a transfer fails for host X, and there are other pending transfers, say to host Y and from Z, then all other pending transfers to/from X gets pushed to the back of the queue, to avoid having to wait a long time for several transfers to time out before doing useful stuff.

The prime example for me was this morning, when a laptop that was turned off had a huge amount of queued transfers to it, resulting in the assistant attempting a load of transfers to that host before it retrieved a new file that I had created on another machine yesterday.

[[!tag design/assistant]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.6.49"
 subject="comment 1"
 date="2012-12-01T19:22:03Z"
 content="""
There are several difficulties with reordering the queue that way. One is that the failure may be intermittent; another is that the queue is fed by a scanning process, so doesn't always have a well-defined end.

Another way to deal with this problem, which I think I prefer, is to allow multiple actions from the queue to run at once. Then slow or unreachable remotes don't block it from using other remotes.
"""]]


code:
[[!comment format=mdwn
 username="EskildHustvedt"
 ip="84.48.83.221"
 subject="comment 2"
 date="2012-12-01T19:31:18Z"
 content="""
I agree your method might be preferable, the end result is the same, and would have avoided the issues I had (and, of course, running multiple transfers at once has other benefits as well).

An alternate way would be to push every transfer NOT from host X to the front of the queue (avoiding most of the \"no defined end\" issue and largely solving the problem), but if multiple actions at once is feasible then that'd still be much nicer.
"""]]


code:
It would be nice if the 'unused' command could optionally display info about the actual files behind its cryptic keys.

I created a (very rough) bash script that simply splices in some info from git log -S'KEY' --numstat into the unused list, like so:

    arand@mas:~/annex(master)$ bash ~/utv/scripts/annex-vunused
    unused . (checking for unused data...) (checking master...) (checking synced/master...) (checking origin/HEAD...) (checking seagate/master...)
    Some annexed data is no longer used by any files:
    NUMBER  KEY
    1       SHA256E-s1073741824--49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14.img
                    8f479a4 Sat Feb 23 16:14:12 2013 +0100 remove bigfile
                    0       1       dummy_bigfile.img
                    2988d18 Sat Feb 23 16:13:48 2013 +0100 dummy file
                    1       0       dummy_bigfile.img
    (To see where data was previously used, try: git log --stat -S'KEY')To remove unwanted data: git-annex dropunused NUMBER
    ok
The script:

    #!/bin/bash

    pipe="$(mktemp -u)"
    mkfifo "$pipe"

    git annex unused  >"$pipe" || exit 1 &

    while read -r line
    do
    	key="$(echo "$line" | sed 's/^[^-]*-\([^-]*\)-.*/\1/')"
    	echo -n "$line"
    	test -n "$key" && \
    		echo && \
    		git log --format="%h %cd %s" --numstat -S"$key" | \
    			sed '/^$/d;/git-annex automatic sync/,/^ /d;s/^/\t\t/'

    done < "$pipe"
    rm "$pipe"

It would be nice if something like this was available as an option, since it's good way to get a quick overview of what the content is, and if it's safe to drop it.


code:
The metadata storage for direct mode (V3) is this. In directory .git/annex/objects, there is one .map for all annexed file, and one .cache for all files in the working tree. Both are small files, containing only 1 line or a few lines. I have a repo with lots of photos, and this created lots of small files. I believe this will cause many performance issues.

It would be great if these files are packed, maybe also in the git pack files format.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.35"
 subject="comment 1"
 date="2014-01-04T19:18:16Z"
 content="""
\"I believe this will cause many performance issues.\"

Such as?
"""]]


code:
[[!comment format=mdwn
 username="namelessjon"
 ip="193.132.159.169"
 subject="inode starvation"
 date="2014-01-09T15:30:16Z"
 content="""
This happened to me a few times when creating new annex folders.

I use lvm to create virtual partitions and have/had several 'bulk media' logical volumes which used ext4 with -T largefile4 (i.e. one inode per 4mb) because they're storing files with 20mb+ sizes (RAW images, downloaded screencasts, FLAC soundfiles, etc). Between the extra directories git annex creates, the extra files, and the .git/object directory, I ran out of inodes on a few occasions from the profusion of small files.  In some cases, I worked around this by shunting data around, or adding incrementally and then 'git gc'ing a lot to at least have a small .git/objects dir.  A packed metadata would help to deal with this.
"""]]


code:
Currently, when `fsck`'ing a remote, files are first downloaded to a temporary
file locally, decrypted if needed, and finally digested; the temporary file is
then either thrown away, or quarantined, depending on the value of that digest.

Whereas this approach works with any kind of remote, in the particular case
where the user is granted execution rights on the digest command, one could
avoid cluttering the network and digest the file remotely. I propose the
addition of a per-remote git option `annex-remote-fsck` to switch between the
two behaviors.

There is an issue with encrypted specialremotes, though. As hinted at
[[here|tips/beware_of_SSD_wear_when_doing_fsck_on_large_special_remotes/#comment-70055f166f7eeca976021d24a736b471]],
since the digest of a ciphertext can't be deduced from that of a plaintext in
general one would needs, before sending an encrypted file to such a remote, to
digest it and store that digest somewhere (together with the cipher's size and
perhaps other meta-information).

The usual directory structure (`.../.../{backend}-s{size}--{digest}.log`) seems
perfectly suitable to store these informations. Lines there would look like
`{timestamp}s {numcopy} {UUID} {remote digest}`. Of course, it implies that
remote digest commands are trustworthy (are doing the right thing), and that
the digest output are not tampered by others who have access to the git repo.
But that's outside the current threat model, I guess.

Actually, since git-annex always includes a MDC in the ciphertexts, we could do
something clever and even avoid running a digest algorithm. According to the
[[OpenPGP standard|https://tools.ietf.org/html/rfc4880#section-5.14]] the MDC
is essentially a SHA-1 hash of the plaintext. I'm still investigating if it's
even possible, but in theory it would be enough (with non-chained ciphers at
least) to download a few bytes from the encrypted remote, decrypt those bytes
to retrieve the hash, and compare that hash with the known value. Of course
there is a downside here, namely that files tampered anywhere but on the MDC
packets would not be detected by `fsck` (but gpg will warn when decrypting the
file).

My 2 cents :-) Is there something I missed? I suppose there was a reason to
perform `fsck` locally at the first place...


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.63"
 subject="comment 1"
 date="2013-08-22T15:18:35Z"
 content="""
The only reason fsck is done locally for remotes is ease of implementation and it being a generic operation that supports any kind of special remote.

Seems that the the only types of remotes where a remote fsck is a possibility are some rsync remotes and git remotes.
git remotes already have git-annex installed, so the fsck could be run locally on the remote system using it.

I don't know if I see a benefit with the MDC check. Any non-malicious data corruption on the remote is likely to affect the body of the file and not the small portion that holds the MDC. So checking the MDC does not seem much better than the current existence check done by `git annex fsck --fast --from remote`.

As for storing the remote digest on the git-annex branch, my initial reaction was just that it's potentially a lot of bloat. Thinking about it some more, when using non-shared encryption, there is currently no way, given just a clone of a git repository, to match up files in git with encrypted objects stored on a special remote. So storing the remote digest might be considered to weaken the security.
"""]]


code:
[[!comment format=mdwn
 username="guilhem"
 ip="129.16.20.209"
 subject="comment 2"
 date="2013-08-22T16:56:55Z"
 content="""
Oh yeah, the MDC paragraph was pretty much pointless indeed. Oops :-P

I agree that this would potentially add some noise to the index, and weaken the
security, but depending on the threat model and people's preferences that's an
option that's worth considering IMHO.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="68.119.96.78"
 subject="comment 3"
 date="2013-12-21T16:27:16Z"
 content="""
There is now a remoteFsck action in the Remote data structure. So far it is only implemented for git remotes, and `git annex fsck` does not use it; only the assistant does.
"""]]


code:
Based on an irc conversation earlier today:

19:50 < warp> joeyh: what is the best way to figure out the (remote) filename for a file stored in an rsync remote?

20:43 < joeyh> warp: re your other question, probably the best thing would be to make the whereis command print out locations for each remote, as it always does for the web special remotes


code:
I think it would be interesting to have a way to recursively import a local directory without actually moving files around. And to be able to checksum these files as well (without moving them into the annex).

This would work somewhat similar to looping over a directory and adding file:// remotes for each file.

A use case is importing optical media (read-only), whilst keeping that media as a remote, and being able to calculate checksums directly without moving any files around.

For single files, it would also be interesting if addurl had a "--localchecksum" option that would only work for file:// urls, and make it checksum files directly from their source location?)


code:
[[!comment format=mdwn
 username="https://launchpad.net/~arand"
 nickname="arand"
 subject="comment 1"
 date="2013-03-10T23:29:55Z"
 content="""
Recursively adding urls is already feasiable with some simple scripting:

[https://gitorious.org/arand-scripts/arand-scripts/blobs/master/annex-importdir](https://gitorious.org/arand-scripts/arand-scripts/blobs/master/annex-importdir)

but this is obviously missing the checksumming bit though.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-03-11T03:04:44Z"
 content="""
See also: [[forum/Managing a large number of files archived on many pieces of read-only medium (E.G. DVDs)]]
particularly [[this comment|forum/Managing a large number of files archived on many pieces of read-only medium (E.G. DVDs)#comment-908dbe02f29e011f030bba4ab5ef73d1]]
"""]]