However, in my experience, this solution is noticeably racey. When using IMAP with the IDLE command set, some mail clients (thunderbird, probably others) will under certain circumstances immediately tell the mail server to mark the message as unread and not new. In maildir, this moves the message for new/ to cur/. Thus, if this happens before the server gets around to executing the mv command, the move fails and the message is not marked as read.

Truthfully, procmail should be enhanced to allow you to process a message marked as read in an atomic fashion. But that doesn’t seem likely, so the hack is to tell the mv command to look in both the cur/ and new/ directory. Doing so is straightforward:

Put this in ~/.procmailrc_mark_as_read_maildir:

:0
{
:0c # store as new
$folder/

:0 # move to cur/ and append a ,S to mark as read
* LASTFOLDER ?? /\/[^/]+$
{ tail=$MATCH }

TRAP=”mv -v $LASTFOLDER $folder/cur/$tail 2> /dev/null ; mv -v $folder/cur/${tail}* $folder/cur/$tail:2,S”

HOST
}

Then in your normal procmailrc, the usage is this:

:0
* someregexhere
{
folder=Maildir/.myfolder
SWITCHRC=$HOME/.procmailrc_mark_as_read_maildir
}

Now it will mark the message read whether or not your client is IDLEing in that folder. It will also log that it has done so, if you have logging enabled.

The solution isn’t perfect, as if you mark the message read inbetween the moves, it will log bogus errors; at least it will now mark the mail correctly.

UPDATE:

I’ve been asked by a few people how a remote host running IMAP can possibly win the race with procmail executing mv sequentially after initially creating the file. My IMAP server, dovecot, uses inotify to watch for new messages in maildirs. So when the kernel writes out the file, it notifies dovecot who immediately notifies its clients who mark the message as unread but not new (assuming they’re IDLEing that folder). Now, true, there is some network latency, but since dovecot notices the new mail immediately it is plausible that the network latency will be less than the system latency in exec()ing a new command, especially if load is moderate or high.

I don’t agree.

Her implied assumption is that configuration data is easily separated from the OS, an assumption that is suspect, at best. In UNIXy systems, all configuration data is supposedly stored in /etc, and /etc is supposed to contain only configuration data. In reality, that line is hard to draw, and harder still to enforce.

Consider cron jobs (regularly executed scripts and programs) stored in /etc/cron*. I’m sure Windows has a similar concept. Some cronjobs are scheduled by administrators, some by users, and some by OS distributors. Some which are shipped by the OS, such as logwatch, are often modified by administrators. Modifying system configuration files and settings is difficult under a read-only OS scheme.

Presumably it should be possible to merge such changes via a read-write layer on top of the read-only system files, via UnionFS or similar. This is harder than it seems; to do the merge, the update mechanism needs to know the state of the file as shipped with the older version of OS, as modified by the user, and as shipped with the newer version of the OS. An update mechanism which relies solely on image-based updates would not have access to that information. Contrast with conary, which has access to a networked repository containing every version of every file ever shipped in a given OS. This allows conary to do intelligent merging of changes, preventing a new (and potentially critically important) configuration option from either the OS or the user from being lost on update.

There are some ways to mitigate this failing of image-based update schemes. The OS can be redesigned to have, as Linux distro maintainers call them, dot-d directories for every possible configuration option. So, instead of having a single file, you have a directory (usually ending in .d, hence the name) which can contain both user- and system-defined options. However, this clutters the filesystem and doesn’t address what would happen should the filenames ever collide. Another option is to redesign every system component so the OS defaults are built in to the programs themselves, leaving /etc only for user-defined configuration. That would be *alot* of work, and would make it harder for users to see which config options the system is using by default. Presumably this is what folks like TiVO do, but the number of user-configurable options TiVO has is laughable compared with the options needed by an enterprise (or even a small/medium business) deploying a software appliance. Enterprises need to be able to change things like login schemes, security and auditing models, configuring for external databases, SAN- and NAS-based storage, etc. Addressing all these intelligently in an image-based setup would be very difficult.

An image-based solution simply can’t keep up with my needs. So, for now, I’m very happy using conary to manage my appliances… long live conary!

The “patch” command basically assumes that we have no version control. Thus, if it detects anything even slightly amiss (such as a line number offset) it makes a backup of the file, even if it successfully patches the original. This backup usually ends in a ~, giving, for example, “Makefile~” as the backup of a patched “Makefile”. When mercurial adds the files to the repo, it includes *~. So if the patch you want to port touches any of the same files, it will modify the *~ file, causing mercurial to believe that a valid change was made, and thus to include that change in the diff it produces after you’re finished hacking on the sources.

The fix is very simple:

#!/bin/sh
# start up the repo
hg init
# remove files who’s existence is only due to patch doing backups
find . -name ‘*~’ -delete
# add all files to the repo, but don’t be overly verbose about it
hg addremove > /dev/null
# commit all files to the repo
hg ci -m “initial commit”

Of course, this also has the benefit of making the checkout a wee bit cleaner while working on it

The problem I was having was one of control. When approaching a bug in a package who’s codebase was unfamiliar, as I was often required to do, it would naturally take a number of attempts to get a bugfix correct. My build dir was often getting overwritten, causing my changes to unintentionally disappear. Further, when making changes, it was a pain to copy around build trees to get good diffs, especially on large packages, and it was impossible to undo something if I made a mistake.

Enter mercurial. Because hg has a totally distributed model, it is trivial to set up repositories, allowing one to use them essentially as scratch space. Now, every time I work on a package, I “cvc cook –prep” it to get the sources, and then run this script (if, indeed, you can call it that):

#!/bin/sh
hg init
hg addremove > /dev/null # addremove is overly verbose
hg ci -m “initial commit”

Then I hack on the package to my heart’s content without worrying what I’m screwing up in the process. I save “checkpoints” by doing hg commits, then later I can generate a diff from the initial commit all the way to tip, or to any point in the tree. When I’m done, I generate a patch and commit it to the conary repo. The hg tree is blown away by the next cvc cook, or by a tmpwatch cronjob that watches over my build dir. This allows me to not worry about forgetting parts of the changes I’ve made as it is all in the repo, and allows me to not worry about whether upstream patches or other applied patches conflict as they are all included in the first hg commit.

This method has the added benefit of making patches cleaner. Hg always generates a minimal unified level 1 diff. Always. No matter where you are in the tree or what nastiness you’ve done, making the patch both easier for humans to read and easier for cvc to apply.