emlx files, or something else? (was mail storage formats (was Getting ready to implement the UI))

[Tim Gray]

They’d show up in Spotlight and elsewhere as Messages rather than
Documents. I know the Spotlight / QuickLook integration angle was covered
earlier but I think it’s worth pointing out that .emlx files get special
treatment in the UI.

Good point. This isn't something that is accessible by non-Apple
developers?

[Michael McCracken]

They’d show up in Spotlight and elsewhere as Messages rather than
Documents. I know the Spotlight / QuickLook integration angle was covered
earlier but I think it’s worth pointing out that .emlx files get special
treatment in the UI.

Good point.  This isn't something that is accessible by non-Apple
developers?

I bet if you have "public.message" and "public.email-message" in
kMDItemContentTypeTree, it'll get treated as a message.

run mdls on one of your .emlx files and you'll see what other info
Apple's importer fills in, it's interesting.

-mike

[Michael McCracken]

On Fri, Jan 22, 2010 at 4:37 PM, Michael McCracken

Post by Michael McCracken

They’d show up in Spotlight and elsewhere as Messages rather than
Documents. I know the Spotlight / QuickLook integration angle was covered
earlier but I think it’s worth pointing out that .emlx files get special
treatment in the UI.

Good point.  This isn't something that is accessible by non-Apple
developers?

I bet if you have "public.message" and "public.email-message" in
kMDItemContentTypeTree, it'll get treated as a message.

I should have included this -
what I see is that the (singular) content type is emlx:

kMDItemContentType = "com.apple.mail.emlx"

but the tree has several types:

kMDItemContentTypeTree = (
"com.apple.mail.emlx",
"public.data",
"public.item",
"public.email-message",
"public.message"
)

and only one of those is apple-specific.

Post by Michael McCracken
run mdls on one of your .emlx files and you'll see what other info
Apple's importer fills in, it's interesting.
-mike

[Tim Gray]

Post by Michael McCracken
I should have included this -
kMDItemContentType = "com.apple.mail.emlx"
kMDItemContentTypeTree = (
"com.apple.mail.emlx",
"public.data",
"public.item",
"public.email-message",
"public.message"
)
and only one of those is apple-specific.

For reference, a Mailsmith message looks like the following. Note there is
no quick look preview for Mailsmith messages, though they do show up in
Spotlight searches if you limit it to kind:email.

kMDItemContentType =
"com.barebones.mailsmith.messagedata"

kMDItemContentTypeTree = (
"com.barebones.mailsmith.messagedata",
"public.data",
"public.item",
"public.email-message",
"public.message"
)

[Olivier Scherler]

The maildir filenames *are* sensible, just not very human readable.
It's a unique name constructed from a time() call, gethostname(),
plus some other stuff. The crap at the end are the message flags -
lets you get the status of each message in the mailbox by just doing
a dir listing, instead of opening and reading files.

What bothers me is that the message file name changes every time you
change a flag. Then Time Machine will back it up again, that pretty
inefficient, and you still have a database with the rest of the
metadata.

Ö.

[Tim Gray]

What bothers me is that the message file name changes every time you change
a flag. Then Time Machine will back it up again, that pretty inefficient,
and you still have a database with the rest of the metadata.

It's not *that* inefficient. And I do want the status of my messages backed
up.

I get about 300-400 emails a day delivered into maildir. I also have these
messages delivered into 'backup' mbox files. My Time Machine backups are
about 30-34 mb each time when I'm not doing real work on my computer. And
most of that is not mail.

[Olivier Scherler]

Post by Tim Gray

Post by Olivier Scherler
What bothers me is that the message file name changes every time
you change a flag. Then Time Machine will back it up again, that
pretty inefficient, and you still have a database with the rest of
the metadata.

It's not *that* inefficient. And I do want the status of my
messages backed up.

My point was that if you do something to a message, then you rebackup
it if it changed the flags stored in the file name and rebackup the
database with metadata anyway.

But you're right, it's not that bad.

Ö.

[Tim Gray]

My point was that if you do something to a message, then you rebackup it if
it changed the flags stored in the file name and rebackup the database with
metadata anyway.

You are correct. However, it's only 'read', 'flagged', and a couple of
other flags, not everything under the sun. Tangent - is Time Machine smart
enough to rename a file if I rename the original? Or does it delete the
backup and rebackup the moved/renamed file?

[John Gruber]

You never break the file into multiple parts. You keep a "pure" version of the file around. You'll never risk losing an attachment. It's basically the same(ish) concept you see in Aperture with RAW Files. You always keep your master file around.

I rather like this idea conceptually -- that the message data that comes from the server is sacrosanct, never modified directly. We store modifications and whatever additional metadata externally, in sidecar files or a database or whatever. Obviously we want our own Letters-specific metadata to be rock-solid, stable, reliable, etc. But I think the stability of the raw original message data is a bedrock.

Lightroom does the same thing, too. I don't think it ever writes changes to the original files you import from your camera. That way, the worst case scenario for data loss caused by the app itself is a loss of metadata, not the raw data.

—J.G.

[Tim Gray]

Post by John Gruber
Lightroom does the same thing, too. I don't think it ever writes changes to
the original files you import from your camera. That way, the worst case
scenario for data loss caused by the app itself is a loss of metadata, not
the raw data.

As a fellow photographer, the idea appeals to me as well. However the big
difference is that the email format is documented. RAW files aren't. I
think you can safely add a header to an email and resave it to the server...

That being said, I advocate NOT stripping attachments out of a message's
body by default. It's an interesting command to offer, but when emails come
in and are saved, the attachments should stay encoded in the message, even
if they are saved to a folder.

[Brad Lindsay]

Post by Tim Gray
That being said, I advocate NOT stripping attachments out of a message's
body by default. It's an interesting command to offer, but when emails come
in and are saved, the attachments should stay encoded in the message, even
if they are saved to a folder.

I agree. This was one of the things that bugged me about Eudora. Of
course, since I was using POP then, it also made changing email
clients more difficult.

Brad

[Wade Maxfield]

Post by Brad Lindsay

Post by Tim Gray
That being said, I advocate NOT stripping attachments out of a message's
body by default. It's an interesting command to offer, but when emails come
in and are saved, the attachments should stay encoded in the message, even
if they are saved to a folder.

I agree. This was one of the things that bugged me about Eudora.

And I have to disagree. 8) That's the thing I miss most from Eudora.

I have clients that send crap loads of email with useless attachments.
With Eudora I could purge the attachments folder, while still keeping
the original email (which I use for my history of communication and
tasks - and ultimately how I remember what to bill for).

These attachments vary from 30KB to a couple of MBs, and do add up, and
Thunderbird can't always strip them from the original message (plus
there's no way to do it to a batch of emails as a single action).

- Wade

[John C. Welch]

Post by Wade Maxfield

Post by Brad Lindsay

Post by Tim Gray
That being said, I advocate NOT stripping attachments out of a message's
body by default. It's an interesting command to offer, but when emails come
in and are saved, the attachments should stay encoded in the message, even
if they are saved to a folder.

I agree. This was one of the things that bugged me about Eudora.

And I have to disagree. 8) That's the thing I miss most from Eudora.
I have clients that send crap loads of email with useless attachments.
With Eudora I could purge the attachments folder, while still keeping
the original email (which I use for my history of communication and
tasks - and ultimately how I remember what to bill for).

But then you don't have the original email. You have part of the original
email. It seems picky, but the distinction is important. The 'original'
email includes the attachment.

[Tim Gray]

Post by John C. Welch
But then you don't have the original email. You have part of the original
email. It seems picky, but the distinction is important. The 'original'
email includes the attachment.

Agreed. An option to re-save the email without the attachment is enough.
And you should be able to run that action on several messages at once.

[Caio Chassot]

Post by John C. Welch
But then you don't have the original email. You have part of the original
email. It seems picky, but the distinction is important. The 'original'
email includes the attachment.

Agreed. An option to re-save the email without the attachment is enough. And you should be able to run that action on several messages at once.

So you want to delete the attachments both locally and from the server, so that the pristine full message is lost forever? This sounds bad.

I've always assumed deleting attachments would be supported, but I like the idea of not touching the raw message. And if we open an exception for attachments, gateway drug something something.

But attachments do add up. Most (as in more than 50%) of my email bytes go to attachments. And most of them as completely useless attachments. But then again most of messages with completely useless attachments are useless themselves, so could be deleted. Unless you're attached (awful pun) to your mail corpus as an unbreakable entity.


All that is to say:
- Pristine raw messages are very compelling
- Deleting attachments will be much desired
- \o/

[Tim Gray]

Post by Caio Chassot
So you want to delete the attachments both locally and from the server, so
that the pristine full message is lost forever? This sounds bad.

I don't personally want it. But I'm sure others find it a useful *option*.
But it's a command the user has to run. If the user wants to delete an
attachment, let him. Are we going to stop him from deleting emails too? :D

[Tony Meyer]

Post by Caio Chassot
So you want to delete the attachments both locally and from the server, so that the pristine full message is lost forever? This sounds bad.

Forgive me if people are aware of this, but it seems worth pointing
out that if Letters is going to store a "pristine full" copy of all
messages, then this implies other things about how it will retrieve
messages.

For example, if you wanted a 'low bandwidth' mode (whether plug-in or
not), it could fetch only the (MIME) parts of the message as they are
required (e.g. download attachments when requested, skip alternative
parts unless requested, etc). IMAP is well suited to this - it's not
a RETR/TOP protocol like POP3. From the protocol's point of view, the
pristine copy of the message is on the server.

I'm not saying that Letters shouldn't always fetch a full copy of the
message - just that this decision should probably be kept in mind when
considering local cache formats.

John C. Welch is absolutely right every time he says that IMAP is complex.

Cheers,
Tony

[Caio Chassot]

Post by Tony Meyer

Post by Caio Chassot
So you want to delete the attachments both locally and from the server, so that the pristine full message is lost forever? This sounds bad.

Forgive me if people are aware of this, but it seems worth pointing
out that if Letters is going to store a "pristine full" copy of all
messages, then this implies other things about how it will retrieve
messages.
For example, if you wanted a 'low bandwidth' mode (whether plug-in or
not), it could fetch only the (MIME) parts of the message as they are
required (e.g. download attachments when requested, skip alternative
parts unless requested, etc). IMAP is well suited to this - it's not
a RETR/TOP protocol like POP3. From the protocol's point of view, the
pristine copy of the message is on the server.

I was thinking about that right now, and I'm in the graduated download camp (headers, message, attachments).

It may well be that while the entire message has not been retrieved, we don't store it in the main cache. But then do we rebuild the message from its parts? Download the entire thing again? Either sounds kludgy.

As much as I'm lured by the idea of the pristine message, it's more of a beautiful goal than an important user feature.

[John C. Welch]

Post by Tim Gray

Post by John Gruber
Lightroom does the same thing, too. I don't think it ever writes changes to
the original files you import from your camera. That way, the worst case
scenario for data loss caused by the app itself is a loss of metadata, not
the raw data.

As a fellow photographer, the idea appeals to me as well. However the big
difference is that the email format is documented. RAW files aren't. I
think you can safely add a header to an email and resave it to the server...
That being said, I advocate NOT stripping attachments out of a message's
body by default. It's an interesting command to offer, but when emails come
in and are saved, the attachments should stay encoded in the message, even
if they are saved to a folder.

Well, you can do both. A few clients do that, in different ways, but that's
a tricky implementation on a good day.