File system cache for imap messages.

[John Gruber]

We have talked about storing some things like that on a dummy message on
the IMAP server. That would be useful for when you setup Letters on a
second machine. Immediately, I wonder what we'd do with data that is not
account-specific, in the case of a multi-account setup.

I'd recommend against this. Having used programs that store configuration
information on the server (like pine), I think it causes clutter on the
server. I still have folders (labels) on Gmail left over from an experiment
with it.

This leaves us with a fundamental issue to ponder.

I am nearly certain that we are going to want to store our own per-message metadata. Probably, I'm thinking, per-plugin-per-message metadata -- i.e. a way for a plugin that wants to store its own data associated with a particular message in a standard supported way.

I think it's a safe guess to say it'd be easier implementation wise to just make this metadata store local to Letters. Could be all in one SQLite database, or one database per account perhaps. Details don't really matter here. The point is, it'd be data that is local to Letters. The downside to this is that it wouldn't sync back to the server, so if you're running Letters on multiple machines accessing the same account, the Letters-specific metadata wouldn't sync between the two machines.

On the hand, if we do syncing, I don't know how we send the data back to the server in a way that doesn't muck up your IMAP account in some way if you're using any other client. See Mail's "Apple To Do" mailboxes, or whatever the fuck they're named -- those damn things even muck up your mailbox listing in MobileMail.

Use Case 1: Someone with multiple Macs.

Use Case 2: Imagine a shared tech support IMAP account, ***@indiedevcompany.com, which is manned by several staff members. A hypothetical Letters plugin might offer a way to associate an email message with an issue in the company's issue tracker. It'd be nice to have that sync between different Letters instances.

So let me end with a question: Are there any known IMAP clients that store client-specific metadata on the server which do so in an unobjectionable way?

—J.G.

[Jon Buys]

Post by John Gruber
. The point is, it'd be data that is local to Letters. The downside
to this is that it wouldn't sync back to the server, so if you're
running Letters on multiple machines accessing the same account, the
Letters-specific metadata wouldn't sync between the two machines.

Would it be feasable to sync this data between installs of Letters
using MobileMe? I think Yojimbo does this.

-- Jon

[Caio Chassot]

Post by John Gruber
. The point is, it'd be data that is local to Letters. The downside to this is that it wouldn't sync back to the server, so if you're running Letters on multiple machines accessing the same account, the Letters-specific metadata wouldn't sync between the two machines.

Would it be feasable to sync this data between installs of Letters using MobileMe? I think Yojimbo does this.

Kinda limiting, in that it's paid.

Then there's Dropbox which I mentioned, and I'm counting the seconds until rsync and ftp show up.

They're all kludges.

[Jon Buys]

Post by Caio Chassot

On Jan 25, 2010, at 3:59 PM, John Gruber

Post by John Gruber
. The point is, it'd be data that is local to Letters. The
downside to this is that it wouldn't sync back to the server, so
if you're running Letters on multiple machines accessing the same
account, the Letters-specific metadata wouldn't sync between the
two machines.

Would it be feasable to sync this data between installs of Letters
using MobileMe? I think Yojimbo does this.

Kinda limiting, in that it's paid.

True, but it seems the most "Mac-like" solution, other than storing
the data on the IMAP server.
-- Jon

[Kelan Champagne]

Would it be feasable to sync this data between installs of Letters using
MobileMe? I think Yojimbo does this.

This wouldn't really help with the specific use-case that John
mentioned (a shared tech support account), because you can't expect
all the people on the support staff to also share a single MobileMe
account. Especially since you can only use one MobileMe account at a
time on your computer.



As a 1.0 solution, it might be good enough to store all the local
metadata in a single location, and let the user set up their own
syncing solution (i.e. Dropbox) if they want.

However, this brings up 2 important things to keep in mind to allow
for a setup like this.

1) Make sure all the local data is stored in a single place, and that
it's either easy to relocate from the UI, and/or that Letters' access
of the data make sure to follow symlinks (so the user can move it
manually and replace the original with a symlink to within their own
sync'ed dir).

2) It would be really nice to avoid using a straight database file
(i.e. sqlite) as the on-disk representation of this data, because
that's too easy to have sync conflicts if Letters is running on
multiple machines at the same time. Maybe something like an
append-only store (for example, Jens Alfke's Ottoman[1]) would work
better? Or, I think OmniFocus stores everything as xml files that
are each basically some database transactions. This on-disk
representation should be easily sync-able, but doesn't have to
necessarily be the most efficient format, because Letters can maintain
a more-optimized representation in memory (and in a cache).

-Kelan


[1]: http://bitbucket.org/snej/ottoman/wiki/Home

[Caio Chassot]

Post by John Gruber
On the hand, if we do syncing, I don't know how we send the data back to the server in a way that doesn't muck up your IMAP account in some way if you're using any other client. See Mail's "Apple To Do" mailboxes, or whatever the fuck they're named -- those damn things even muck up your mailbox listing in MobileMail.

The fucking .DS_Store of email.

It's a shitty solution, but if it's the only solution (other than, say, Dropbox), we'll 1) need to do it anyway 2) definitely need a pref to disable it.

Post by John Gruber
So let me end with a question: Are there any known IMAP clients that store client-specific metadata on the server which do so in an unobjectionable way?

Let's hope.

[Jim Puls]

Post by Caio Chassot

Post by John Gruber
On the hand, if we do syncing, I don't know how we send the data back to the server in a way that doesn't muck up your IMAP account in some way if you're using any other client. See Mail's "Apple To Do" mailboxes, or whatever the fuck they're named -- those damn things even muck up your mailbox listing in MobileMail.

The fucking .DS_Store of email.

This isn't message-specific metadata, this is "shoehorn something that has no business being represented as an email message onto an IMAP server as an email message". Big difference.

Post by Caio Chassot

Post by John Gruber
So let me end with a question: Are there any known IMAP clients that store client-specific metadata on the server which do so in an unobjectionable way?

Let's hope.

Take Mail's "Notes", for example. These look like regular messages anywhere but Mail, and they'll do the right thing between Mail on one machine and Mail on another machine. Extra X-* headers in the messages.

-> jp

[Daniel James]

Post by John Gruber
So let me end with a question: Are there any known IMAP clients that store client-specific metadata on the server which do so in an unobjectionable way?

I don't think they're widely supported but it might be worth
investigating the ANNOTATE and METADATA (formerly ANNOTATEMORE)
extensions:

http://www.rfc-editor.org/rfc/rfc5257.txt
http://www.rfc-editor.org/rfc/rfc5464.txt

But really, I wouldn't bother at first. Later perhaps, with an
abstraction over whatever method is available.

Daniel

[Tim Gray]

Post by John Gruber
Use Case 1: Someone with multiple Macs.

.mac or whatever the hell it's called. Which tells you my opinion of it.
Isn't there a Sync Services API?

One option to get around some of this metadata storage is to stash it in
email headers. You could do it easily with tags (X-Letters-Tags). Other
metadata might take a bit more work.

Post by John Gruber
So let me end with a question: Are there any known IMAP clients that store
client-specific metadata on the server which do so in an unobjectionable
way?

Pine does. In a semi-unobjectionable way. There's a special pine folder
(or two) where things are stashed. You could do that, and just not
subscribe to said folder in other clients. Problem with this is I foresee a
LOT of metadata being shuffled around, which means lots of traffic over
IMAP. Change one tag on one email and rewrite the whole file. Pine only
wrote to it when you changed settings or added an address and saved.

Herein lies the problem with IMAP in my mind. We want it to do all kinds of
magical things, but it's ultimately limited to the concept of folders.
Gmail obviously does something internally to get around this, but it exports
it to the rest of the world as folders.

[John C. Welch]

Post by Tim Gray

Post by John Gruber
Use Case 1: Someone with multiple Macs.

.mac or whatever the hell it's called. Which tells you my opinion of it.
Isn't there a Sync Services API?

Sync services is for applications on the local machine, not IMAP operations,
and if you think IMAP is a pain in the ass, you ain't seen nothing.

Post by Tim Gray
One option to get around some of this metadata storage is to stash it in
email headers. You could do it easily with tags (X-Letters-Tags). Other
metadata might take a bit more work.

That's probably the best way to do it in the long run.

Post by Tim Gray

Post by John Gruber
So let me end with a question: Are there any known IMAP clients that store
client-specific metadata on the server which do so in an unobjectionable
way?

Pine does. In a semi-unobjectionable way. There's a special pine folder
(or two) where things are stashed. You could do that, and just not
subscribe to said folder in other clients. Problem with this is I foresee a
LOT of metadata being shuffled around, which means lots of traffic over
IMAP. Change one tag on one email and rewrite the whole file. Pine only
wrote to it when you changed settings or added an address and saved.
Herein lies the problem with IMAP in my mind. We want it to do all kinds of
magical things, but it's ultimately limited to the concept of folders.
Gmail obviously does something internally to get around this, but it exports
it to the rest of the world as folders.

Well, IMAP is not exactly new. You can't bag on it for not working with
stuff that didn't exist when it was written. Having said that, the folder
metaphor is not evil, no matter what our masters at Legoog want us to think.
It's not like Gmail's UI implementation is perfect, or even vaguely close.

[Tim Gray]

Post by John C. Welch
Sync services is for applications on the local machine, not IMAP operations,
and if you think IMAP is a pain in the ass, you ain't seen nothing.

I only meant for metadata, Letters specific stuff, etc. I know NNW for
example used to let you sync your subscriptions between macs with it, that
kind of thing. I'm not sure how useful it is in practice, how complicated
it is, or much else about it other than I'm not particularly interested in
it because it requires Apple's service which I'm not going to pay for.

Post by John C. Welch
Well, IMAP is not exactly new. You can't bag on it for not working with
stuff that didn't exist when it was written. Having said that, the folder
metaphor is not evil, no matter what our masters at Legoog want us to think.
It's not like Gmail's UI implementation is perfect, or even vaguely close.

I'm not ragging on it at all. I realize there are limits, and the
work-arounds for some of those limits results in things like the giant mess
that is Gmail. I used Gmail pretty exclusively for about 2 years (after
having an account with them for about 6ish). It's got major problems.

I think Letters has a chance of being a good app if it's NOT some super
duper cutting edge app that tries to redefine email. Instead, I think it
should try to be a great IMAP app (which is in the Vision document). Try to
be a modern GUI version of Pine or mutt, bring Mulberry into the present, or
add IMAP to Mailsmith. If the best of those apps are brought together into
a modern, fast, reliable, and scriptable application, I think it will be
popular. Things like Smart folders on top of that would just be gravy.

I personally like folders. Haha.

[John C. Welch]

Post by Tim Gray
I only meant for metadata, Letters specific stuff, etc. I know NNW for
example used to let you sync your subscriptions between macs with it, that
kind of thing. I'm not sure how useful it is in practice, how complicated
it is, or much else about it other than I'm not particularly interested in
it because it requires Apple's service which I'm not going to pay for.

You're confusing .Mac sync with Sync Services. They aren't the same, nor do
they require each other.

Post by Tim Gray

Post by John C. Welch
Well, IMAP is not exactly new. You can't bag on it for not working with
stuff that didn't exist when it was written. Having said that, the folder
metaphor is not evil, no matter what our masters at Legoog want us to think.
It's not like Gmail's UI implementation is perfect, or even vaguely close.

I'm not ragging on it at all. I realize there are limits, and the
work-arounds for some of those limits results in things like the giant mess
that is Gmail. I used Gmail pretty exclusively for about 2 years (after
having an account with them for about 6ish). It's got major problems.
I think Letters has a chance of being a good app if it's NOT some super
duper cutting edge app that tries to redefine email. Instead, I think it
should try to be a great IMAP app (which is in the Vision document). Try to
be a modern GUI version of Pine or mutt, bring Mulberry into the present, or
add IMAP to Mailsmith. If the best of those apps are brought together into
a modern, fast, reliable, and scriptable application, I think it will be
popular. Things like Smart folders on top of that would just be gravy.

It would also save an assload of time. IMAP is really stupid hard to do.
Steal the crap out of mulberry's engine, it's really quite good at IMAP

[Tim Gray]

Post by John C. Welch
You're confusing .Mac sync with Sync Services. They aren't the same, nor do
they require each other.

Got you. I was referring to the Apple cloud syncing service, yes.

[Drummond Field]

Out of interest : how much of an impact would the various IMAP server
implementations out have on how data is stored locally. My guess is that it
shouldn't - but if one is expecting to store certain information in certain
places - would the server not have to support it? Problem with RFC's is they
are not always clear and people implement things the way they understand it
- which isn't always the same as the next person. Probably irrelevant
comment, but just thought I would check

Post by Tim Gray

Post by John C. Welch
You're confusing .Mac sync with Sync Services. They aren't the same, nor do
they require each other.

Got you. I was referring to the Apple cloud syncing service, yes.
_______________________________________________
List help: http://lists.ranchero.com/listinfo.cgi/letters-dev-ranchero.com

[John C. Welch]

Post by John Gruber
I think it's a safe guess to say it'd be easier implementation wise to just
make this metadata store local to Letters. Could be all in one SQLite
database, or one database per account perhaps. Details don't really matter
here. The point is, it'd be data that is local to Letters. The downside to
this is that it wouldn't sync back to the server, so if you're running Letters
on multiple machines accessing the same account, the Letters-specific metadata
wouldn't sync between the two machines.

The server store should be mucked with as little as possible. IMAP and the
way it works means you do not know, ever, how someone will be viewing email.
For example, (and yes, I'm an edge case), I may look at the same message
with Entourage for my own use, Mail to support users, my iPhone from the
local BK, and WebMail just because I shut down e'rage, and need to look at
something real quick and Safari is right there.

What is on the server needs to be un-fucked-with as much as possible, so
that when you don't user letters, it still looks right.

Post by John Gruber
On the hand, if we do syncing, I don't know how we send the data back to the
server in a way that doesn't muck up your IMAP account in some way if you're
using any other client. See Mail's "Apple To Do" mailboxes, or whatever the
fuck they're named -- those damn things even muck up your mailbox listing in
MobileMail.

This, by the way, is a great reason for subscribing to folders. Or not. I
know I have those silly things, but since E'rage does the right thing here,
I never see them. They don't exist. Life is good, but if I ever need them
because some other client shoved data I need in them, I can still get to
them.

Post by John Gruber
Use Case 1: Someone with multiple Macs.

Replace "Macs" with "devices that read email"

Post by John Gruber
Use Case 2: Imagine a shared tech support IMAP account,
hypothetical Letters plugin might offer a way to associate an email message
with an issue in the company's issue tracker. It'd be nice to have that sync
between different Letters instances.

Categories don't suck for this, but, they are a bear to sync when the client
does the implementation, just ask Microsoft.