UX- download messages upfront, or as needed?

[Gus Mueller]

As I'm looking at some of this code and thinking about caching and such, which is better? Downloading messages right away (takes longer for message list to appear) or grab them when they are needed?

I can see arguments either way:

If there's a thread with 100 messages in it that you don't care about, why would you want the bandwidth taken up when you're just going to delete them?

What if maybe, just maybe you want to connect and pull everything down, then hop on that mythical plane and check your email?

Or maybe there's a 3rd option:
Download all the headers and such first, and once that's done start downloading the bodies? Is this what the iPhone does?

-gus

--

August 'Gus' Mueller
Flying Meat Inc.
http://flyingmeat.com/

[Jonathan Hohle]

Post by Gus Mueller
Download all the headers and such first, and once that's done start downloading the bodies? Is this what the iPhone does?

This sounds like a sane default. This makes caching more interesting, though. Are headers cached separately from the bodies?

Jon
<><

[Lane Roathe]

Post by Jonathan Hohle
This sounds like a sane default. This makes caching more interesting,
though. Are headers cached separately from the bodies?

on topic: one thing that would be good is to delay downloading of
attachments, but where to draw that line might be tough and for some of
us would circle back around to the HTML vs text email debate. (ie, I
don't want HTML parts of an email unless I specifically request them,
etc.) However, this again is effected by filters the user may have in
effect for handling attachments of certain types, certain sizes (ie, do
not auto download attachments over X size), etc.

off topic: Curious, I notice some of you always have a "smime.p7s" file
attached to your emails. What and why?

Lane Roathe <http://www.ideasfromthedeep.com>
___________________________________________________________________
Data, data everywhere . . . and not a thought to to be had.

[Steven Canfield]

Option 3 please. Assume broadband, but also assume I want to see to/
from/subject asap. Seems like the best overall UX.
Steve Canfield

Post by Gus Mueller
As I'm looking at some of this code and thinking about caching and
such, which is better? Downloading messages right away (takes
longer for message list to appear) or grab them when they are needed?
If there's a thread with 100 messages in it that you don't care
about, why would you want the bandwidth taken up when you're just
going to delete them?
What if maybe, just maybe you want to connect and pull everything
down, then hop on that mythical plane and check your email?
Download all the headers and such first, and once that's done start
downloading the bodies? Is this what the iPhone does?
-gus
--
August 'Gus' Mueller
Flying Meat Inc.
http://flyingmeat.com/
_______________________________________________
List help: http://lists.ranchero.com/listinfo.cgi/letters-dev-ranchero.com

[Bob Warwick]

Post by Gus Mueller
As I'm looking at some of this code and thinking about caching and
such, which is better? Downloading messages right away (takes
longer for message list to appear) or grab them when they are needed?
If there's a thread with 100 messages in it that you don't care
about, why would you want the bandwidth taken up when you're just
going to delete them?
What if maybe, just maybe you want to connect and pull everything
down, then hop on that mythical plane and check your email?
Download all the headers and such first, and once that's done start
downloading the bodies? Is this what the iPhone does?

If we're caching at all, let's cache consistently.

UX wise, I think headers first, messages next. When a message is
selected it gets bumped to the head of the priority queue.

-Bob

[Nathan Herald]

Post by Gus Mueller
Downloading messages right away (takes longer for message list to appear) or grab them when they are needed?

99% of the time, I want the headers first (to populate the view of what's available), then the bodies, then the attachments - in that order.

I pretty much always want it all to download, but to update the UI with new content as often as possible.



Nathan

[Guy English]

Post by Gus Mueller
Download all the headers and such first, and once that's done start downloading the bodies? Is this what the iPhone does?

Fancier routing rules will require the body content so it’ll have to come down at some point.

- Guy

[Caio Chassot]

Post by Gus Mueller
Download all the headers and such first, and once that's done start downloading the bodies? Is this what the iPhone does?

I've always thought it would be like that.

And keep a second connection open for "emergencies". Like when you select a message that has not yet been downloaded, just go ahead and use the second connection to get it.

If at all possible, I would even download headers first, messages second, and only then attachments. (Well, for big attachments anyway.)

[Mo McRoberts]

Post by Caio Chassot

Post by Gus Mueller
Download all the headers and such first, and once that's done start
downloading the bodies? Is this what the iPhone does?

I've always thought it would be like that.
And keep a second connection open for "emergencies". Like when you
select a message that has not yet been downloaded, just go ahead and
use the second connection to get it.

The “second connection” idea is a good one, though I’m not keen on the
idea of just having a second connection open *all the time*; it strikes
me as being perhaps being wasteful of server resources. Having said
that, how many connections does a typical client open currently?

(It’d be quite useful if we could get some IMAP server authors on the
list for things like this…)

I’d like to see some degree of intelligence: e.g., a slight delay
before actually utilising the second connection, both in case the user
is just scrolling through messages and the focus would have changed by
the time the message was actually downloaded, and also in case the
message we’re downloading at the point the one we don’t have is
focussed has finished.

Hmm, perhaps focussing an undownloaded message causes it to get
inserted into a “fetch this next” queue, but removed again if the focus
changes or before it’s started, or if the current activity on the
connection is taking too long and the second connection needs to be
utilised?

M.

[Patrick Burleson]

The “second connection” idea is a good one, though I’m not keen on the idea
of just having a second connection open *all the time*; it strikes me as
being perhaps being wasteful of server resources. Having said that, how many
connections does a typical client open currently?
(It’d be quite useful if we could get some IMAP server authors on the list
for things like this…)

One thing to be careful of is having too many connections open to some
servers. When 10.6 first came out, I had a hard time with the Google IMAP
servers and my 6 email accounts and lots of labels^w folders. I got locked
out of the Gmail servers for about an hour because Mail was way too
aggressive with the number of connections it opened.

Patrick

[Gus Mueller]

(It’d be quite useful if we could get some IMAP server authors on the list for things like this…)

We've got one , DINH Viêt Hoà (and he's the author of the library we're using). He'll probably pop up at some point…

-gus

--

August 'Gus' Mueller
Flying Meat Inc.
http://flyingmeat.com/

[Ian Eiloart]

Post by Gus Mueller

Post by Mo McRoberts
(It’d be quite useful if we could get some IMAP server authors on the
list for things like this…)

We've got one , DINH Viêt Hoà (and he's the author of the library we're
using). He'll probably pop up at some point…

That's a client library, not a server library.

[Ian Eiloart]

Post by Mo McRoberts
The “second connection” idea is a good one, though I’m not keen on
the idea of just having a second connection open *all the time*; it
strikes me as being perhaps being wasteful of server resources. Having
said that, how many connections does a typical client open currently?
(It’d be quite useful if we could get some IMAP server authors on the
list for things like this…)

Two connections per mailbox on my network is a sure way to get Letters
banned from the network. Remember Apple X-Serves are limited to 2500
processes per server, and each connection is a process on the server.

[Caio Chassot]

The “second connection” idea is a good one, though I’m not keen on
the idea of just having a second connection open *all the time*; it
strikes me as being perhaps being wasteful of server resources. Having
said that, how many connections does a typical client open currently?
(It’d be quite useful if we could get some IMAP server authors on the
list for things like this…)

Two connections per mailbox on my network is a sure way to get Letters banned from the network. Remember Apple X-Serves are limited to 2500 processes per server, and each connection is a process on the server.

Two connections per account may do, and you rotate mailboxes.

[Tim Gray]

Post by Gus Mueller
Download all the headers and such first, and once that's done start
downloading the bodies? Is this what the iPhone does?

Was thinking of exactly that as I was reading the first part of your mail. I
think that might be the thing to do.

[Jesse Read]

I'm torn between one and three. (Downloading all right away or headers
first, then the rest.)

Option three could result a populated message list where I see something
really important, go to open it, and am greatly annoyed that I now have to
wait to fetch the message. Option one could take a long time though,
depending on how much email I got while off the grid.

I think the iPhone method (three) is the better of the two though. I've
gotten used to it, and it feels "good". Even if there is that "important
message", I can scan most of the mail that came in while waiting for the
fetch to finish. I start prioritizing in my head which need addressed first.

As for the second option (load as needed), I am greatly opposed to it. There
are many situations where I open up my laptop at work (or any place where I
have a known connection) to let things "sync". Email pulls down mainly,
calendars update that need to. I may close the laptop up and move on without
actually sitting down to go through my mail. Later, when I open it up at a
client site and go to check my inbox for a message that my boss told me he'd
sent or just to consume and draft replies during a bit of downtime, I would
be greatly frustrated if everything wasn't loaded and readable. (Many times
while working I end up places that even if there is internet, setting up my
own machine on the network is not going to happen.) Not an airplane, but can
result in a similar feeling.

– Jesse

On Jan 22, 2010, at 9:05 PM, Tim Gray wrote:

On Fri 22, Jan'10 at 5:49 PM -0800, Gus Mueller wrote:

Or maybe there's a 3rd option:

Download all the headers and such first, and once that's done start
downloading the bodies? Is this what the iPhone does?


Was thinking of exactly that as I was reading the first part of your mail. I
think that might be the thing to do.

[Lane Roathe]

Post by Gus Mueller
As I'm looking at some of this code and thinking about caching and such,
which is better? Downloading messages right away (takes longer for
message list to appear) or grab them when they are needed?
Download all the headers and such first, and once that's done start
downloading the bodies? Is this what the iPhone does?

Overall I like the 3rd option best, but as another response pondered ...
how does this effect filtering? While most of my filters are header
based (sender, subject mainly) some of the very important ones are body
based (throw any HTML email in the trash, check email w/SpamSeive, etc).

Maybe if you have body based filters the program uses the first option,
otherwise option 3? If so, employing a body based filter should advise
the user that this will take place.

Whatever the decision, I think the end user's expectations of what
happens to email based on his/her filter settings should be consistent.
So reading a message should not cause the message state/location to
change because the body was downloaded after the headers and thus
delayed the spam filter or some such.

Lane Roathe <http://www.ideasfromthedeep.com>
___________________________________________________________________
We are Micro$oft. Resistance is futile. You will be assimilated.

[Duncan Wilcox]

Post by Gus Mueller
As I'm looking at some of this code and thinking about caching and such, which is better? Downloading messages right away (takes longer for message list to appear) or grab them when they are needed?

My thinking is the bandwidth issues apply to attachments, not to messages, you'd always be downloading the 100 messages in the thread.

Since filtering rules often move the message to a different folder, downloading the headers only is no use because you wouldn't know in what folder to display it, and you don't want to display it in the inbox shortly and then move it -- that would be ugly to see. Typically large messages will be large because of the attachments, not because of the message body.

So headers+message first, attachments later. But later when?

In theory this could be a setting (download right after message bodies, or download on demand), but we all hate settings. The rationale here is that in the vast majority of cases if you're on narrow or metered bandwidth you don't want to download attachments in the background, but only download them if you actually want them.

One way to make this UI would be show the attachment in the message viewer pane and embed the download options there, such as "download this attachment" and "always download all attachments". The counterpart to this setting would be in an activity viewer, "stop downloading attachments in the background" and "stop downloading this attachment". If you wanted to make this really smart, you'd tie the setting to the network you're connected to (and discovering what that is is another can of worms).

As a side note regarding the local cache, storing attachments next to the email instead of inside it would make them spotlight indexed, which is something Apple Mail doesn't do, though the downside is you either don't have the original email text as-is, or you store the attachment both in the original text and as a file, with duplication.

Duncan

[Marcus Karlsson]

Post by Duncan Wilcox
My thinking is the bandwidth issues apply to attachments, not to
messages, you'd always be downloading the 100 messages in the thread.
Since filtering rules often move the message to a different folder,
downloading the headers only is no use because you wouldn't know in
what folder to display it, and you don't want to display it in the
inbox shortly and then move it -- that would be ugly to see. Typically
large messages will be large because of the attachments, not because
of the message body.
So headers+message first, attachments later. But later when?

That depends on the filtering used. Most people I think just filter on
header data, like whom the message was sent from, subject line, mailing
list id and so on. That's all availible in the header.

Post by Duncan Wilcox
In theory this could be a setting (download right after message
bodies, or download on demand), but we all hate settings. The
rationale here is that in the vast majority of cases if you're on
narrow or metered bandwidth you don't want to download attachments in
the background, but only download them if you actually want them.

I think settings are a good idea. Remember that the target audience is
not the casual sends-maybe-an-email-a-day user, it's the poweruser that
lives all day long in this app. Having even very complicated settings
will not be a problem.

Marcus

[Drummond Field]

As a user and not a dev (no experience there), it would be nice if the
solution could be optimised around a slower connection - perhaps 3G (not
necessarily around 56kbps). So often apps (that require network
connectivity) assume the availability of a high speed connection and don't
take into account/really become unusable when used on a slower connection.

I would love to use a client that I can use over 3G and find it still
usable. Someones previous mention of how it works on the iPhone sounds like
a good direction to take

Post by Marcus Karlsson

Post by Duncan Wilcox
My thinking is the bandwidth issues apply to attachments, not to
messages, you'd always be downloading the 100 messages in the thread.
Since filtering rules often move the message to a different folder,
downloading the headers only is no use because you wouldn't know in
what folder to display it, and you don't want to display it in the
inbox shortly and then move it -- that would be ugly to see. Typically
large messages will be large because of the attachments, not because
of the message body.
So headers+message first, attachments later. But later when?

That depends on the filtering used. Most people I think just filter on
header data, like whom the message was sent from, subject line, mailing
list id and so on. That's all availible in the header.

Post by Duncan Wilcox
In theory this could be a setting (download right after message
bodies, or download on demand), but we all hate settings. The
rationale here is that in the vast majority of cases if you're on
narrow or metered bandwidth you don't want to download attachments in
the background, but only download them if you actually want them.

I think settings are a good idea. Remember that the target audience is
not the casual sends-maybe-an-email-a-day user, it's the poweruser that
lives all day long in this app. Having even very complicated settings
will not be a problem.
Marcus
_______________________________________________
List help: http://lists.ranchero.com/listinfo.cgi/letters-dev-ranchero.com

[John Gruber]

Post by Marcus Karlsson
Having even very complicated settings
will not be a problem.

It's a problem for us. Settings add development and maintenance complexity, not just user-perspective complexity. It may well be that we wind up with more settings than typical Mac apps, because of the nature of email and the nature of our intended audience, but we still need to strive to minimize them. In fact it might be more important than usual for us to consciously minimize preferences because the *minimal* number of prefs for Letters will likely be so large.

—J.G.

[John C. Welch]

Post by John Gruber
It's a problem for us. Settings add development and maintenance complexity,
not just user-perspective complexity. It may well be that we wind up with more
settings than typical Mac apps, because of the nature of email and the nature
of our intended audience, but we still need to strive to minimize them. In
fact it might be more important than usual for us to consciously minimize
preferences because the *minimal* number of prefs for Letters will likely be
so large.

Split them up if need be. For example, have application prefs separate from
account prefs separate from rules from sigs, etc.

There's no reason to put them all in one window, and Eudora shows good
reason why you shouldn't.

[Neil]

Post by Duncan Wilcox
One way to make this UI would be show the attachment in the message viewer
pane and embed the download options there, such as "download this
attachment" and "always download all attachments". The counterpart to this
setting would be in an activity viewer, "stop downloading attachments in the
background" and "stop downloading this attachment". If you wanted to make
this really smart, you'd tie the setting to the network you're connected to
(and discovering what that is is another can of worms).

Just as a minor point: settings involving the word "always" or that function
globally should be placed in the Preferences section, not anywhere in the
viewer.

[Duncan Wilcox]

Just as a minor point: settings involving the word "always" or that function globally should be placed in the Preferences section, not anywhere in the viewer.

All absolutes are definitely always wrong. :)

Honestly I strongly disagree with the concept that a "power" or even a "business" user doesn't need simpler interfaces. That's how you end up with convoluted, mutually exclusive, non-deterministic ("let's try this setting and see if it works") settings panels and ultimately applications. Say the VLC/Word/Photoshop settings panels.

I would rather pull a "see figure 1" and pick a default than add a setting.

From another perspective, think about the complexity of supporting all the settings in code. Suppose there are 10 on/off independent settings that affect a specific section of code, that's 2^10 ways it needs to be tested.

Duncan

[Brent Simmons]

Post by Duncan Wilcox

Just as a minor point: settings involving the word "always" or that function globally should be placed in the Preferences section, not anywhere in the viewer.

All absolutes are definitely always wrong. :)
Honestly I strongly disagree with the concept that a "power" or even a "business" user doesn't need simpler interfaces. That's how you end up with convoluted, mutually exclusive, non-deterministic ("let's try this setting and see if it works") settings panels and ultimately applications. Say the VLC/Word/Photoshop settings panels.

Assume that power users are people who are very busy and who use a lot more apps than the average bear. Remember that they're also *Mac* users, with all that suggests about what they like in apps.

-Brent

[John Ford]

Post by Duncan Wilcox
So headers+message first, attachments later. But later when?
In theory this could be a setting (download right after message bodies, or download on demand), but we all hate settings. The rationale here is that in the vast majority of cases if you're on narrow or metered bandwidth you don't want to download attachments in the background, but only download them if you actually want them.

It would be nice if Letters were location aware. It would be delicious if we had access to GPS information but in its absence we have the Location setting in Network system preferences and the network service in use. We could use that information to choose how to download email attachments. Examples: If Location is "home" or "office" and the network service in use is ethernet then Letters downloads all attachments. If Location isn't "home" and the network service is wifi Letters first downloads all attachments under 1MB before beginning on bigger attachments. If the network service is a 3G modem only attachments under 200KB are downloaded automatically. Etc.

I think it's really important (and the "Mac way") to have very smart defaults for stuff like this and for Letters to take advantage of all the system information it can to determine what it should be doing. And, naturally, there should be a way to override Letters' good intentions when necessary.

-John

[Caio Chassot]

Post by John Ford
If Location isn't "home" and the network service is wifi Letters first downloads all attachments under 1MB before beginning on bigger attachments. If the network service is a 3G modem only attachments under 200KB are downloaded automatically. Etc.

I think going for the small ones first is always a win, no matter what connection. If the connection is good, however, a second connection could be working on the big ones.


On location awareness, I like it in theory, but:
- Not for 1.0
- Location detection can be done system-wide, with Letters having only a plugin to change defaults on the fly. Let someone shave the detection yak first.
- Doesn't matter if your location is home with super mega cable if your mac mini in the basement is hogging all the bandwidth downloading torrents.

[Mo McRoberts]

Post by Caio Chassot
- Location detection can be done system-wide, with Letters having only
a plugin to change defaults on the fly. Let someone shave the detection
yak first.

FWIW, 10.6 ships with CoreLocation.

(but at a guess, I would put the chances of this being in 1.0 at
somewhere between 0 and 0.001%)

M.

[Chris Ridd]

Post by Duncan Wilcox

Post by Gus Mueller
As I'm looking at some of this code and thinking about caching and such, which is better? Downloading messages right away (takes longer for message list to appear) or grab them when they are needed?

My thinking is the bandwidth issues apply to attachments, not to messages, you'd always be downloading the 100 messages in the thread.
Since filtering rules often move the message to a different folder, downloading the headers only is no use because you wouldn't know in what folder to display it, and you don't want to display it in the inbox shortly and then move it -- that would be ugly to see. Typically large messages will be large because of the attachments, not because of the message body.
So headers+message first, attachments later. But later when?

I'd refine this a bit. I think you know how many messages there are that you haven't seen, but you don't have to fetch them (headers) all at once. Just fetch the ones (headers) that need displaying. When they're displayed, fetch the others.

I don't know if you'd want to handle this by having message "proxies" being displayed in the message list, or just allowing them to insert themselves. The latter, I guess, if you're threading.

Cheers,

Chris

[Ian Eiloart]

Post by Gus Mueller
As I'm looking at some of this code and thinking about caching and such,
which is better? Downloading messages right away (takes longer for
message list to appear) or grab them when they are needed?

Mulberry has progressive download, with a user option. It downloads N
messages initially, then increments by M when required. Configuration can
be set differently for local and remote mailboxes, but would be better
configured per account. That's reasonable for a desktop in a relatively
fixed environment.

Letters should adapt to the environment, starting with a few messages and
ramping up the increment if they're delivered quickly. It might remember
the last configuration that it used for a given environment (wireless
network name, IP address, whatever clues you have). Could also remember the
best connection per account, and the worst connection per local network to
help with the heuristics.

That's the only way I can see to cope with the variable network
environments that a laptop might encounter.

Post by Gus Mueller
If there's a thread with 100 messages in it that you don't care about,
why would you want the bandwidth taken up when you're just going to
delete them?
What if maybe, just maybe you want to connect and pull everything down,
then hop on that mythical plane and check your email?
Download all the headers and such first, and once that's done start
downloading the bodies? Is this what the iPhone does?
-gus
--
August 'Gus' Mueller
Flying Meat Inc.
http://flyingmeat.com/
_______________________________________________
List help: http://lists.ranchero.com/listinfo.cgi/letters-dev-ranchero.com