Prototype update.

[Gus Mueller]

I promised that I'd have a prototype by the weekend, and that's what we've got now. However- I need to stress that this isn't Letters, and in fact the name of the prototype is CrashyEmailApp.

It still isn't a very good foundation to build on, but it's something and if you'd like to follow along with it, you can check it out here:
http://github.com/ccgus/letters

It's a skeleton. My next task is to start rearranging things a bit in LetterBox (which is the backend framework based off MailCore that does smtp and imap stuff via libetpan) so that you tell it "hey, go fetch me info for this account, and load this folder".

The general design for this I've come up with is CEA (CrashyEmailApp) will just tell LetterBox "I'm the delegate for this account, here's a block that I want you to use when my request is ready" and it'll go do it's thing. CEA will then get various updates along the lines of this:

- Authentication was good / bad.
- I've downloaded some headers for the folder you requested, you can pull them out of the cache and display them. I've still got more to come, and will let you know when I'm finished with this task.
- I've downloaded the message bodies for the folder, you can pull these out of the cache too.
- You didn't ask for it, but I checked the server and there's new messages in INBOX.
- I did something, the results are in the cache.
- etc.

The basic idea is that CEA will never have a direct connection to the server. It asks LetterBox to do something, LetterBox will then grab a server connection (from a pool, libetpan isn't thread safe) and do the request, update the cache, and then tell CEA that it's updated (with relevant userinfo supplied). CEA will rarely have a direct line to the server. In a sense, this is almost like the daemon idea but way easier for me to set breakpoints and debug.

Of course, this is all just in my head and needs to be pounded out into code.

Suggestions for improvements to this idea or reasonable alternatives? I've left lots of things vague on purpose, since things always seem to change a bit once you start coding. But I think the general architecture is a decent and obvious.

-gus

--

August 'Gus' Mueller
Flying Meat Inc.
http://flyingmeat.com/

[Mo McRoberts]

Post by Gus Mueller
Suggestions for improvements to this idea or reasonable alternatives?
I've left lots of things vague on purpose, since things always seem to
change a bit once you start coding. But I think the general
architecture is a decent and obvious.

Sounds like a decent (+ sensible) approach, and has reminded me, in a
round-about way, of one of my big Mail.app bugbears: it often looks
like it’s doing absolutely nothing in response to a user action, even
though it’s actually very very busy. If you don’t have the activity
window open, you’d be forgiven for thinking it had ignored you,
especially if the remote server is being slow at something. What I’m
driving at is ways that LetterBox can notify CEA that something is or
isn’t happening that the user should be aware of—not necessarily that
something needs intervention, but indications of progress and so forth
(perhaps establishing a connection is taking a while, for example, or a
large message is being uploaded).

It’s difficult, because depending upon context you can end up with
information overload. I suppose you could categorise activity into
three broad groups:

- Automatic stuff: connecting to the server, checking for new messages)
- Explicit actions: copying messages, sending an e-mail, downloading an
attachment)
- Automatic dependents of explicit actions (the tricky ones): copying
an individual message within a bulk-copy operation, or downloading a
message body as part of a “fetch the bodies of all of these messages”
action

What I tend to find is that clients often either very little feedback
at all (in an attempt to hide the automatic dependents and general
housekeeping), or give _way_ too much — every little conversation the
client has with the server gets reported, so for a bulk operation you
lose sight of your overall progress.

Code-wise, LetterBox invokes blocks to pass notifications back, but
somewhere along the way we need to figure out how those notifications
can be sanely presented (allowing for context), and how that needs to
be reflected in the notification mechanism. Something to think about?

I’m off to clone and build now :)

M.

[Gus Mueller]

Code-wise, LetterBox invokes blocks to pass notifications back, but somewhere along the way we need to figure out how those notifications can be sanely presented (allowing for context), and how that needs to be reflected in the notification mechanism. Something to think about?

Yea, I agree with this (and the stuff you said that I deleted). We'll have to make sure the hooks are in there correctly so it can happen.

I'll let John figure out what that looks like though. Ha.

-gus

--

August 'Gus' Mueller
Flying Meat Inc.
http://flyingmeat.com/

[brian pink]

This looks killer Gus, hopefully I don't lose all evening hacking and
tweaking things in brainstorm mode. =) Code all looks entirely
sensible and easy to expand/think on.

Rock!

- brian

[Gus Mueller]

Post by brian pink
This looks killer Gus, hopefully I don't lose all evening hacking and
tweaking things in brainstorm mode. =) Code all looks entirely
sensible and easy to expand/think on.

Well, LetterBox is going to change sorta dramatically probably this next week. Talking with some folks on irc, I'm thinking of changing the class hierarchy a little bit and shaking things up conceptually to deal with queuing and such. And when that happens CEA will have to change. Well, it'll have to change quite a bit anyway since nothing's cached right now. I'm not saying don't code, just beware that things will be changing for quite some time I'd imagine.

For those looking at the code, wondering what the caching stuff that's there already is for- ignore it. I rushed it and it sucked so it's going to be deleted.

-gus

--

August 'Gus' Mueller
Flying Meat Inc.
http://flyingmeat.com/

[Gus Mueller]

Post by Gus Mueller
Well, LetterBox is going to change sorta dramatically probably this next week. Talking with some folks on irc, I'm thinking of changing the class hierarchy a little bit and shaking things up conceptually to deal with queuing and such. And when that happens CEA will have to change. Well, it'll have to change quite a bit anyway since nothing's cached right now.

I made these changes that I was thinking of. Basically, it now goes like this:

LAAppDelegate has multiple LBAccount(s) (although we only ever operate on the last one for now).
LBAccount has an LBServer, which has a pool of LBIMAPConnections. If one is already in use, it'll make a new one.
You talk to LBServer and it sends requests to an imap connection in the background.
For instant results, while you're waiting for imap stuff to be done, just poll the current (maybe outdated) cache from LBServer (right now it's all in memory, and gone on quit though).
When LBServer has done something of note (folder listing, message listing, etc), it sends out a notification saying so. The UI then updates the folder listing, or mail listing, or whatever.

Right now all the objects that are returned for the cached message listings are LBMessage objects, which have connections to the imap server. That's dumb. Maybe a new class, LBCachedMessage or something along those lines would make more sense.

Anyway, I sort of like this general model. Next up will be the caching bit.

-gus

--

August 'Gus' Mueller
Flying Meat Inc.
http://flyingmeat.com/

[Duncan Wilcox]

Post by Gus Mueller
http://github.com/ccgus/letters

Follow along is good, but what about play along? What area would you like others to work on? Features that could be developed more or less standalone and eventually integrated into crashy?

Duncan

[Gus Mueller]

Post by Duncan Wilcox

Post by Gus Mueller
http://github.com/ccgus/letters

Follow along is good, but what about play along? What area would you like others to work on? Features that could be developed more or less standalone and eventually integrated into crashy?

Good question. I have no idea.

One thing I can think of that could be done right now, which won't have to be rewritten with my planned LB changes, is prefs for adding multiple accounts, and some sort of SMTP list editor. I figure there would be a pool of smtp servers, and an account would then pick one of that list as it's "preferred" one. Maybe sort of copy what Mail.app does a bit there?

-gus

--

August 'Gus' Mueller
Flying Meat Inc.
http://flyingmeat.com/

[Michael McCracken]

Post by Duncan Wilcox

Post by Gus Mueller
http://github.com/ccgus/letters

Follow along is good, but what about play along? What area would you like others to work on? Features that could be developed more or less standalone and eventually integrated into crashy?

Good question.  I have no idea.
One thing I can think of that could be done right now, which won't have to be rewritten with my planned LB changes, is prefs for adding multiple accounts, and some sort of SMTP list editor.  I figure there would be a pool of smtp servers, and an account would then pick one of that list as it's "preferred" one.  Maybe sort of copy what Mail.app does a bit there?

One import-related suggestion - if eventually we want to integrate
with Mozilla Messaging's database of email ISP configs, how about
someone gets started on writing that code?
(I'm not sure if that's fully set up to use yet, but TB3 seems to use it)
https://wiki.mozilla.org/Thunderbird:Autoconfiguration

Also, I have some PyObjC code that parsed the Mail.app user defaults
to get the appropriate IMAP & SMTP account settings.
I put that code up here: http://pastie.org/792846
It should be a quick translation into plain ObjC if anyone's so inclined.

Cheers,
-mike

-gus
--
August 'Gus' Mueller
Flying Meat Inc.
http://flyingmeat.com/
_______________________________________________
List help: http://lists.ranchero.com/listinfo.cgi/letters-dev-ranchero.com

[Gus Mueller]

Post by Michael McCracken
One import-related suggestion - if eventually we want to integrate
with Mozilla Messaging's database of email ISP configs, how about
someone gets started on writing that code?
(I'm not sure if that's fully set up to use yet, but TB3 seems to use it)
https://wiki.mozilla.org/Thunderbird:Autoconfiguration

Yea, that looks worthy.

Post by Michael McCracken
Also, I have some PyObjC code that parsed the Mail.app user defaults
to get the appropriate IMAP & SMTP account settings.
I put that code up here: http://pastie.org/792846
It should be a quick translation into plain ObjC if anyone's so inclined.

Oh hey neato. Good idea.

-gus

--

August 'Gus' Mueller
Flying Meat Inc.
http://flyingmeat.com/

[Ian Eiloart]

Post by Gus Mueller

Post by Michael McCracken
One import-related suggestion - if eventually we want to integrate
with Mozilla Messaging's database of email ISP configs, how about
someone gets started on writing that code?
(I'm not sure if that's fully set up to use yet, but TB3 seems to use it)
https://wiki.mozilla.org/Thunderbird:Autoconfiguration

Yea, that looks worthy.

But, it's a completely crap implementation. It doesn't reuse the IMAP or
SMTP code. In fact, it isn't even written in the same language. It fails to
configure properly on several sites because of that. It won't autoconfigure
the best-practice port 587 on sites that require correct SMTP syntax from
their clients (like ours).

One rule for SMTP autoconfiguration: use port 587 with STARTTLS and plain
authentication if you can. Avoid the non-standard port 465, and the often
firewalled port 25.

Post by Gus Mueller

Post by Michael McCracken
Also, I have some PyObjC code that parsed the Mail.app user defaults
to get the appropriate IMAP & SMTP account settings.
I put that code up here: http://pastie.org/792846
It should be a quick translation into plain ObjC if anyone's so inclined.

Oh hey neato. Good idea.
-gus
--
August 'Gus' Mueller
Flying Meat Inc.
http://flyingmeat.com/
_______________________________________________
List help: http://lists.ranchero.com/listinfo.cgi/letters-dev-ranchero.com

[Steven Canfield]

Hey all,
I've ported that PyObjC code, and it's on gus' branch now. Give it a test and let me know if there are any problems with specific accounts, I'll fix them. Send me your com.apple.mail.plist (feel free to open it in plist editor and anonymize it).
-Steve

Post by Gus Mueller

Post by Michael McCracken
One import-related suggestion - if eventually we want to integrate
with Mozilla Messaging's database of email ISP configs, how about
someone gets started on writing that code?
(I'm not sure if that's fully set up to use yet, but TB3 seems to use it)
https://wiki.mozilla.org/Thunderbird:Autoconfiguration

Yea, that looks worthy.

Post by Michael McCracken
Also, I have some PyObjC code that parsed the Mail.app user defaults
to get the appropriate IMAP & SMTP account settings.
I put that code up here: http://pastie.org/792846
It should be a quick translation into plain ObjC if anyone's so inclined.

Oh hey neato. Good idea.
-gus
--
August 'Gus' Mueller
Flying Meat Inc.
http://flyingmeat.com/
_______________________________________________
List help: http://lists.ranchero.com/listinfo.cgi/letters-dev-ranchero.com

[Ian Eiloart]

Post by Gus Mueller
One thing I can think of that could be done right now, which won't have
to be rewritten with my planned LB changes, is prefs for adding multiple
accounts, and some sort of SMTP list editor. I figure there would be a
pool of smtp servers, and an account would then pick one of that list as
it's "preferred" one. Maybe sort of copy what Mail.app does a bit there?

Should also have an option for "required" SMTP server. As an email admin,
we FORBID the use of other MSAs for sending email from our domain. To send
email from our domain, you're required to authenticate on our SMTP server.

OK, if the recipient isn't in one of our domains, we have no way of
preventing use of third party SMTP servers. But you won't be able to send
mail to a colleague.

Some day, all mail domains will be administered this way. Otherwise, domain
spoofing will never be conquered.

[Jeroen van der Ham]

Post by Ian Eiloart
Should also have an option for "required" SMTP server. As an email
admin, we FORBID the use of other MSAs for sending email from our
domain. To send email from our domain, you're required to authenticate
on our SMTP server.
OK, if the recipient isn't in one of our domains, we have no way of
preventing use of third party SMTP servers. But you won't be able to
send mail to a colleague.

What exactly do you aim to achieve with this measure?
You'll be filtering out about 1% of spam that is sent to your server,
99% of which is probably already caught by your spamfilter.

Post by Ian Eiloart
Some day, all mail domains will be administered this way. Otherwise,
domain spoofing will never be conquered.

You just said yourself that this does not stop spoofing to domains other
than yourself. The only way you can solve that is by using something
like Sender Policy Framework.

That has been proposed a couple of years ago, and still has not gained
much traction.

IMO what you are doing only inconveniences your users for a fake sense
of security.

Jeroen.

[Caio Chassot]

Post by Gus Mueller
One thing I can think of that could be done right now, which won't have
to be rewritten with my planned LB changes, is prefs for adding multiple
accounts, and some sort of SMTP list editor. I figure there would be a
pool of smtp servers, and an account would then pick one of that list as
it's "preferred" one. Maybe sort of copy what Mail.app does a bit there?

Should also have an option for "required" SMTP server. As an email admin, we FORBID the use of other MSAs for sending email from our domain. To send email from our domain, you're required to authenticate on our SMTP server.

The reverse is also true, to send email on certain SMTP servers, your From: has to be authorized to your authenticated user.

The way Mail works is more of a hindrance, as it tries to fallback to SMTPs it just shouldn't.

To, both abilities sound useful:
- lock an account to a particular SMTP server
- lock a SMTP server to a particular account

[Brock Batsell]

So...

Is there any particular area that we can code on without stepping on
what you're doing, Gus? (Apologies if I've missed a post where this
has been discussed.) Are you working on a new caching architecture?
What's the proper way to proceed from this point, is Crashy just a way
to test LetterBox or will it eventually morph into Letters proper once
given some love?

[Gus Mueller]

Post by Brock Batsell
Is there any particular area that we can code on without stepping on
what you're doing, Gus? (Apologies if I've missed a post where this
has been discussed.) Are you working on a new caching architecture?

I'm trying to think of things that can be done by other folks without us stepping on each other's toes. Steven Canfield just added mailto: support tonight for instance, which was a good little task.

Got any ideas? As I think of them, I could add them to a little list, or just post them to this list.

I haven't started touching the caching arch yet. I have yet to really dig into how MailCore / LetterBox pulls down the messages, and how we can should serialize those to the fs. (Obviously it'll basically just be a dump of what we receive from the server).

Post by Brock Batsell
What's the proper way to proceed from this point, is Crashy just a way
to test LetterBox or will it eventually morph into Letters proper once
given some love?

Crashy is really a way to test LetterBox right now, however it's too early to tell if we'll use it for Letters or not. When I start out new projects, I always figure I'm making something to throw away. "Making is thinking", and hopefully we'll learn what our mistakes are before it's too painful to start over, or hopefully we've made the code flexible enough to accommodate future changes. I don't want to throw it away, but it's a possibility.

Some things that can be done now though:

The prefs window will need to load a toolbar, and we'll need code to load in each of it's views depending on what's pressed (using NSViewAnimation or CA to fade them in / out and resize the window accordingly).

I'm thinking, at least for now, a pref pane for Accounts, fonts + colors, and … well that's all we really support right now. We'll add anything else as we need it. Oh and a "General" tab where we can set the preferred mail reader and how often to check for new mail (right now it's hard coded to 2 minutes).

I'm holding off on the main UI for now, as we've got more important priorities and I see it as a distraction until J.G. has some ideas he wants to try out. That'll be a playground as well ("making is thinking again").

-gus

--

August 'Gus' Mueller
Flying Meat Inc.
http://flyingmeat.com/

[Jonathan Wight]

Post by Gus Mueller

Post by Brock Batsell
Is there any particular area that we can code on without stepping on
what you're doing, Gus? (Apologies if I've missed a post where this
has been discussed.) Are you working on a new caching architecture?

I'm trying to think of things that can be done by other folks without us stepping on each other's toes. Steven Canfield just added mailto: support tonight for instance, which was a good little task.
Got any ideas? As I think of them, I could add them to a little list, or just post them to this list.

Just off top of my head, here's what I think could be worked on while gus gets the core app nailed down:

* IMAP test harness (standalone IMAP client tester, python (optional) based fake IMAP script server)

* Rich email display NSView (1) should probably be a standalone proof of concept app that works with the next bullet item

* Produce an archive of as many email "types" as possible for testing (plain text, html text, other text, text with attachments)

* Mail.app importer (just for prefs at first, but maybe entire email archive? - i know pref work has already been started).

* Investigate anti-phishing technologies (1)

* Investigate anti-spam technologies (2)

All of the above can be worked separately without touching the main project.

1: I am making the big assumption that we _want_ to display rich emails correctly and support anti-phishing/anti-spam

Obviously I'm not Gus or Gruber so these are _just_ ideas that I think could be worked on concurrently.

Jon.

[Tom Wanielista]

Since everyone here has extremely high hopes for Letters.app (as do I), as an extensible, power-user based email application, I suggest we start making Letters.app (or CEA) do that one thing extremely well: send/receive email. It doesn't sound hard, but it is just what needs to be done first if we want to actually get to the fun user experience goodness. Thankfully, Gus made us a skeleton and us lazy folk can begin work.

We're not looking to improve the GUI just yet, so we need to make sure that the LetterBox (that's what I believe it's called) framework is rock-solid. For some reason I feel like the organization of this project is still a bit chaotic, and I suggest that Gus could place some issues into the github tracker for us to start working on. If we start dropping issues into the github bug tracker, we can edit libetpan/mailcore (LetterBox) piece-by-piece to our needs in an orderly fashion. This might sound obvious, but I'm mainly making this post as a way to rally-up the mailing-list to take a gander at the source code.

Basically, this is what I suggest:

Improve upon CEA until LetterBox framework is "rock-solid": meaning: fast & stable. I want to be able to use its poor interface to send my emails without fear of it collapsing. To do this, I suggest we start using the github bug tracker to get things moving. After CEA is usable, we can rename it to Letters.app and initiate the future of Letters: it's gorgeous revolutionary UX design, finding out Steve Jobs uses Letters.app as his default email client, finding Atlantis, designing a simple way for a mail client to handle mailing-lists, etc.

If I seem a bit too ambitious please put me in my place,
I am just glad to finally see
that Mail.app
is going to get its arse rocked.
And I want to help.

Thanks,
Tom.

P.S. This is a Mac app, Dammit.