Duncan Wilcox
2010-01-25 22:27:07 UTC
What we have is really two projects. The main app and UI is very much in flux, might feature unicorns and/or ponies, while the IMAP library, LetterBox, has known functionality and less known complexity (but is hard to do).
The good thing about this is that LetterBox is already available in source and hackable (http://github.com/ccgus/letters). It needs to be made rock solid and, given the nature of an open source project, resilient to changes from people who are just jumping in for a quick fix.
LetterBox needs to talk to wild IMAP implementations, but still fulfill a well defined contract, not unlike what WebKit does with funky web servers and HTML.
Having done a couple minor contributions to WebKit, I believe a strong aspect of their development process is their reliance on a large regression testing suite.
I last touched it a couple years ago, but what they basically have is a wrapper around the rendering engine that reads a funky HTML/CSS/JS snippet and dumps the full rendering tree with coordinates and contents, or optionally a bitmap. A matching rendering tree or bitmap (they do image diffs) is a pass, otherwise a fail.
Constraining checkins to passing tests, and writing tests for every bug and every feature added, is a reasonable way of ensuring you only ever make progress. I'd say that for WebKit it shows.
So this got me thinking what could be the equivalent of the WebKit tests for an IMAP client. We can't run a local instance of every version of every IMAP server, if nothing else because gmail or mobileme's IMAP servers aren't available.
What we can do though is capture a session that completed successfully and turn it into a dummy server that replies to the requests with the recorded replies.
What the test would do is exercise the semantics offered by LetterBox, so stuff like:
- login
- enumerate folders
- scan folders and get messages sizes and flags
- download select headers, all headers, body
- create folder
- store message
- copy messages
- move messages
Clearly running this against a new real IMAP server might and will fail, so assuming it finally runs fine (and doesn't run other tests) the session can be captured and turned into a fake server. This will work because the fake replies will always report the same information, and the test will always perform the same commands. Clearly a radical change in how the client works will make the patterns change, and will require a re-capture.
Capturing needs not only do basics but also push the limits of both client and server, so it would include things like:
- creating very long folder names
- creating folder names with unicode characters in them
- create deep folder nesting
Perhaps capturing could be entirely automatic, it creates its own test folders and message structure within an empty IMAP account, or perhaps it could require manually setting up the account contents.
libEtPan has the ability to trace a connection (mailstream_debug = 0 in mailstream_log.c will create a libetpan-stream-debug.log), a quick and dirty python script can turn that into a dummy IMAP server python app.
There are other things that need to be tested, like memory use with a 100k message folder or concurrent access through multiple connections, but that could be for 1.1 of the test suite.
And this is just the IMAP part (though arguably SMTP is a bit simpler).
Thoughts?
Duncan
The good thing about this is that LetterBox is already available in source and hackable (http://github.com/ccgus/letters). It needs to be made rock solid and, given the nature of an open source project, resilient to changes from people who are just jumping in for a quick fix.
LetterBox needs to talk to wild IMAP implementations, but still fulfill a well defined contract, not unlike what WebKit does with funky web servers and HTML.
Having done a couple minor contributions to WebKit, I believe a strong aspect of their development process is their reliance on a large regression testing suite.
I last touched it a couple years ago, but what they basically have is a wrapper around the rendering engine that reads a funky HTML/CSS/JS snippet and dumps the full rendering tree with coordinates and contents, or optionally a bitmap. A matching rendering tree or bitmap (they do image diffs) is a pass, otherwise a fail.
Constraining checkins to passing tests, and writing tests for every bug and every feature added, is a reasonable way of ensuring you only ever make progress. I'd say that for WebKit it shows.
So this got me thinking what could be the equivalent of the WebKit tests for an IMAP client. We can't run a local instance of every version of every IMAP server, if nothing else because gmail or mobileme's IMAP servers aren't available.
What we can do though is capture a session that completed successfully and turn it into a dummy server that replies to the requests with the recorded replies.
What the test would do is exercise the semantics offered by LetterBox, so stuff like:
- login
- enumerate folders
- scan folders and get messages sizes and flags
- download select headers, all headers, body
- create folder
- store message
- copy messages
- move messages
Clearly running this against a new real IMAP server might and will fail, so assuming it finally runs fine (and doesn't run other tests) the session can be captured and turned into a fake server. This will work because the fake replies will always report the same information, and the test will always perform the same commands. Clearly a radical change in how the client works will make the patterns change, and will require a re-capture.
Capturing needs not only do basics but also push the limits of both client and server, so it would include things like:
- creating very long folder names
- creating folder names with unicode characters in them
- create deep folder nesting
Perhaps capturing could be entirely automatic, it creates its own test folders and message structure within an empty IMAP account, or perhaps it could require manually setting up the account contents.
libEtPan has the ability to trace a connection (mailstream_debug = 0 in mailstream_log.c will create a libetpan-stream-debug.log), a quick and dirty python script can turn that into a dummy IMAP server python app.
There are other things that need to be tested, like memory use with a 100k message folder or concurrent access through multiple connections, but that could be for 1.1 of the test suite.
And this is just the IMAP part (though arguably SMTP is a bit simpler).
Thoughts?
Duncan