Let's talk about Server Extension Objects (SEO)
I’ve worked for Microsoft for about 8 years, all in the Exchange group, and I really enjoy it. The people who work here are bright, self-motivated, and we have the resources we need to get our jobs done. It feels great to ship world-class products that our customers love. However, not everything is perfect, it never is. As my father-in-law once said, “It’s not all grapes and gravy.” For example, the engineering group must work well together with the user education group to produce documentation that lets customers easily get the most out of the software we’ve produced. The single most frustrating episode I’ve endured has to do with the documentation for SEO, or the relative lack thereof. It’s not like I’m free of blame here, I worked on SMTP when these Server Extension Objects (SEO) were first implemented many years ago. So without further adieu, let’s talk about what I think should have been documented about SEO. Please read on, if you’d like to learn a little bit more about a soon-to-be obsolete technology (it will remain in some SKUs of Windows SMTP, but will be replaced in E12), and help me get something off my mind at the same time. Let’s first describe SEO a bit. SEO is a COM-based technology that allows customization of the Windows SMTP service. In the SEO model, information flows from the source (the SMTP server) to the sink (the SEO COM object), whenever an interesting event in processing an SMTP connection is reached. After the SMTP server is finished calling all the sinks, it then proceeds with handling the SMTP session, guided by the actions of the sinks. The last piece of the puzzle is the event bindings database, where the ordered collection of sinks to call for each processing event is persisted. One piece of code that customizes the Windows SMTP service is Exchange 2000/2003. Without the customization and extension made possible by SEO, Exchange would not be the compelling mail server that it is. So let’s pick apart the parts of the IIS metabase that contains the SEO information, and we’ll discover an object model along the way. To get us started, let’s use the adsutil.vbs script that comes with an IIS install, to inspect the “eventmanager” part of the metabase. If you run “cscript.exe %systemdrive%\inetpub\adminscripts\adsutil.vbs /enum_all eventmanager” you’ll see a big bunch of unintelligible output. Let’s dissect just a fraction of it: [/eventmanager/SourceTypes/{FB65C4DC-E468-11D1-AA67-00C04FA345F6}/EventTypes/{F6628C8D-0D5E-11d2-AA68-00C04FA35B82}] What does this mean? The first GUID can be seen in the public SDK file SmtpGuid.h, where it is given the name g_szGuidSmtpSourceType. The second GUID, located in the same file, corresponds with g_szcatidSmtpOnInboundCommand. To understand the rest of the string, we need to take a quick peek at the SEO object model. The root object here is IEventManager, which has a programmatic ID of “Event.Manager”, and is reflected in the “eventmanager” string. This object is the one you create and use to get at all the rest of the SEO entities. The IEventManager object contains an IEventSourceTypes object (please notice the “s” at the end here, we’ll see other similarly-named collection classes later), and the IEventSourceTypes object just contains a collection of IEventSourceType objects. Each IEventSourceType object contains an IEventTypes object, which, of course, just contains a set of IEventType objects. So, if we re-parse the string above, we can read it as telling us that the IEventManager contains an IEventSourceType with the ID {FB65C4DC-E468-11D1-AA67-00C04FA345F6}, and that this source knows how to generate {F6628C8D-0D5E-11d2-AA68-00C04FA35B82}, or g_szcatidSmtpOnInboundCommand, events. The rest of the info in the eventmanager part of the IIS metabase serves the same purpose, to define the two sources of SEO events, the SMTP and NNTP server, and to list the types of events that each source may generate. The other SEO information, which describes which consumers are to be called for which SEO event types, and in what order, is located in other parts of the metabase. For example, the first SMTP virtual server has its SEO information listed under smtpsvc/1/eventmanager. So if you type “cscript.exe %systemdrive%\inetpub\adminscripts\adsutil.vbs enum_all smtpsvc/1/eventmanager”, you’ll see info like [/smtpsvc/1/eventmanager/EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82}] [/smtpsvc/1/eventmanager/EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82}/Bindings] [/smtpsvc/1/eventmanager/EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82}/Bindings/{2EFF593B-5451-4D20-891C-1D32BFE95A1C}] [/smtpsvc/1/eventmanager/EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82}/Bindings/{2EFF593B-5451-4D20-891C-1D32BFE95A1C}/DisplayName] [/smtpsvc/1/eventmanager/EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82}/Bindings/{2EFF593B-5451-4D20-891C-1D32BFE95A1C}/SinkClass] [/smtpsvc/1/eventmanager/EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82}/Bindings/{2EFF593B-5451-4D20-891C-1D32BFE95A1C}/SourceProperties] [/smtpsvc/1/eventmanager/EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82}/Bindings/{2EFF593B-5451-4D20-891C-1D32BFE95A1C}/SourceProperties/priority] [/smtpsvc/1/eventmanager/EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82}/Bindings/{2EFF593B-5451-4D20-891C-1D32BFE95A1C}/SourceProperties/rule] What could this mean? The first portion of the string -- /smtpsvc/1/eventmanager/ -- just lets us know we are looking at the IEventSource for the first SMTP virtual server instance. Note that there is a difference between an IEventSourceType and an IEventSource. The IEventSourceType is an abstract, like “SMTP server”, while the IEventSource is a literal instance, like “the first SMTP virtual server instance on this machine”. The next part of the string -- EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82} -- tells us we are looking at the g_szcatidSmtpOnInboundCommand event type. To understand the rest of the string, we need to hearken back to the SEO object model, which defines the IEventSource as containing an IEventBindingManager object. The IEventBindingManager object is used to read, write, create, and destroy IEventBinding objects. There is one IEventBinding object contained in the metabase for each point of customization between the SMTP server and external code. The event binding is how this customization is persisted when IIS is recycled or the system restarted. Finally, the IEventBinding object contains an IEventPropertyBag object, which allows the event binding to contain additional information about itself. Now that we know a little more, we can look at the second fragment of output and see that it’s referring to information regarding an SEO binding. The binding is attached to the first virtual SMTP server instance, and it’s for the SmtpOnInboundCommand event type. The binding has a few properties, too. But why can’t we see more? It turns out that the adsutil.vbs script makes use of the ADSI schema for the metabase to understand and display the data contained in the metabase. The SEO information is not contained in the ADSI schema, and so doesn’t display much interesting information at all. If we run the “displaybindings” script (see near the end of this posting for the full vbs script), we’ll see some more descriptive detail displayed: \Source Types\Sources\Bindings\{F6628C8D-0D5E-11d2-AA68-00C04FA35B82} Display Name = SMTP Protocol OnInboundCommand ID = {2EFF593B-5451-4D20-891C-1D32BFE95A1C} priority = 9000 rule = X-EXPS SinkClass = Exchange.PS-EXPS DisplayName = Exchange SMTP Protocol Security EXPS Sink The SEO information looks a little nicer. We have a bit more context given to the GUIDs, and we can see all of the properties and their values. All of the other information under /smtpsvc/1/eventmanager/ is there for the same purpose, to describe event bindings. Now we’ve looked at the SEO object model a bit, as well as pulled apart the SEO information in the metabase. And we’ve mentioned that SEO is not part of our future product shipments. I would describe ways you can use SEO to improve your system, but there really aren’t any. SEO information should just be left alone, there is nothing to be gained from tweaking it. So just as we’ve started, we’re already done. If you have more questions about SEO, or the customization that can be gained from working with it, take a look at MSDN, and let us know if you have any more questions. It was interesting, from my side, to get this technology out into the world, and it would be interesting to hear from you how you’re using it. Here is the content of the displaybindings vbs script: Option Explicit ' ' \nt\private\inet\iis\staxinc\export\smtpdisp.idl ' Const catidPtSourceType = "{FB65C4DC-E468-11D1-AA67-00C04FA345F6}" Const catidPtSmtpSvc = "{1B3C0666-E470-11D1-AA67-80C04FA345F6}" REM ======================================================================================================== REM Function: REM e -- keystroke saver for WScript.Echo REM REM Arguments: REM strMessage -- message to echo to screen REM REM ======================================================================================================== Sub e (strMessage) WScript.Echo strMessage End Sub REM ======================================================================================================== REM Function: REM WriteCollection -- echoes a property bag REM REM Arguments: REM objCollection -- object to write (must implement _NewEnum) REM REM ======================================================================================================== Sub WriteCollection(objCollection) Dim strProperty For Each strProperty In objCollection If IsObject(objCollection(strProperty)) Then WriteCollection objCollection(strProperty) Else e strProperty & " = " & objCollection(strProperty) End If Next End Sub Function GetEventTypeDesc(strEventType) On Error Resume Next Dim strRetVal Dim objComCat Set objComCat = CreateObject("Event.ComCat") strRetVal = objComCat.GetCategoryDescription(strEventType,0) If(0 = Len(strRetVal)) Then strRetVal = "Unknown (" & strEventType & ")" End If Set objComCat = Nothing GetEventTypeDesc = strRetVal End Function Dim objEventManager Dim objEventUtil Dim GUID_SourceGuid Dim strEventType Dim objBindingManager Dim objBindings Dim objBinding Dim strEventTypeDesc ' ' create server event objects ' Set objEventManager = CreateObject("Event.Manager") Set objEventUtil = CreateObject("Event.Util") ' ' make source guid for instance 1 ' GUID_SourceGuid = objEventUtil.GetIndexedGUID(catidPtSmtpSvc,CLng(1)) ' ' get binding manager ' Set objBindingManager = objEventManager.SourceTypes(catidPtSourceType).Sources(GUID_SourceGuid).GetBindingManager ' ' iterate through the event types ' For Each strEventType In objBindingManager strEventTypeDesc = GetEventTypeDesc(strEventType) e "Event Type : " & strEventTypeDesc 'strEventType e "====================================================" Set objBindings = objBindingManager.Bindings(strEventType) For Each objBinding In objBindings WriteCollection objBinding.EventBindingProperties e "" Next e "" Next ' ' cleanup ' Set objEventManager = Nothing Set objEventUtil = Nothing - Ed PrescottWhy is OOF an OOF and not an OOO?
Here's an interesting historical question - when we say Out of Office, why does it sometimes get shortened to ‘OOF’? Shouldn’t it be ‘OOO’? Inside Microsoft, ‘OOF’ means not just the message which says you’re Out of Office, but it has grown to mean the act of being Out of the Office too - so you’ll get people putting sticky notes on their door saying ‘OOF Thurs & Fri’ or even people verbally saying things like, "Oh, Kevin’s OOF on vacation for the rest of the week’. I suppose that sounds better than "Oh, Kevin’s OOO on vacation ..." OOF was a command used in the days of Microsoft’s Xenix mail system, which set a user as ‘Out of Facility’ - ie Out of the Office. The usage of the term ‘OOF’ just stuck, as did the term ‘Little r’ (e.g. on an email sent to a distribution list, "Who wants to go to the cinema tonight? Little ‘r’ if you’re interested", meaning reply just to me) - as preserved in Outlook with CTRL+R for Reply, and CTRL+SHIFT+R (aka Big R) for Reply All. Ewan Dalton/Mixed-ing it up: Multipart/Mixed Messages and You
Greetings, Exchange Administrators! In today’s edition of “and You”, we’ll be covering Exchange’s handing of messages generated by iPhones, iPads, and Macintosh Mail clients. Specifically, we’re going to cover what /mixed content body messages are, how Exchange handles them now, and how we handle them down the line. Before we can dive into the content, you first have to understand how internet messages are structured, and that means learning a little bit about how Exchange stores messages, and quite a bit about MIME . Not the mute freaks – Multipurpose Internet Mail Extensions, also known as “The Mime not everyone hates.” Exchange, Messages, and LOL Cats Exchange stores messages as a series of properties, where each property has a name and a value. For instance, PR_SUBJECT is the subject property, and “Test Message” is the value. Messages in Exchange have one body with multiple forms of representing it (HTML, Rich Text and Plain text). The HTML view of the body looks like so: This is a cat. Do Not Want The RTF (rich text format) view of the same body is also capable of containing both formatting and images, and so would look like: This is a cat. <Mentally insert picture of cat here – it’s only eight lines up, so honestly, you can do it. > Do Not Want The plain text version of the body is composed of plain text, a fact that should be obvious based on its name. A good way of simulating plain text body generation is to paste your content into notepad. If it survives the paste into notepad, it will be part of the plain text. Sadly, the cat picture does not: This is a cat. Do Not Want Messages in Exchange can have multiple attachments. This is good, because even if Exchange is forced to generate a plain text version of the body, the cat picture can come along so that should you decide to click it, you can view a picture of a cat which does not want something. Key points for non technical and allergic to cats people: For Exchange, messages can have one body and can have many attachments. And in this corner, MIME MIME is a plain text format for email messages. MIME messages are divided into “parts”, each of which might have content or even child parts, like a series of Russian dolls. Each MIME part (even the root part, or the message itself) has a header called Content-Type, which describes the type of content in the part. Content type is divided into major parts and minor parts, separated by a slash. For example, consider the content-types multipart/alternative or multipart/mixed. Every part has a type, even going so far as to define a Miranda type for parts which can't afford to assign one (text/plain). Sometimes the types are quite helpful at understanding the meaning of the content, sometimes not so much. For instance, Content-Type: Application/PDF – that one means that it is Adobe’s Portable Document Format. On the other hand, Content-Type: application/octet means “I can’t tell what this is. Here’s a binary blob for your troubles. Hopefully you can figure out what to do with it” Multipart/ is a general type, meaning that this MIME part may contain many child MIME parts. The sub type of the part (the part after the slash) tells us more about the child parts, and in this case, how they are related to each other. Now we will take a closer look at some of the multipart sub types to see where things can go wrong. Relativity First off we’re going to look at Multipart/related, (also called a “related” body part). Related, in this case, means that the sub MIME parts are actually related to each other – in other words, that give the following MIME structure: 1. Multipart/related 1.1. Text/HTML 1.2. Image/Gif That 1.1 and 1.2 are not meant to be interpreted as “separate” parts – they have meaning as one. In this case the html contains image links to the 1.2 image (our friend the cat). Key point for non technical and allergic to cats people: Multipart/Related means “We belong together.” Alternatives Multipart/alternative means that each child of this part is a different representation of the same data. They are “Alternative” versions of each other. The intention is that a client picks the type that it can best display and displays that one. So given this mime structure: 1. Multipart/alternative 1.1.1. Text/Plain 1.1.2. Text/Html 1.1.3. Application/Pdf The client doesn’t have to show the text/plain part as the body. No, if the client knows how to display a text/html body, it is free to do that. So multipart/alternative is a way of grouping a number of different formats of the same data together and letting the client decide which one it shows best – it’s like kid’s beauty pageant, except that instead of the ladies from the rotary club, you have the email client as the judge. Key point for non technical and allergic to cats people: Multipart/Alternative means “Pick the one you like best.” Mixed Up Multipart/Mixed, according to RFC 1521, means that the parts are completely independent of each other (not related to each other) but that their order matters. What is the expected behavior? “Clients usually display the parts one after the other.” This, however, brings into play another parameter on the MIME part – Content-Disposition. This parameter has a couple of normal values – Inline and Attachment. Attachment is easy to understand – in the context of Exchange, it means “Show me in the well, that I may be blocked by Outlook from being saved or opened.” Inline, on the other hand, we handle differently. Remember that whole “messages have one body, and maybe many attachments” thing? Keep that in mind while we look at how our Cat message looks like with a /mixed body: 1. Multipart/Mixed 1.1. Text/Html - Inline 1.2. Application/Gif - Inline 1.3. Text/Html - Inline And the intention among clients that generate this is that the receiving client should display the text/html part first and then glob on the image to the end of it, and then the rest of the body. There’s no limit to multipart/madness, you can combine them (and dispositions) into nigh endless combinations. For example: 2. Multipart/Mixed 2.1. Text/Html -Inline 2.2. Image/Gif -Inline 2.3. Text/Html -Inline 2.4. Text/Plain -Inline Means “Show 2.1, followed by the image from 2.1 then the html from 2.3 and then the text from 2.4. Do it NOW.” 3. Multipart/Mixed 3.1. Text/HTML -Inline 3.2. Image/Gif –Attachment 3.3. Text/Html –Inline 3.4. Text/Plain –Attachment Means “Show the text from 2.1, NOT the attachment from 3.2 unless someone does something, the text from 3.3, but NOT the text in 3.4 (unless they do something like click an attachment in the well). The problem, of course, comes from Exchange’s original definition of a message – one body (with multiple representations), maybe many attachments. Key point for non technical and allergic to cats people: Multipart/Mixed can mean “Maybe show all of these, in the order listed.” Combo #5 MIME Types are not exclusive. I can combine Multipart/Mixed, Multipart/Alternative and Multipart/Related into a single message, and actually have a meaning 1. Multipart/Mixed 1.1. Multipart/Alternative 1.1.1.1. Text/Plain 1.1.1.2. Multipart/Related 1.1.1.2.1.1. Text/HTML 1.1.1.2.1.2. Image/Gif - inline 1.2. Image/Jpeg - attachment Yes, this structure is legal. And it is meaningful. To understand this you unwrap in order, one level at a time. Multi Mixed – this message is different parts, put together, and the order matters. Multipart/Alternative- I have two children, pick the prettier one and show it off. Text/Plain – I am a blob of plain text Multipart/Related – My children are bits and pieces of each other Text/HTML – Pretty, Pretty Text. Image/Gif – I am a picture of a cat, referenced by pretty, pretty text, I hope. Image/Jpeg – I’m an attachment (in case you couldn’t see the picture of the cat above). Exchange has always dealt well with multipart/alternative bodies, picking the one which we can best support and promoting it. We deal well with multipart/related as well – not every attachment on a message is visible – attachments have a disposition, which is either not set, inline or attachment. Setting neither indeterminate – the client does what the client does (and good luck establishing an algorithm that works for everyone). On the other hand, messages with /mixed content bodies where multiple parts are inline, those do not work so well. Blender’d Messages In the case of /mixed bodies, there are multiple MIME parts which are meant to combine together like Japanese robots to form Voltron, or an image of a cat and some text. Today, if you receive such a message, we do the best we can with it (which is pretty dis-satisfying): We pick the first “body type part” – aka, a text/<something> part, and that one becomes the body as seen by Outlook. All of the rest of them, those are attachments and we shove them into the attachment well. Oh, sure they might have a disposition of inline, but because the most common usage of inline is in HTML, we actually check, and anything that isn’t referenced by a link from the body, we won’t be fooled by. Into the well it goes. From an Outlook perspective you get the first part of the message, then two attachments, one of which is the picture and the other is the trailing text. You are welcome to open the attachments in order, and combine them in your head to form a message, but once you get more than a couple parts it isn’t reasonable. Exchange has never supported “proper” display of /mixed body messages in OWA or Outlook, until now. Blended Messages Starting with Exchange 2007 service pack 3 roll up 3 (E12SP3RU3) and Exchange 2010 service pack 1 rollup 4 (E14SP1RU4) are a set of changes to how Exchange handles /mixed body messages. We think that in general your users will enjoy the less mangled nature of these messages, but if I’ve learned anything from fourteen years of working on Exchange, it’s that “different == angry”. People get used to our behavior, so even when it’s wrong (or incomplete) they expect the same behavior, and come to rely on it. Consider this is your fair warning that this change is coming, some details on how it works, where it works, and where it doesn’t. We're adding support for Exchange to combine multiple body parts into a single, aggregate body. The short of this is that broken up messages should show combined together, and readable in OWA and Outlook. There are, however, limitations to this. First off, right now this will only work for message generated by Apple iPods, iPads, or Apple Mail clients. This isn’t an accident – we developed the rules used to combine these bodies using test data from our counterparts at Apple, and while we handle messages by them well, the internet is wide and wondrous, and anyone can write messages with multipart/mixed bodies. For now this is restricted to clients we have good test data on, good rules for and a good way to identify. To create the aggregate body, we check each MIME part in the /mixed body. If a MIME part has a disposition of Attachment, it goes to the attachment well. I am not going to argue with a client that specifies that it is an attachment. If a body part has a disposition of inline or not set, if it is a plain text or html body part, we add it to the aggregate body. If the body part is an image which can be displayed inline, we add a link to it in the aggregate body. If the body part is not text, or is an image we can't display inline, it goes to the well. How do I know if this breaks me? If you're the owner of an application which is used to sending in /mixed content messages with MIME parts that you rely on Exchange treating as well attachments, and you send them from an Apple platform, and you haven’t been setting a content disposition, and the parts are text or image types (and you are setting content type), then you need to add a Content-Disposition to the MIME parts you want to be attachments, and set them to Attachment. If you're a normal consumer wondering why messages with images or signatures got split into pieces, you don’t need to do anything. Conclusions Today we’ve covered different MIME structures, different representations of bodies, dispositions, types, and a host of other things, but the hard boiled summary is that mail between Apple clients and Exchange clients will be handled better. The best case scenario isn’t that your users call up and say “Wow, I’m really impressed that the messages I got from John on his Mac aren’t chopped up into a dozen pieces anymore.” The best case scenario is that things just work. No one has to call anyone, because you neither notice nor care what platform the message came from, or what format it was sent in. You can see your email, your signatures, and yes, your LOL cats. Enjoy! Epilogue: How things went wrong Already I’ve read (and responded to) reports on the Exchange Update Forums that this new code introduced a new problem. PDF attachments from Mac clients which declare their disposition to be Inline aren't visible in the attachment well. Visible in Outlook Web Access, yes, but MAPI, no. So how on earth did this happen? And how did we miss this bug? (“Do you do any quality control?” as one person asked.) The answer to this is yes, we do. But rather than just say that, let’s look at the journey we took to get this fix included in a rollup, what the problem is, how we missed it, and what we're doing about it. This fix begins with a conference call between me, a few team mates, and our counterparts at Apple, in which someone asked “Well, can you guys do anything to actually support this style of message body?” I spent the next few days researching what it would take to build a synthetic body out of the parts and eventually concluded that it would be possible. With that we began the discussions of “Why now?” and “What is it going to take?” That phase took quite a while. The core problem was that we were well aware that producing a new type of body was going to require a large investment in testing. Eventually we concluded that we could and would do it. And we did. We wrote the code that supports this type of body, we wrote the code that tests it. We have a huge number of tests that test various types and formats of bodies. We had only the new ones to test the new body. Line for line there's more code to test this than there's to support it. So the fix was done, the code was tested, but we still weren’t ready to check in the code. Instead, we took a much more cautious approach. The fix was available as an interim update and that interim update was tightly controlled since we wanted it to go to customers who would be putting it into daily use. After a month or so we loosened it up and the fix went out to four more customers, eventually being in play at around 12 customers for about three months. After three months of field deployment with positive results, we decided to schedule it for a rollup (which should've been RU2, but wound up being RU3). But there’s a problem. You see, Exchange 2007 and 2010 don’t pay much attention to Content-Disposition, because for years inline has been a great way to make sure the attachment is essentially invisible (an inline attachment has its hidden property set to true, since it is displayed inline with body content). And the test code missed this case – it’s an inline attachment which can't be displayed inline by Exchange, but in a message which contains multiple body parts which can be merged to build the synthetic body. Unfortunately neither our in-house nor field testing encountered this. The bug is that in processing these messages, we honor attachment disposition in a case where we should not. Any attachment which can't be displayed inline should never be allowed to become part of the synthetic body. We know of two types of attachments – TIFF and PDF, which can be displayed inline on Mac clients but not on Windows clients. The fix for these two also fixes any other types where the client might render inline but we can't. How did we miss this? We went back over our test code and test data and said “It contains PDFs, and validates the attachment well status.” It does indeed (and TIFFs as well). It also doesn't ever attempt to create a PDF attachment and render it inline in the body. That’s the missed case which I certainly wish we had found before it ever hit any customers. So to resolve this we're creating an interim update, and we’ll be rolling it into the main branch to prevent further incidents of this. Over time I expect that we'll expand the number of mailers we produce synthetic bodies for. For now, I think this experience has validated that we should keep it limited and expand support where we can closely test. I remain convinced that improving interop between Mac clients and Exchange is a good idea. So there we have it: The problem, the fix, the problem with the fix, and the fix for the problem with the fix. We now return to your regularly scheduled blog. Jason NelsonHow the M: Drive came about
In Exchange 2000, we introduced a new feature called IFS. IFS stands for “Installable File System”. This uses a little known and even less used feature of NT that allows the OS’s file system (like NTFS or FAT) to be replaced. The initial reason for doing that was as an optimization: it would allow protocols, such as NNTP and SMTP, to transfer the MIME messages directly as files. In Exchange 5.5, MIME messages are broken down into MAPI properties and stored in database tables. When they need to be accessed as MIME, they are put back together. In E2K, MIME messages are stored as MIME files in IFS and only converted into MAPI if a MAPI client (such as Outlook) accesses them. The other perceived benefit of IFS was that the Exchange storage objects could then be made visible through the file system. So you could go to a drive letter (M: was chosen for two reasons: first, “M” for “Mail”, and second, because it was in the middle of the alphabet and least likely to collide either with actual storage drives -which start at A and move up - or mapped network drives - which start at Z and move down), and get a list of mailboxes, navigate to mail folders via cmd or windows explorer and look at actual messages. This was considered pretty neat at the time and since it didn’t seem to be much more work to allow that access, it was thrown in (there may have been other, better reasons but I’m not aware of them). This ended up causing some challenges down the line related to the intricacies in how email objects need to be handled and mapping the file access behavior to them One of the biggest problems encountered was around security descriptors. This is difficult to explain without a detailed understanding of NT security descriptors, so I will simplify the explanation for the purpose of this discussion. The main part of an NTSD is called a DACL (discretionary access control list). It contains a list of users and groups and what they can do to that object. There are two main types of entries: allows, which say what an entity can do; and denies, which say what they can’t. The order of this list is very important. A standard sequence of entry types is called “canonical”. NT canonical form calls for a particular sequence. Because of legacy issues, MAPI canonical form requires a difference sequence of entry types. Applications that modify security expect a particular sequence and will behave erratically if the sequence is wrong. By creating or modifying objects through the M: drive, it will change the canonical format of the DACL’s and result in unexpected security behavior. This is bad. A related issue here has to do with item level security. E2K also introduced this feature, which is that items in a folder can be secured independently of each other and the folder. While this has some great uses, for many email systems this level of security is not needed. When a message has the folder default security, it simply references that property in the folder. When a message has its own security, there is an additional property that needs to be stored (this also has an affect on how folder aggregate properties, such as unread count, are computed). Having lots of individual security descriptors can result in both increased storage size and poor performance. When a message is created or modified through the M: drive, it will always get an individual security descriptor stamped on it, even if it is exactly the same as the folder default. This can also lead to unexpected behavior. For instance, if you change the default security on the folder, it will not change the security on any messages in it that have their own security descriptors. They have to be resecured individually. Another challenge is in relation to virus scanners. Virus scanners typically look for valid storage drives and spin through all the files on those drives and check them against virus signatures. The M: drive appears as a normal drive, so virus scanners were picking this up and processing it. This can have very detrimental affects on system performance and may also result in message corruption in some cases. Finally, IFS runs in kernel mode. This is a privileged execution path and it means that problems in this area can have much more severe affects (and be harder to track down) than in other areas of Exchange, which all run in user mode. Blue screens are one possibility if something goes wrong. IFS has given Exchange 2000 and Exchange 2003 a lot of advantages: we maintain content parity for MIME and make MIME message handling faster and more efficient as well as increasing the performance of such messages retrieved via internet protocols. But as I described above, there can be problems if IFS is misused via the M: drive. In Exchange 2003 we have disabled the M: drive by default to hopefully help reduce the likelihood that customers will encounter any of the issues described above. I encourage every system administrator to keep this disabled on E2K3 and disable it on all E2K servers as well. Jon AvnerThe history of content conversion in Exchange
The Exchange server and the client (that evolved into Outlook) were originally (circa 1992) based on the MAPI standard (stands for Messaging Application Programming Interface). Broadly this can be divided into the MAPI data model, and the MAPI object schema. Very simplistically, the MAPI data model can be summed as: message stores (think mailboxes) that contain folders that contain items that contain attachments. And all of the above entities are simply a collection of scalar properties (name, value pairs). The original MAPI schema laid out the common list of attributes applicable to email messages, for eg: Subject, Received Time etc. This has since been extensively extended by Outlook and Exchange for various advanced functionality. The Exchange server store was implemented to store the above data model in a very efficient manner, using a database technology called internally as Jet Blue. This data model, although simplistic in this day and age, has proved extremely flexible and surprisingly powerful to model many things that the original inventors could never have even imagined. And the Exchange server, due to its excellent implementation of this data model, has reaped great rewards. The primary mail transport protocol used for business e-mail back then was X.400. At about 1995 or so, the internet wave came along. The Web, the Browser, and HTML changed the world. Along with them the popularity of other standards based protocols for email increased, such as SMTP for mail exchange and MIME as the serialization format for email messages. Both of these rocked the ship of Exchange back then. The Exchange product had already been in development since '92 and more delays to complete it and ship it would have pretty much meant disbanding it altogether. The pressure from the executive level to ship it was intense. But folks in Exchange back then were wise enough to see that this internet wave was too big to just be a passing fancy and that these new standards must be embraced. As a compromise, support for SMTP as a mail exchange protocol (instead of just X.400) and MIME as a data interchange format (instead of just MAPI) were added to the product, but on the periphery so that the core product won't be affected much. Fortunately, the existing transport architecture had supported the notion of Gateways to connect to disparate other mail systems that existed then (again circa '92) who were foolish enough to not support X.400 natively! For example to connect with ccMail, GroupWise and the likes. We decided to lump this new kid on the block (SMTP) in the same category and modelled it simply as a gateway add-on to connect to this thing called the "internet"! This was called the Internet Mail Connector or simply the IMC. This IMC also then got the fun task of back and forth conversion between the data format stored native in the Exchange store (MAPI) and the data format that was the new internet standard (MIME). A conversion library was created for this purpose and was named, somewhat oddly now, as IMAIL (Internet Mail). This was the how we shipped Exchange 4.0 (March 13 1996). The next release, 5.0, was a very focused one in which we made very minor changes, although we did also add POP support. Both the IMC and IMAIL stayed pretty much the same as they were in 4.0. In the next release, 5.5, we decided to add support for IMAP4 as well. Now, POP and IMAP were similar to SMTP in insisting on MIME as the data interchange format. So it was decided to move this IMAIL conversion library closer to the actual data store (that was still a MAPI store of course) to improve performance. In addition, we invested significantly in this library to do a top notch job of translating to and from these 2 formats, and this has been quite a success overall. Exchange 5.5 was a very successful release thanks to the right balance of functionality and simplicity. The next release was Exchange 2000. By this time the domination of the internet protocols was so complete that we was decided to swap the roles of SMTP and X.400. That is, to make SMTP our primary mail exchange protocol even between Exchange servers, and treat X.400 as a one-off Gateway to reach pockets of the world that were still using it (mostly Europe). Also, we decided to go even further along the road of embracing MIME as the data format. We decided to actually enhance the core Exchange store to make it natively store and "understand" MIME. In this release we also invested in a new file system technology that would let us store these large "streams" of MIME content more efficiently. See Jon Avner's blog on the M: drive for more details on this one. In order to satisfy our richest client (Outlook) that continued to demand a MAPI data model, we had to engineer the Exchange store to do on-demand or deferred conversions back and forth between the 2 formats. As one can imagine, this was a very challenging piece of engineering, fraught with subtle traps. In the Exchange 2003 release we did not make any significant changes in this area, and status quo from the 2000 release remains. Naresh SundararajanHow does your Exchange garden grow?
One question we often get asked when talking to customers contemplating an Exchange upgrade or a switch from a competitor’s mail system is, "how many users per server can Exchange handle?" Nowadays, that’s an open question - it very much depends on what kind of users you have, what kind of storage you’re using and how powerful your servers are. When Exchange 4.0 was released in 1996, a decent server might have had 256Mb of RAM and a 90MHz Pentium processor, with maybe a handful of GBs of SCSI disk in the box and possibly a DAT tape drive. Users’ mailboxes might have been in the 10-20Mb size range, and the average user sent or received only a small amount of email per day. At that time, server sizing was pretty much a function of how much computing horsepower you could afford - the CPU power, disk size & speed and memory capacity available (along with the all-important user profile) would determine the number of users per server, and arbitrary decisions would be made about maximum message sizes, mailbox size etc. Now, it’s possible to buy even mid-range servers that will cope with many thousands of users, and the bottleneck has moved down to the storage level in many instances as user mailboxes have grown in size and we send and receive far more mail, and many larger messages too. We’d be interested in hearing anecdotes about particularly unusual messaging environments you might have - how large do your servers grow? What’s the biggest mailbox you’ve ever seen (assuming you’re not restricting with quotas)? How much data does your total Exchange system manage? What’s the biggest single database - etc, etc... you get the picture. Some factoids to get you started... if you have anything along these lines, please post a comment: - One customer (who is part of the Exchange Technology Adoption Program) sets no limit on message sizes internally - and one time saw a single message with attachments totaling 2.4Gb in size! (which Exchange delivered! - the recipient’s mailbox quota was instantly blown apart though...) - The largest single Exchange 5.5 Public Folder Information Store I’ve come across was 800Gb in size - There’s a limit of 1,000 Exchange 2000/2003 servers in an org - has anyone come close, or hit that limit? - One customer deploys around 250,000 Exchange users per server and has over 9,000,000 users in total (you guessed it, they’re an ISP) - The most heavy user of public folders in the world (that we know about) has about 1,500,000 public folders containing around 8Tb of data So come on: let’s hear your war stories - post a comment with your amazing data :) - Ewan DaltonSqueaky Lobster rears its head again in the Microsoft employee giving campaign
This post is mainly for other softies who read the blog, but we hope our customers and partners enjoy it with us as the squeaky lobster has long been part of Exchange culture and lore and -- if we have anything to do with it -- will continue to embarrass the developer who started it all for years to come. A little over 11 years ago, a developer on the JET team made an infamous checkin, to create a reg key controlling some obscure perf counters, named "Squeaky Lobster". The name was a reference to a girl he had a crush on at the time who donated a squeaky lobster to the annual Microsoft Giving Campaign charity auction. He thought it was a funny thing to donate to a giving campaign and gee whilikers, he had a checkin to make that day that surely nobody else would ever see, so why not? And so, it began. Not with a bang, but a whimper. But it doesn't end there, no no no. Fortunately, said developer has some friends and co-workers who won't let him forget about this incident of his youth, and celebrate Squeaky Lobster Day every September 26 th by attempting to embarrass him. This year, the celebration is a little delayed, because the giving campaign doesn't start until October. and we decided that after 11 years (we turned it up to 11, awww yeah), it was time to donate another Squeaky Lobster to the giving campaign - except this time, it would be signed by Andrew himself! Unfortunately this auction is only available inside Microsoft, but for those of you who do work here we encourage you to please bid on this (priceless, really) piece of Exchange history and continue the noble tradition of never letting Andrew forget: (URL to donation site removed) - KC LemsonA brief history of Exchange Team gifts over the years...
Note: There is absolutely no useful content in this posting whatsoever. If you are looking for some little nuggets to help you administrator Exchange, you aren't going to find it here. If, however, you're a bit bored because Exchange is working so well and there's nothing for you to do, read on. Whenever we release a new product or a major new version of a product at Microsoft (or even when we hit a milestone during a release), we celebrate the accomplishment. We have ship parties, and we get ship gifts. We also have a party and get a gift around the holidays. The parties have mellowed a bit over the years but something about the gifts remains timeless. Rather than being like gifts that keep on giving, however, they are more typically gifts that never really gave in the first place. Getting gifts is a lot of fun, and I love getting them, but somehow the search for a single item that is fun or useful for several hundred people tends to yield some highly odd results. In the early years, we tended to get clothing: hats, shirts, even jackets. These are typically the most popular and successful gifts (which is why we've stuck with T-shirts for Exchange 2007), but like all gifts of clothing they're pretty boring, so let's take a look at a few more creative examples. I have tacked up to my cork board a small gift that I think came for one of our beta releases for Exchange 2000 (that was a while ago, so I don't remember exactly when we got it). It's a VR Puppy. Remember these? It's got a small LCD display and buttons on it that you can use to pretend to feed it, walk it, etc. Mine's still in the original packaging, which either means it's worth some money to a collector or it died of starvation several years ago. Next, let's turn to my memorabilia shelf and consider one of the more notorious ship gifts we've: a ceramic popcorn bucket. It came with a package of microwave popcorn. We got this as a holiday gift a few years ago when our morale budget was a bit tight due to other expenditures that year. The really impressive thing about the popcorn bucket is its weight. Perhaps it's lined with lead, possibly to prevent any leftover microwaves from leaking out of the popcorn and into your lap while you're eating from it, although I highly doubt anyone has been so bold (or strong of thigh) as to actually use it for that purpose. Below that we have a very nice, elegant, Exchange 2000 pocket watch. It comes in a very nice, elegant wood case. It's still in its nice, elegant wood case. And last on our tour we'll take a look at the pocket combination tool. This is one of those combo tools that can fold out into a pliers, or individual blades can be extracted like a Swiss Army Knife. This was actually a pretty good idea, and I've even tried to use it once or twice. Mine could be the last one in existence, though, because most of them broke almost immediately upon opening them. Let's just say they weren't particularly well made. And then there's the micro-camera, which we got for Exchange 2003. Another quite good idea that didn't quite pan out due to quality problems. Like my VR puppy, it's also still in its original packaging. Anyway, I hope this posting isn't read by any of the people responsible for purchasing these gifts, because I'd rather get another popcorn bucket than a lump of coal in my next stocking. I hope your Exchange server continues to provide you the free time necessary to read more useless posts like this. - Jon AvnerMe Too!
One way of telling how long a Microsoft employee has been working here is their reaction to the phrase “Bedlam DL3”. Just for grins, I was at lunch in the cafeteria with a bunch of co-workers and I blurted out, totally out of context: “Bedlam DL3”. About 3 of the old-timers in the group responded, in chorus “Me Too!” So why does everyone know about this rather mysterious phrase? Well, Microsoft’s a pretty big organization. We’ve got well over 100,000 mailboxes in our email infrastructure, and at times it can become rather cumbersome to manage all these. One of the developers in our Internal Technologies Group (also known as ITG, basically the MIS department at Microsoft) was working on a new tool to manage communications with the various employees at Microsoft, and as a part of this tool, he created several distribution lists. Each distribution list had about a quarter of the mailboxes in the company on it (so there were about 13,000 mailboxes on each list). For whatever reason, the distribution lists were named “Bedlam DL<n>” (maybe the tool was named Bedlam? I’m not totally sure). Well the name of the lists certainly proved prophetic. It all started one morning when someone looked at the list of DL’s they were on, and discovered that they were on this mysterious distribution list called “Bedlam DL3”. So they did what every person should do in that circumstance (not!). They sent the following email: To: Bedlam DL3 From: <User> Subject: Why am I on this mailing list? Please remove me from it. Remember, there are 25,000 people on this mailing list. So all of a sudden, all 25,000 people received the message. And almost to a person, they used the “reply-all” command and sent: To: Bedlam DL3 From: <User> Subject: RE: Why am I on this mailing list? Please remove me from it. Me too! In addition, there were some really helpful people on the mailing list too: They didn’t respond with just “Me Too!” They responded with: To: Bedlam DL3 From: <User> Subject: RE: Why am I on this mailing list? Please remove me from it. Stop using reply-all – it bogs down the email system. You know what? They were right - the company’s email system did NOT deal with this gracefully. Why? Well, you’ve got to know a bit more about how Exchange works internally. First off, the original mail went to 13,000 users. Assuming that 1,000 of those 13,000 users replied, that means that there are 1,000 replies being sent to those 13,000 users. And it turns out that a number of these people had their email client set to request read receipts and delivery receipts. Each read and delivery receipt causes ANOTHER email to be sent from the recipient back to the sender (all 13,000 recipients). Assuming that 20% of the 1,000 users replying had read receipts or delivery receipts set, that meant that every one of the message that they sent caused another message to be sent for every one of the 13,000 recipients. So how many messages were sent? First there were the basic messages – that’s 13,000,000 messages. Next there were the receipts – 200 users, 13,000 receipts – that’s and additional 2,600,000 messages. So about 15.5 MILLION messages were sent through the system. In about an hour. So at a minimum, 15,600,000 email messages will be delivered into peoples mailboxes. But Exchange can handle 15,600,000 email messages EASILY. There’s another problem that’s somewhat deeper. An Exchange email message actually has TWO recipient lists – there’s the recipient list that the user sees in the To: line on their email message. This is called the P2 recipient list. This is the recipient list that the user typed in. There’s also a SECOND recipient list, called the P1 recipient list that contains the list of ACTUAL recipients of the message. The P1 recipient list is totally hidden from the user, it's used by the MTA to route email messages to the correct destination server. Internally, the P1 list is kept as the original recipient list, plus all of the users on the destination servers. As a result, the P1 list is significantly larger than the P2 list. For the sake of argument, let’s assume that 10% of the recipients on each message (130) are on each server. So each message had 100 recipients in the P1 header, plus the original DL. Assuming 100 bytes per recipient email address, this bloats each email message by 13K. And this assumes that there are 0 bytes in the message – just the headers involve 13K. So those 15,000,000 email messages collectively consumed 195,000,000,000 bytes of bandwidth. Yes, 195 gigabytes of bandwidth bouncing around between the email servers. Compounding this problem was a bug in the MTA that caused the MTA to crash that occurred only when it received a message with more than 8,000 recipients. But it crashed only AFTER processing up to 8,000 recipients. So 8,000 of the 13,000 recipients of the message would get it and 5,000 wouldn’t. When the MTA was restarted, it would immediately start processing the messages in its queue – and since the messages hadn’t been delivered yet, it would retry to deliver the message, sending to the SAME 8,000 recipients and crashing. And because of the way the Exchange store interacts with the MTA, even if we shut down the MTA, the messages would still queue up waiting on delivery to the MTA –shutting down the MTA wouldn’t fix the problem, it would only defer the problem (since the message store would immediately start delivering the queued messages into the MTA the second the MTA came back up). So what did we do to fix it? Well, the first thing that we did was to fix the MTA. And we tried to scrub the MTA’s message queues. This helped a lot, but there were still millions of copies of this message floating around the system. It took about 2 days of constant work before the email system recovered from this one. When it was over, the team firefighting the crisis had t-shirts made with “I survived Bedlam DL3” on the front and “Me Too! (followed by the email addresses of everyone who had replied)” on the back. To prevent anything like this happening in the future, we added a message recipient limit to Exchange – the server now has the ability to enforce a site-wide limit on the number of recipients in a single email message, which neatly prevents this from being a problem in the future. Larry OstermanExchange "12" has a new name!
With Beta 2 approaching, Exchange "12" needed a name. I know the suspense has been killing everyone. Well the wait is finally over: Exchange Server 2007 If you keep hearing the team talk about "E12" , "12", or "Exchange 12" - please understand - it's all the same thing. It just takes a while for us to adjust. Thanks! - Terry Myerson
