Client Connectivity in an Exchange 2016 Coexistence Environment with Exchange 2010
Our goal with this article is to articulate the various client connectivity scenarios you may encounter in your Exchange 2016 designs. To that end, this article will begin with a walk through of a deployment that consists of Exchange 2010 in a multi-site architecture and show how the connectivity changes with the introduction of Exchange 2016. Existing Environment As you can see from the above diagram, this environment contains three Active Directory sites: Internet Facing AD Site (Site1) - This is the main AD site in the environment and has exposure to the Internet. This site has Exchange 2010 servers. There are two namespaces associated with this location – mail.contoso.com and autodiscover.contoso.com resolve to the CAS2010 infrastructure. Regional Internet Facing AD Site (Site2) - This is an AD site that has exposure to the Internet. This site has Exchange 2010 servers. The primary namespace is mail-region.contoso.com and resolves to the CAS2010 infrastructure located within this AD site. Non-Internet Facing AD Site (Site3) - This is an AD site that does not have exposure to the Internet. This site contains Exchange 2010 infrastructure. Note: The term, Internet Facing AD Site, simply means any Active Directory site containing Exchange servers whose virtual directories have the ExternalURL property populated. Similarly, the term, Non-Internet Facing AD Site, simply means any Active Directory site containing Exchange servers whose virtual directories do not have ExternalURL property populated. To understand the client connectivity before we instantiate Exchange 2016 into the environment, let’s look at how each protocol works for each of the three users. Autodiscover The Autodiscover namespace, autodiscover.contoso.com, as well as, the internal SCP records resolve to the CAS2010 infrastructure located in Site1. Outlook clients and ActiveSync clients (on initial configuration) will submit Autodiscover requests to the CAS2010 infrastructure and retrieve configuration settings based on their mailbox’s location. Note: The recommended practice is to configure the 2010 Client Access server’s AutoDiscoverServiceInternalUri value (which is the property value you use to set the SCP record) to point to autodiscover.contoso.com, assuming split-brain DNS is in place. If split-brain DNS is not configured, then set AutoDiscoverServiceInternalUri to a value that resolves to the load balanced VIP for the 2010 Client Access servers in your environment. For more information on how Autodiscover requests are performed, see the whitepaper, Understanding the Exchange 2010 Autodiscover Service. Internal Outlook Connectivity For internal Outlook clients using RPC/TCP connectivity whose mailboxes exist on Exchange 2010, they will connect to the Exchange 2010 RPC Client Access array endpoint (assuming one exists). Keep in mind the importance of configuring the RPC Client Access array endpoint correctly, as documented in Ambiguous URLs and their effect on Exchange 2010 to Exchange 2013 Migrations. Outlook Anywhere Red User will connect to mail.contoso.com as his RPC proxy endpoint. CAS2010 in Site1 will de-encapsulate the RPC data embedded within the HTTP packet and since the target mailbox database is located within the local site, will retrieve the necessary data from the local Exchange 2010 Mailbox server. Blue User will connect to mail.contoso.com as his RPC proxy endpoint. CAS2010 in Site1 will de-encapsulate the RPC data embedded within the HTTP packet, determine that the mailbox server hosting the mailbox is located in another AD site (in this case Site3) and proxy the Outlook RPC data to the target Exchange 2010 Client Access server. Orange User will connect to mail-region.contoso.com as his RPC proxy endpoint. CAS2010 in Site2 will de-encapsulate the RPC data embedded within the HTTP packet and since the target mailbox database is located within the local site, will retrieve the necessary data from the local Exchange 2010 Mailbox server. Note: In addition to the mail and directory connections, Outlook Anywhere clients also utilize Exchange Web Services and an Offline Address Book, url’s for which are provided via the Autodiscover response. Outlook Web App For more information, see the article Upgrading Outlook Web App to Exchange 2010. Red User will connect to mail.contoso.com as his namespace endpoint. CAS2010 in Site1 will authenticate the user, do a service discovery, determine that the mailbox is located within the local AD site and retrieve the necessary data from the Exchange 2010 Mailbox server. Blue User will connect to mail.contoso.com as his namespace endpoint. CAS2010 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox is located within Site3, which does not contain any OWA ExternalURLs. CAS2010 in Site1 will proxy the request to CAS2010 in Site3. CAS2010 in Site3 will retrieve the necessary data from the Exchange 2010 Mailbox server. For the Orange User, there are three possible scenarios depending on what namespace the users enters and how the environment is configured: Orange User will connect to mail-region.contoso.com as his namespace endpoint. CAS2010 in Site2 will authenticate the user, do a service discovery, and determine that the mailbox is located within the local AD site and retrieve the necessary data from the Exchange 2010 Mailbox server. Orange User will connect to mail.contoso.com as his namespace endpoint. CAS2010 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox is located within Site2, which does contain an OWA ExternalURL. CAS2010 in Site1 is not configured to do a cross-site silent redirection, therefore, the user is prompted to use the correct URL to access his mailbox data. Orange User will connect to mail.contoso.com as his namespace endpoint. CAS2010 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox is located within Site2, which does contain an OWA ExternalURL. CAS2010 in Site1 is configured to do a cross-site silent redirection, therefore, CAS2010 will initiate a single sign-on silent redirect (assumes FBA is enabled on source and target) to mail-region.contoso.com. CAS2010 in Site2 will then facilitate the request and retrieve the necessary data from the Exchange 2010 Mailbox server. Exchange ActiveSync For more information, see the article Upgrading Exchange ActiveSync to Exchange 2010. Red User’s ActiveSync client will connect to mail.contoso.com as the namespace endpoint. CAS2010 in Site1 will authenticate the user, do a service discovery, determine that the mailbox is located within the local AD site and retrieve the necessary data from the Exchange 2010 Mailbox server. Blue User’s ActiveSync client will connect to mail.contoso.com as the namespace endpoint. CAS2010 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox is located within Site3, which does not contain any EAS ExternalURLs. CAS2010 in Site1 will issue a cross-site proxy the request to CAS2010 in Site3. CAS2010 in Site3 will retrieve the necessary data from the Exchange 2010 Mailbox server. For the Orange User, there are two possible scenarios: Orange User will connect to mail-region.contoso.com as his namespace endpoint. CAS2010 in Site2 will authenticate the user, do a service discovery, and determine that the mailbox is located within the local AD site and retrieve the necessary data from the Exchange 2010 Mailbox server. Orange User will connect to mail.contoso.com as his namespace endpoint. CAS2010 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox is located within Site2, which does contain an EAS ExternalURL. Assuming the device supports Autodiscover and is on a defined list of devices that supports the 451 redirect response, CAS2010 will issue a 451 response to the device, notifying the device it needs to use a new namespace, mail-region.contoso.com. CAS2010 in Site2 will then facilitate the request and retrieve the necessary data from the Exchange 2010 Mailbox server. If the device is not on the supported list, then CAS2010 in Site1 will proxy the request to CAS2010 in Site2 Exchange Web Services For the Red User, Autodiscover will return the mail.contoso.com namespace for the Exchange Web Service URL. CAS2010 in Site1 will service the request. For the Blue User, Autodiscover will return the mail.contoso.com namespace for the Exchange Web Service URL. CAS2010 will then proxy the request to an Exchange 2010 Client Access server located in Site3. For the Orange User, Autodiscover will return the mail-region.contoso.com namespace for the Exchange Web Service URL. CAS2010 in Site2 will service the request. Client Connectivity with Exchange 2016 in Site1 Exchange 2016 has now been deployed in Site1 following the guidance documented within the Exchange Deployment Assistant. As a result, Outlook Anywhere has been enabled on all Client Access servers within the infrastructure and the mail.contoso.com and autodiscover.contoso.com namespaces have been moved to resolve to Exchange 2016 Client Access component infrastructure. To understand the client connectivity now that Exchange 2016 exists in the environment, let’s look at the four users. Autodiscover The Autodiscover external namespace, autodiscover.contoso.com, as well as, the internal SCP records resolve to the MBX2016 infrastructure located in Site1. Outlook clients and ActiveSync clients (on initial configuration) will submit Autodiscover requests to the MBX2016 infrastructure and depending on the mailbox version, different behaviors occur: For mailboxes that exist on Exchange 2010, MBX2016 will proxy the request to an Exchange 2010 Client Access server that exists within the mailbox’s local site. This means that for Red User, this will be a local proxy to a CAS2010 in Site1. For Blue User and Orange User, this will be a cross-site proxy to a CAS2010 located in the user’s respective site. CAS2010 will then generate the Autodiscover response. For mailboxes that exist on Exchange 2016 (Purple User), MBX2016 will proxy the request to the Exchange 2016 Mailbox server that is hosting the active copy of the user’s mailbox which will generate the Autodiscover response. Internal Outlook Connectivity For internal Outlook clients using RPC/TCP connectivity whose mailboxes exist on Exchange 2010, they will still connect to the Exchange 2010 RPC Client Access array endpoint. When you have an Exchange 2016 mailbox you are using Outlook Anywhere (RPC/HTTP) or MAPI/HTTP, either within the corporate network or outside of the corporate network; RPC/TCP connectivity no longer exists for Exchange 2016 mailboxes. Important: When you introduce Exchange 2016 into a native Exchange 2010 environment, MAPI/HTTP will be enabled by default. Prior to moving any mailboxes to Exchange 2016, ensure you have configured your load balancer and/or firewall rules to allow traffic on /mapi/* via TCP443. In Exchange 2010, the way Outlook Anywhere was implemented is that you had one namespace you could configure. In Exchange 2016, you have both an internal host name and an external host name. Think of it as having two sets of Outlook Anywhere settings, one for when you are connected to the corporate domain, and another for when you are not. You will see this returned to the Outlook client in the Autodiscover response via what looks like a new provider, ExHTTP. However, ExHTTP isn’t an actual provider, it is a calculated set of values from the EXCH (internal Outlook Anywhere) and EXPR (External Outlook Anywhere) settings. To correctly use these settings, the Outlook client must be patched to the appropriate levels (see the Outlook Updates for more information). Outlook will process the ExHTTP in order – internal first and external second. Important: In the event that you are utilizing a split-brain DNS infrastructure, then you must use different names for Outlook Anywhere inside and out. Outlook will also prefer the internal settings over the external settings and since the same namespace is used for both, regardless of whether the client is internal or external, it will utilize only the internal authentication settings. By utilizing two namespaces for Outlook Anywhere, you can ensure that your internal clients can connect utilizing Kerberos authentication. The default Exchange 2016 internal Outlook Anywhere settings don’t require HTTPS. By not requiring SSL, the client should be able to connect and not get a certificate pop-up for the mail and directory connections. However, you will still have to deploy a certificate that is trusted by the client machine for Exchange Web Services and OAB downloads. In environments that are native Exchange 2010, when you introduce Exchange 2016, MAPI/HTTP will be enabled by default. As long as the client supports the protocol, once a mailbox is moved to Exchange 2016, the client will reconfigure itself to use MAPI/HTTP. Upon next boot of the client, MAPI/HTTP will be utilized. It is very important to ensure that you have the correct firewall and load balancer settings in place prior to moving mailboxes, otherwise client connectivity will fail. External Outlook Anywhere & MAPI/HTTP Connectivity In order to support access for Outlook Anywhere clients whose mailboxes are on legacy versions of Exchange, you will need to make some changes to your environment which are documented in the steps within the Exchange Deployment Assistant. Specifically, you will need to enable Outlook Anywhere on your legacy Client Access servers and enable NTLM in addition to basic authentication for the IIS Authentication Method. The Exchange 2016 Client Access component’s RPC proxy component sees the incoming connections, authenticates and chooses which server to route the request to (regardless of version), proxying the HTTP session to the endpoint (Exchange 2010 CAS or Exchange 2016 Mailbox server). Red User will connect to mail.contoso.com as his RPC proxy endpoint. MBX2016 in Site1 will determine the mailbox version is 2010 and that the mailbox database hosting the user’s mailbox is located in the local site. MBX2016 will proxy the request to CAS2010 in Site1. CAS2010 will de-encapsulate the RPC from the HTTP packet and obtain the data from the Exchange 2010 Mailbox server. Purple User will connect to mail.contoso.com as his RPC proxy or MAPI/HTTP endpoint. MBX2016 in Site1 will perform a local proxy request, proxying the HTTP session to the Mailbox 2016 server that hosts the active database copy in Site1. Blue User will connect to mail.contoso.com as his RPC proxy endpoint. MBX2016 in Site1 will determine the mailbox version is 2010 and that the mailbox database hosting the user’s mailbox is located in Site3. MBX2016 will proxy the request to CAS2010 in Site3. CAS2010 will de-encapsulate the RPC from the HTTP packet and obtain the data from the Exchange 2010 Mailbox server. Orange User will continue to access his mailbox using the Exchange 2010 regional namespace, mail-region.contoso.com. Outlook on the web For Outlook on the web, the user experience will depend on the mailbox version and where the mailbox is located. Red User will connect to mail.contoso.com as his namespace endpoint. MBX2016 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox version is 2010 and is located within the local AD site. MBX2016 will proxy the request to an Exchange 2010 Client Access server which will retrieve the necessary data from the Exchange 2010 Mailbox server. Purple User will connect to mail.contoso.com as his endpoint. MBX2016 in Site1 will perform a local proxy request, proxying the HTTP session to the Mailbox 2016 server that hosts the active database copy in Site1. Blue User will connect to mail.contoso.com as his namespace endpoint. MBX2016 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox is located within Site3, which does not contain any OWA ExternalURLs. MBX2016 in Site1 will proxy the request to CAS2010 in Site3. CAS2010 in Site3 will retrieve the necessary data from the Exchange 2010 Mailbox server. For the Orange User, there are two possible scenarios: Orange User will connect to mail-region.contoso.com as his namespace endpoint. In this case, MBX2016 is not involved in any fashion. Orange User will connect to mail.contoso.com as his namespace endpoint. MBX2016 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox is located within Site2, which does contain an OWA ExternalURL. MBX2016 in Site1 will initiate a single sign-on silent redirect (assumes FBA is enabled on source and target) to mail-region.contoso.com. CAS2010 in Site2 will then facilitate the request and retrieve the necessary data from the Exchange 2010 Mailbox server. Exchange ActiveSync For Exchange ActiveSync clients, the user experience will depend on the mailbox version and where the mailbox is located. In addition, Exchange 2016 no longer supports the 451 redirect response – Exchange 2016 will always proxy ActiveSync requests (except in hybrid scenarios, where redirection is allowed). Red User’s ActiveSync client will connect to mail.contoso.com as the namespace endpoint. MBX2016 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox version is 2010 and the mailbox is located within the local AD site. MBX2016 will proxy the request to an Exchange 2010 Client Access server which will retrieve the necessary data from the Exchange 2010 Mailbox server. Purple User’s ActiveSync client will connect to mail.contoso.com as his endpoint. MBX2016 in Site1 will perform a local proxy request, proxying the HTTP session to the Mailbox 2016 server that hosts the active database copy in Site1. Blue User’s ActiveSync client will connect to mail.contoso.com as the namespace endpoint. MBX2016 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox is located within Site3. MBX2016 in Site1 will issue a cross-site proxy the request to CAS2010 in Site3. CAS2010 in Site3 will retrieve the necessary data from the Exchange 2010 Mailbox server. For the Orange User, there are two possible scenarios depending on how the device is configured: Orange User’s ActiveSync client is configured to connect to mail-region.contoso.com as the namespace endpoint. In this case, MBX2016 is not involved in any fashion. Note that any new device that is configured will automatically be configured to use mail-regi on.contoso.com. Orange User’s ActiveSync client is configured to connect to mail.contoso.com as the namespace endpoint. MBX2016 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox version is 2010 and the mailbox is located within another AD site. MBX2016 will issue a cross-site proxy request to an Exchange 2010 Client Access server that resides in the same site as the mailbox. Exchange Web Services Coexistence with Exchange Web Services is rather simple. For the Red User, Autodiscover will return the mail.contoso.com namespace for the Exchange Web Service URL. MBX2016 will then proxy the request to an Exchange 2010 Client Access server within the local site. For the Purple User, Autodiscover will return the mail.contoso.com namespace for the Exchange Web Service URL. MBX2016 in Site1 will perform a local proxy request, proxying the HTTP session to the Mailbox 2016 server that hosts the active database copy in Site1. For the Blue User, Autodiscover will return the mail.contoso.com namespace for the Exchange Web Service URL. MBX2016 will then proxy the request to an Exchange 2010 Client Access server located in Site3. For the Orange User, Autodiscover will return the mail-region.contoso.com namespace for the Exchange Web Service URL. Offline Address Book Like with Exchange Web Services, Autodiscover will provide the Offline Address Book URL. For the Red User, Autodiscover will return the mail.contoso.com namespace for the Offline Address Book URL. MBX2016 will then proxy the request to any Client Access server within the local site that is a web distribution point for the Offline Address Book in question. For the Purple User, Autodiscover will return the mail.contoso.com namespace for the Offline Address Book URL. MBX2016 will then proxy the request to the Exchange 2013/2016 Mailbox server that is hosting the active copy of an OABGEN mailbox that contains a copy of OAB that is closest to the user’s mailbox. For the Blue User, Autodiscover will return the mail.contoso.com namespace for the Offline Address Book URL. MBX2016 will then proxy the request to any Client Access server within the local site that is a web distribution point for the Offline Address Book in question. For the Orange User, Autodiscover will return the mail-region.contoso.com namespace for the Offline Address Book URL. How MBX2016 Picks a Target Legacy Exchange Server It’s important to understand that when MBX2016 proxies to a legacy Exchange Client Access server, it constructs a URL based on the server FQDN, not a load balanced namespace or the InternalURL value. But how does MBX2016 choose which legacy Client Access server to proxy the connection? When a MBX2016 starts up, it connects to Active Directory and enumerates a topology map to understand all the Client Access servers that exist within the environment. Every 50 seconds, MBX2016 will send a lightweight request to each protocol end point to all the Client Access servers in the topology map; these requests have a user agent string of HttpProxy.ClientAccessServer2010Ping. MBX2016 expects a response - a 200/300/400 response series indicates the target server is up for the protocol in question; a 502, 503, or 504 response indicates a failure. If a failure response occurs, MBX2016 immediately retries to determine if the error was a transient error. If this second attempt fails, MBX2016 marks the target CAS as down and excludes it from being a proxy target. At the next interval (50 seconds), MBX2016 will attempt to determine the health state of the down CAS to determine if it is available. The IIS log on a legacy Client Access server will contain the ping events. For example: 2015-08-11 14:00:00 W3SVC1 DF-C14-02 192.168.1.76 HEAD /ecp - 443 - 192.168.1.42 HTTP/1.1 HttpProxy.ClientAccessServer2010Ping - - coe-e16-1.coe.lab 302 0 0 277 170 0 2015-08-11 14:00:00 W3SVC1 DF-C14-02 192.168.1.76 HEAD /PowerShell - 443 - 192.168.1.27 HTTP/1.1 HttpProxy.ClientAccessServer2010Ping - - coe-e16-1.coe.lab 401 0 0 309 177 15 2015-08-11 14:00:00 W3SVC1 DF-C14-02 192.168.1.76 HEAD /EWS - 443 - 192.168.1.134 HTTP/1.1 HttpProxy.ClientAccessServer2010Ping - - coe-e16-1.coe.lab 401 0 0 245 170 0 2015-08-11 14:00:00 W3SVC1 DF-C14-02 192.168.1.76 GET /owa - 443 - 192.168.1.220 HTTP/1.1 HttpProxy.ClientAccessServer2010Ping - - coe-e16-1.coe.lab 301 0 0 213 169 171 2015-08-11 14:00:01 W3SVC1 DF-C14-02 192.168.1.76 HEAD /Microsoft-Server-ActiveSync/default.eas - 443 - 192.168.1.29 HTTP/1.1 HttpProxy.ClientAccessServer2010Ping - - coe-e16-1.coe.lab 401 2 5 293 194 31 2015-08-11 14:00:04 W3SVC1 DF-C14-02 192.168.1.76 HEAD /OAB - 443 - 10.166.18.213 HTTP/1.1 HttpProxy.ClientAccessServer2010Ping - - coe-e16-1.coe.lab 401 2 5 261 170 171 If for some reason, you would like to ensure a particular CAS2010 is never considered a proxy endpoint (or want to remove it for maintenance activities), you can do so by executing the following cmdlet on Exchange 2010: Set-ClientAccessServer <server> -IsOutofService $True IMAP & POP Coexistence All this discussion about HTTP-based clients is great, but what about POP and IMAP clients? Like the HTTP-based client counterparts, IMAP and POP clients are also proxied from the Exchange 2016 Client Access component to a target server (whether that be an Exchange 2016 Mailbox server or a legacy Client Access server). However, there is one key difference, there is no health-checking on the target IMAP/POP services. When the Exchange 2016 Client Access component receives a POP or IMAP request, it will authenticate the user and perform a service discovery. If the target mailbox is E2010, MBX2016 will enumerate the POP or IMAP InternalConnectionSettings property value for each Exchange 2010 Client Access server within the mailbox’s site. Therefore, it is important to ensure that the InternalConnectionSettings maps to the server's FQDN, and not a load-balanced namespace. MBX2016 will choose a server to proxy the request based on the incoming connection’s configuration. If the incoming connection is over an encrypted channel, MBX2016 will try to locate an SSL proxy target first, TLS next, plaintext lastly. If the incoming connection is over a plaintext channel, MBX2016 will try to locate a plaintext proxy target first, SSL next, TLS lastly. Important: Exchange 2016 Mailbox servers do not validate that the POP or IMAP services are actually running on the target Client Access servers. It's important, therefore, to ensure that the services are running on every legacy Client Access server if you have POP or IMAP clients in your environment. Conclusion Hopefully this information dispels some of the myths around proxying and redirection logic for Exchange 2016 when coexisting with Exchange 2010. Please let us know if you have any questions. Ross Smith IV Principal Program Manager Office 365 Customer ExperienceThe Exchange 2016 Preferred Architecture
The Preferred Architecture (PA) is the Exchange Engineering Team’s best practice recommendation for what we believe is the optimum deployment architecture for Exchange 2016, and one that is very similar to what we deploy in Office 365. While Exchange 2016 offers a wide variety of architectural choices for on-premises deployments, the architecture discussed below is our most scrutinized one ever. While there are other supported deployment architectures, they are not recommended. The PA is designed with several business requirements in mind, such as the requirement that the architecture be able to: Include both high availability within the datacenter, and site resilience between datacenters Support multiple copies of each database, thereby allowing for quick activation Reduce the cost of the messaging infrastructure Increase availability by optimizing around failure domains and reducing complexity The specific prescriptive nature of the PA means of course that not every customer will be able to deploy it (for example, customers without multiple datacenters). And some of our customers have different business requirements or other needs which necessitate a different architecture. If you fall into those categories, and you want to deploy Exchange on-premises, there are still advantages to adhering as closely as possible to the PA, and deviate only where your requirements widely differ. Alternatively, you can consider Office 365 where you can take advantage of the PA without having to deploy or manage servers. The PA removes complexity and redundancy where necessary to drive the architecture to a predictable recovery model: when a failure occurs, another copy of the affected database is activated. The PA is divided into four areas of focus: Namespace design Datacenter design Server design DAG design Namespace Design In the Namespace Planning and Load Balancing Principles articles, I outlined the various configuration choices that are available with Exchange 2016. For the namespace, the choices are to either deploy a bound namespace (having a preference for the users to operate out of a specific datacenter) or an unbound namespace (having the users connect to any datacenter without preference). The recommended approach is to utilize the unbounded model, deploying a single Exchange namespace per client protocol for the site resilient datacenter pair (where each datacenter is assumed to represent its own Active Directory site - see more details on that below). For example: autodiscover.contoso.com For HTTP clients: mail.contoso.com For IMAP clients: imap.contoso.com For SMTP clients: smtp.contoso.com Each Exchange namespace is load balanced across both datacenters in a layer 7 configuration that does not leverage session affinity, resulting in fifty percent of traffic being proxied between datacenters. Traffic is equally distributed across the datacenters in the site resilient pair, via round robin DNS, geo-DNS, or other similar solutions. From our perspective, the simpler solution is the least complex and easier to manage, so our recommendation is to leverage round robin DNS. For the Office Online Server farm, a namespace is deployed per datacenter, with the load balancer utilizing layer 7, maintaining session affinity using cookie based persistence. Figure 1: Namespace Design in the Preferred Architecture In the event that you have multiple site resilient datacenter pairs in your environment, you will need to decide if you want to have a single worldwide namespace, or if you want to control the traffic to each specific datacenter by using regional namespaces. Ultimately your decision depends on your network topology and the associated cost with using an unbound model; for example, if you have datacenters located in North America and Europe, the network link between these regions might not only be costly, but it might also have high latency, which can introduce user pain and operational issues. In that case, it makes sense to deploy a bound model with a separate namespace for each region. However, options like geographical DNS offer you the ability to deploy a single unified namespace, even when you have costly network links; geo-DNS allows you to have your users directed to the closest datacenter based on their client’s IP address. Figure 2: Geo-distributed Unbound Namespace Site Resilient Datacenter Pair Design To achieve a highly available and site resilient architecture, you must have two or more datacenters that are well-connected (ideally, you want a low round-trip network latency, otherwise replication and the client experience are adversely affected). In addition, the datacenters should be connected via redundant network paths supplied by different operating carriers. While we support stretching an Active Directory site across multiple datacenters, for the PA we recommend that each datacenter be its own Active Directory site. There are two reasons: Transport site resilience via Shadow Redundancy and Safety Net can only be achieved when the DAG has members located in more than one Active Directory site. Active Directory has published guidance that states that subnets should be placed in different Active Directory sites when the round trip latency is greater than 10ms between the subnets. Server Design In the PA, all servers are physical servers. Physical hardware is deployed rather than virtualized hardware for two reasons: The servers are scaled to use 80% of resources during the worst-failure mode. Virtualization adds an additional layer of management and complexity, which introduces additional recovery modes that do not add value, particularly since Exchange provides that functionality. Commodity server platforms are used in the PA. Commodity platforms are and include: 2U, dual socket servers (20-24 cores) up to 192GB of memory a battery-backed write cache controller 12 or more large form factor drive bays within the server chassis Additional drive bays can be deployed per-server depending on the number of mailboxes, mailbox size, and the server’s scalability. Each server houses a single RAID1 disk pair for the operating system, Exchange binaries, protocol/client logs, and transport database. The rest of the storage is configured as JBOD, using large capacity 7.2K RPM serially attached SCSI (SAS) disks (while SATA disks are also available, the SAS equivalent provides better IO and a lower annualized failure rate). Each disk that houses an Exchange database is formatted with ReFS (with the integrity feature disabled) and the DAG is configured such that AutoReseed formats the disks with ReFS: Set-DatabaseAvailabilityGroup <DAG> -FileSystem ReFS BitLocker is used to encrypt each disk, thereby providing data encryption at rest and mitigating concerns around data theft or disk replacement. For more information, see Enabling BitLocker on Exchange Servers. To ensure that the capacity and IO of each disk is used as efficiently as possible, four database copies are deployed per-disk. The normal run-time copy layout ensures that there is no more than a single active copy per disk. At least one disk in the disk pool is reserved as a hot spare. AutoReseed is enabled and quickly restores database redundancy after a disk failure by activating the hot spare and initiating database copy reseeds. Database Availability Group Design Within each site resilient datacenter pair you will have one or more DAGs. DAG Configuration As with the namespace model, each DAG within the site resilient datacenter pair operates in an unbound model with active copies distributed equally across all servers in the DAG. This model: Ensures that each DAG member’s full stack of services (client connectivity, replication pipeline, transport, etc.) is being validated during normal operations. Distributes the load across as many servers as possible during a failure scenario, thereby only incrementally increasing resource use across the remaining members within the DAG. Each datacenter is symmetrical, with an equal number of DAG members in each datacenter. This means that each DAG has an even number of servers and uses a witness server for quorum maintenance. The DAG is the fundamental building block in Exchange 2016. With respect to DAG size, a larger DAG provides more redundancy and resources. Within the PA, the goal is to deploy larger DAGs (typically starting out with an eight member DAG and increasing the number of servers as required to meet your requirements). You should only create new DAGs when scalability introduces concerns over the existing database copy layout. DAG Network Design The PA leverages a single, non-teamed network interface for both client connectivity and data replication. A single network interface is all that is needed because ultimately our goal is to achieve a standard recovery model regardless of the failure - whether a server failure occurs or a network failure occurs, the result is the same: a database copy is activated on another server within the DAG. This architectural change simplifies the network stack, and obviates the need to manually eliminate heartbeat cross-talk. Note: While your environment may not use IPv6, IPv6 remains enabled per IPv6 support in Exchange. Witness Server Placement Ultimately, the placement of the witness server determines whether the architecture can provide automatic datacenter failover capabilities or whether it will require a manual activation to enable service in the event of a site failure. If your organization has a third location with a network infrastructure that is isolated from network failures that affect the site resilient datacenter pair in which the DAG is deployed, then the recommendation is to deploy the DAG’s witness server in that third location. This configuration gives the DAG the ability to automatically failover databases to the other datacenter in response to a datacenter-level failure event, regardless of which datacenter has the outage. If your organization does not have a third location, consider placing the witness in Azure; alternatively, place the witness server in one of the datacenters within the site resilient datacenter pair. If you have multiple DAGs within the site resilient datacenter pair, then place the witness server for all DAGs in the same datacenter (typically the datacenter where the majority of the users are physically located). Also, make sure the Primary Active Manager (PAM) for each DAG is also located in the same datacenter. Data Resiliency Data resiliency is achieved by deploying multiple database copies. In the PA, database copies are distributed across the site resilient datacenter pair, thereby ensuring that mailbox data is protected from software, hardware and even datacenter failures. Each database has four copies, with two copies in each datacenter, which means at a minimum, the PA requires four servers. Out of these four copies, three of them are configured as highly available. The fourth copy (the copy with the highest Activation Preference number) is configured as a lagged database copy. Due to the server design, each copy of a database is isolated from its other copies, thereby reducing failure domains and increasing the overall availability of the solution as discussed in DAG: Beyond the “A”. The purpose of the lagged database copy is to provide a recovery mechanism for the rare event of system-wide, catastrophic logical corruption. It is not intended for individual mailbox recovery or mailbox item recovery. The lagged database copy is configured with a seven day ReplayLagTime. In addition, the Replay Lag Manager is also enabled to provide dynamic log file play down for lagged copies when availability is compromised. By using the lagged database copy in this manner, it is important to understand that the lagged database copy is not a guaranteed point-in-time backup. The lagged database copy will have an availability threshold, typically around 90%, due to periods where the disk containing a lagged copy is lost due to disk failure, the lagged copy becoming an HA copy (due to automatic play down), as well as, the periods where the lagged database copy is re-building the replay queue. To protect against accidental (or malicious) item deletion, Single Item Recovery or In-Place Hold technologies are used, and the Deleted Item Retention window is set to a value that meets or exceeds any defined item-level recovery SLA. With all of these technologies in play, traditional backups are unnecessary; as a result, the PA leverages Exchange Native Data Protection. Office Online Server Design At a minimum, you will want to deploy two Office Online Servers in each datacenter that hosts Exchange 2016 servers. Each Office Online Server should have 8 processor cores, 32GB of memory and at least 40GB of space dedicated for log files. Note: The Office Online Server infrastructure does not need to be exclusive to Exchange. As such, the hardware guidance takes into account usage by SharePoint and Skype for Business. Be sure to work with any other teams using the Office Online Server infrastructure to ensure the servers are adequately sized for your specific deployment. The Exchange servers within a particular datacenter are configured to use the local Office Online Server farm via the following cmdlet: Set-MailboxServer <East MBX Server> –WACDiscoveryEndPoint https://oos-east.contoso.com/hosting/discovery Summary Exchange Server 2016 continues in the investments introduced in previous versions of Exchange by reducing the server role architecture complexity, aligning with the Preferred Architecture and Office 365 design principles, and improving coexistence with Exchange Server 2013. These changes simplify your Exchange deployment, without decreasing the availability or the resiliency of the deployment. And in some scenarios, when compared to previous generations, the PA increases availability and resiliency of your deployment. Ross Smith IV Principal Program Manager Office 365 Customer ExperienceClient Connectivity in an Exchange 2013 Coexistence Environment
This article is part 3 in a series that discusses namespace planning, load balancing principles, client connectivity, and certificate planning. Over the last several months, we have routinely fielded questions on how various clients connect to the infrastructure once Exchange 2013 is deployed. Our goal with this article is to articulate the various connectivity scenarios you may encounter in your designs. To that end, this article will begin with a walk through of a deployment that consists of Exchange 2007 and Exchange 2010 in a multi-site architecture and show how the connectivity changes with the introduction of Exchange 2013. Existing Environment Figure 1: Exchange 2007 & Exchange 2010 Multi-Site Architecture As you can see from the above diagram, this environment contains three Active Directory sites: Internet Facing AD Site (Site1) - This is the main AD site in the environment and has exposure to the Internet. This site also has a mix of Exchange 2007 and Exchange 2010 servers. There are three namespaces associated with this location – mail.contoso.com and autodiscover.contoso.com resolve to the CAS2010 infrastructure and legacy.contoso.com resolves to the CAS2007 infrastructure. Regional Internet Facing AD Site (Site2) - This is an AD site that has exposure to the Internet. This site has Exchange 2010 servers. The primary namespace is mail-region.contoso.com and resolves to the CAS2010 infrastructure located within this AD site. Non-Internet Facing AD Site (Site3) - This is an AD site that does not have exposure to the Internet. This site contains Exchange 2010 infrastructure. Note: The term, Internet Facing AD Site, simply means any Active Directory site containing Client Access servers whose virtual directories have the ExternalURL property populated. Similarly, the term, Non-Internet Facing AD Site, simply means any Active Directory site containing Client Access servers whose virtual directories do not have ExternalURL property populated. To understand the client connectivity before we instantiate Exchange 2013 into the environment, let’s look at the four users. Autodiscover The Autodiscover namespace, autodiscover.contoso.com, as well as, the internal SCP records resolve to the CAS2010 infrastructure located in Site1. Outlook clients and ActiveSync clients (on initial configuration) will submit Autodiscover requests to the CAS2010 infrastructure and retrieve configuration settings based on their mailbox’s location. The Autodiscover service on Exchange 2010 can process Autodiscover requests for both Exchange 2007 and Exchange 2010 mailboxes. Note: The recommended practice is to configure the 2007 and 2010 Client Access server’s AutoDiscoverServiceInternalUri value (which is the property value you use to set the SCP record) to point to autodiscover.contoso.com, assuming split-brain DNS is in place. If split-brain DNS is not configured, then set AutoDiscoverServiceInternalUri to a value that resolves to the load balanced VIP for the 2010 Client Access servers in your environment. For more information on how Autodiscover requests are performed, see the whitepaper, Understanding the Exchange 2010 Autodiscover Service. Internal Outlook Connectivity For internal Outlook clients using RPC/TCP connectivity whose mailboxes exist on Exchange 2010, they will connect to the Exchange 2010 RPC Client Access array endpoint (assuming one exists). Keep in mind the importance of configuring the RPC Client Access array endpoint correctly, as documented in Ambiguous URLs and their effect on Exchange 2010 to Exchange 2013 Migrations. For internal Outlook clients using RPC/TCP connectivity whose mailboxes exist on Exchange 2007, they will connect directly to the Exchange 2007 Mailbox server instance hosting the mailbox. Outlook Anywhere Red User will connect to mail.contoso.com as his RPC proxy endpoint. CAS2010 in Site1 will de-encapsulate the RPC data embedded within the HTTP packet and since the target mailbox database is located within the local site, will retrieve the necessary data from the local Exchange 2010 Mailbox server. Purple User will connect to mail.contoso.com as his RPC proxy endpoint. CAS2010 in Site1 will de-encapsulate the RPC data embedded within the HTTP packet, determine that the mailbox server is located on an Exchange 2007 server, and proxy the Outlook RPC data to the target Exchange 2007 Mailbox server. Blue User will connect to mail.contoso.com as his RPC proxy endpoint. CAS2010 in Site1 will de-encapsulate the RPC data embedded within the HTTP packet, determine that the mailbox server hosting the mailbox is located in another AD site (in this case Site3) and proxy the Outlook RPC data to the target Exchange 2010 Client Access server. Orange User will connect to mail-region.contoso.com as his RPC proxy endpoint. CAS2010 in Site2 will de-encapsulate the RPC data embedded within the HTTP packet and since the target mailbox database is located within the local site, will retrieve the necessary data from the local Exchange 2010 Mailbox server. Note: In addition to the mail and directory connections, Outlook Anywhere clients also utilize Exchange Web Services and an Offline Address Book, which are provided via the Autodiscover response. Outlook Web App For more information, see the article Upgrading Outlook Web App to Exchange 2010. Red User will connect to mail.contoso.com as his namespace endpoint. CAS2010 in Site1 will authenticate the user, do a service discovery, determine that the mailbox is located within the local AD site and retrieve the necessary data from the Exchange 2010 Mailbox server. Purple User will connect to mail.contoso.com as his namespace endpoint. CAS2010 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox is located within the local AD site on an Exchange 2007 Mailbox server. CAS2010 will initiate a single sign-on silent redirect (assumes FBA is enabled on source and target) to legacy.contoso.com. CAS2007 will then process the request and retrieve the necessary data from the Exchange 2007 Mailbox server. Blue User will connect to mail.contoso.com as his namespace endpoint. CAS2010 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox is located within Site3, which does not contain any OWA ExternalURLs. CAS2010 in Site1 will proxy the request to CAS2010 in Site3. CAS2010 in Site3 will retrieve the necessary data from the Exchange 2010 Mailbox server. For the Orange User, there are three possible scenarios depending on what namespace the users enters and how the environment is configured: Orange User will connect to mail-region.contoso.com as his namespace endpoint. CAS2010 in Site2 will authenticate the user, do a service discovery, and determine that the mailbox is located within the local AD site and retrieve the necessary data from the Exchange 2010 Mailbox server. Orange User will connect to mail.contoso.com as his namespace endpoint. CAS2010 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox is located within Site2, which does contain an OWA ExternalURL. CAS2010 in Site1 is not configured to do a cross-site silent redirection, therefore, the user is prompted to use the correct URL to access his mailbox data. Orange User will connect to mail.contoso.com as his namespace endpoint. CAS2010 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox is located within Site2, which does contain an OWA ExternalURL. CAS2010 in Site1 is configured to do a cross-site silent redirection, therefore, CAS2010 will initiate a single sign-on silent redirect (assumes FBA is enabled on source and target) to mail-region.contoso.com. CAS2010 in Site2 will then facilitate the request and retrieve the necessary data from the Exchange 2010 Mailbox server. Exchange ActiveSync For more information, see the article Upgrading Exchange ActiveSync to Exchange 2010. Red User’s ActiveSync client will connect to mail.contoso.com as the namespace endpoint. CAS2010 in Site1 will authenticate the user, do a service discovery, determine that the mailbox is located within the local AD site and retrieve the necessary data from the Exchange 2010 Mailbox server. Purple User’s ActiveSync client will connect to mail.contoso.com as the namespace endpoint. CAS2010 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox is located within the local AD site on an Exchange 2007 Mailbox server. CAS2010 proxies the request to CAS2007. Blue User’s ActiveSync client will connect to mail.contoso.com as the namespace endpoint. CAS2010 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox is located within Site3, which does not contain any EAS ExternalURLs. CAS2010 in Site1 will issue a cross-site proxy the request to CAS2010 in Site3. CAS2010 in Site3 will retrieve the necessary data from the Exchange 2010 Mailbox server. For the Orange User, there are two possible scenarios: Orange User will connect to mail-region.contoso.com as his namespace endpoint. CAS2010 in Site2 will authenticate the user, do a service discovery, and determine that the mailbox is located within the local AD site and retrieve the necessary data from the Exchange 2010 Mailbox server. Orange User will connect to mail.contoso.com as his namespace endpoint. CAS2010 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox is located within Site2, which does contain an EAS ExternalURL. Assuming the device supports Autodiscover and is on a defined list of devices that supports the 451 redirect response, CAS2010 will issue a 451 response to the device, notifying the device it needs to use a new namespace, mail-region.contoso.com. CAS2010 in Site2 will then facilitate the request and retrieve the necessary data from the Exchange 2010 Mailbox server. If the device is not on the supported list, then CAS2010 in Site1 will proxy the request to CAS2010 in Site2 Exchange Web Services For the Red User, Autodiscover will return the mail.contoso.com namespace for the Exchange Web Service URL. CAS2010 in Site1 will service the request. For the Blue User, Autodiscover will return the mail.contoso.com namespace for the Exchange Web Service URL. CAS2010 will then proxy the request to an Exchange 2010 Client Access server located in Site3. For the Purple User, Autodiscover will provide back the legacy.contoso.com namespace for the Exchange Web Service URL. CAS2007 in Site1 will service the request. For the Orange User, Autodiscover will return the mail-region.contoso.com namespace for the Exchange Web Service URL. CAS2010 in Site2 will service the request. Client Connectivity with Exchange 2013 in Site1 Exchange 2013 has now been deployed in Site1 following the guidance documented within the Exchange Deployment Assistant. As a result, Outlook Anywhere has been enabled on all Client Access servers within the infrastructure and the mail.contoso.com and autodiscover.contoso.com namespaces have been moved to resolve to Exchange 2013 Client Access server infrastructure. Figure 2: Exchange 2013 Coexistence with Exchange 2007 & Exchange 2010 in a Multi-Site Architecture To understand the client connectivity now that Exchange 2013 exists in the environment, let’s look at the four users. Autodiscover The Autodiscover external namespace, autodiscover.contoso.com, as well as, the internal SCP records resolve to the CAS2013 infrastructure located in Site1. Outlook clients and ActiveSync clients (on initial configuration) will submit Autodiscover requests to the CAS2013 infrastructure and depending on the mailbox version, different behaviors occur: For mailboxes that exist on Exchange 2010, CAS2013 will proxy the request to an Exchange 2010 Client Access server that exists within the mailbox’s local site. This means that for Red User, this will be a local proxy to a CAS2010 in Site1. For Blue User and Orange User, this will be a cross-site proxy to a CAS2010 located in the user’s respective site. CAS2010 will then generate the Autodiscover response. For mailboxes that exist on Exchange 2007, CAS2013 will proxy the request to an Exchange 2013 Mailbox server in the local site, which will generate the Autodiscover response. This means for Purple User, the MBX2013 server in Site1 will generate the response. For mailboxes that exist on Exchange 2013, CAS2013 will proxy the request to the Exchange 2013 Mailbox server that is hosting the active copy of the user’s mailbox which will generate the Autodiscover response. Internal Outlook Connectivity For internal Outlook clients using RPC/TCP connectivity whose mailboxes exist on Exchange 2010, they will still connect to the Exchange 2010 RPC Client Access array endpoint. For internal Outlook clients using RPC/TCP connectivity whose mailboxes exist on Exchange 2007, they will still connect directly to the Exchange 2007 Mailbox server instance hosting the mailbox. When you have an Exchange 2013 mailbox you are using Outlook Anywhere, both within the corporate network and outside of the corporate network; RPC/TCP connectivity no longer exists for Exchange 2013 mailboxes. In Exchange 2007/2010, the way Outlook Anywhere was implemented is that you had one namespace you could configure. In Exchange 2013, you have both an internal host name and an external host name. Think of it as having two sets of Outlook Anywhere settings, one for when you are connected to the corporate domain, and another for when you are not. You will see this returned to the Outlook client in the Autodiscover response via what looks like a new provider, ExHTTP. However, ExHTTP isn’t an actual provider, it is a calculated set of values from the EXCH (internal Outlook Anywhere) and EXPR (External Outlook Anywhere) settings. To correctly use these settings, the Outlook client must be patched to the appropriate levels (see the Exchange 2013 System Requirements for more information). Outlook will process the ExHTTP in order – internal first and external second. Important: In the event that you are utilizing a split-brain DNS infrastructure, then you must utilize the same authentication value for both your internal and external Outlook Anywhere settings, or switch to use different names for Outlook Anywhere inside and out. Outlook gives priority to the internal settings over the external settings and since the same namespace is used for both, regardless of whether the client is internal or external, it will utilize only the internal authentication settings. The default Exchange 2013 internal Outlook Anywhere settings don’t require HTTPS. By not requiring SSL, the client should be able to connect and not get a certificate pop-up for the mail and directory connections. However, you will still have to deploy a certificate that is trusted by the client machine for Exchange Web Services and OAB downloads. External Outlook Connectivity In order to support access for Outlook Anywhere clients whose mailboxes are on legacy versions of Exchange, you will need to make some changes to your environment which are documented in the steps within the Exchange Deployment Assistant. Specifically, you will need to enable Outlook Anywhere on your legacy Client Access servers and enable NTLM in addition to basic authentication for the IIS Authentication Method. The Exchange 2013 Client Access server’s RPC proxy component sees the incoming connections, authenticates and chooses which server to route the request to (regardless of version), proxying the HTTP session to the endpoint (legacy CAS or Exchange 2013 Mailbox server). Red User will connect to mail.contoso.com as his RPC proxy endpoint. CAS2013 in Site1 will determine the mailbox version is 2010 and that the mailbox database hosting the user’s mailbox is located in the local site. CAS2013 will proxy the request to CAS2010 in Site1. CAS2010 will de-encapsulate the RPC from the HTTP packet and obtain the data from the Exchange 2010 Mailbox server. Purple User will connect to mail.contoso.com as his RPC proxy endpoint. CAS2013 in Site1 will determine the mailbox version is 2007 and that the mailbox database hosting the user’s mailbox is located in the local site. CAS2013 will proxy the request to CAS2007 in Site1. CAS2007 will de-encapsulate the RPC from the HTTP packet and obtain the data from the Exchange 2007 Mailbox server. Blue User will connect to mail.contoso.com as his RPC proxy endpoint. CAS2013 in Site1 will determine the mailbox version is 2010 and that the mailbox database hosting the user’s mailbox is located in Site3. CAS2013 will proxy the request to CAS2010 in Site3. CAS2010 will de-encapsulate the RPC from the HTTP packet and obtain the data from the Exchange 2010 Mailbox server. Orange User will continue to access his mailbox using the Exchange 2010 regional namespace, mail-region.contoso.com. Outlook Web App For Outlook Web App, the user experience will depend on the mailbox version and where the mailbox is located. Red User will connect to mail.contoso.com as his namespace endpoint. CAS2013 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox version is 2010 and is located within the local AD site. CAS2013 will proxy the request to an Exchange 2010 Client Access server which will retrieve the necessary data from the Exchange 2010 Mailbox server. Purple User will connect to mail.contoso.com as his namespace endpoint. CAS2013 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox is located within the local AD site on an Exchange 2007 Mailbox server. CAS2013 will initiate a single sign-on silent redirect (assumes FBA is enabled on source and target) to legacy.contoso.com. CAS2007 will then facilitate the request and retrieve the necessary data from the Exchange 2007 Mailbox server. Blue User will connect to mail.contoso.com as his namespace endpoint. CAS2013 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox is located within Site3, which does not contain any OWA ExternalURLs. CAS2013 in Site1 will proxy the request to CAS2010 in Site3. CAS2010 in Site3 will retrieve the necessary data from the Exchange 2010 Mailbox server. For the Orange User, there are two possible scenarios: Orange User will connect to mail-region.contoso.com as his namespace endpoint. In this case, CAS2013 is not involved in any fashion. Orange User will connect to mail.contoso.com as his namespace endpoint. CAS2013 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox is located within Site2, which does contain an OWA ExternalURL. CAS2013 in Site1 will initiate a single sign-on silent redirect (assumes FBA is enabled on source and target) to mail-region.contoso.com. CAS2010 in Site2 will then facilitate the request and retrieve the necessary data from the Exchange 2010 Mailbox server. Note: Let’s assume that Site3 contains Exchange 2007 servers as well. In this scenario, CAS2013 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox version is 2007 and the mailbox is located in Site3. CAS2013 will issue a single sign-on silent redirect to legacy.contoso.com. CAS2007 in Site1 will authenticate the user and proxy the request to the Exchange 2007 Client Access server infrastructure in Site3. Exchange ActiveSync For Exchange ActiveSync clients, the user experience will depend on the mailbox version and where the mailbox is located. In addition, Exchange 2013 no longer supports the 451 redirect response – Exchange 2013 will always proxy ActiveSync requests. Red User’s ActiveSync client will connect to mail.contoso.com as the namespace endpoint. CAS2013 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox version is 2010 and the mailbox is located within the local AD site. CAS2013 will proxy the request to an Exchange 2010 Client Access server which will retrieve the necessary data from the Exchange 2010 Mailbox server. Purple User’s ActiveSync client will connect to mail.contoso.com as the namespace endpoint. CAS2013 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox version is 2007. CAS2013 will proxy the request to a local Exchange 2013 Mailbox server. The Exchange 2013 Mailbox server will send the request to an Exchange 2007 Client Access server that exists within the mailbox’s site (specifically, the InternalURL value of the Microsoft-Server-ActiveSync virtual directory on CAS2007), which will retrieve the data from the Exchange 2007 Mailbox server. Blue User’s ActiveSync client will connect to mail.contoso.com as the namespace endpoint. CAS2013 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox is located within Site3. CAS2013 in Site1 will issue a cross-site proxy the request to CAS2010 in Site3. CAS2010 in Site3 will retrieve the necessary data from the Exchange 2010 Mailbox server. For the Orange User, there are two possible scenarios depending on how the device is configured: Orange User’s ActiveSync client is configured to connect to mail-region.contoso.com as the namespace endpoint. In this case, CAS2013 is not involved in any fashion. Note that any new device that is configured will automatically be configured to use mail-region.contoso.com. Orange User’s ActiveSync client is configured to connect to mail.contoso.com as the namespace endpoint. CAS2013 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox version is 2010 and the mailbox is located within another AD site. CAS2013 will issue a cross-site proxy request to an Exchange 2010 Client Access server that resides in the same site as the mailbox. Note: Let’s assume that Site3 contains Exchange 2007 servers as well. In this scenario, CAS2013 in Site1 will authenticate the user, do a service discovery, and determine that the mailbox version is 2007 and the mailbox is located in Site3. CAS2013 will proxy the request to a local Exchange 2013 Mailbox server. The Exchange 2013 Mailbox server will send the request to an Exchange 2007 Client Access server that exists within the mailbox’s site, which will retrieve the data from the Exchange 2007 Mailbox server. Exchange Web Services Coexistence with Exchange Web Services is rather simple. For the Red User, Autodiscover will return the mail.contoso.com namespace for the Exchange Web Service URL. CAS2013 will then proxy the request to an Exchange 2010 Client Access server within the local site. For the Purple User, Autodiscover will return the legacy.contoso.com namespace for the Exchange Web Service URL. For the Blue User, Autodiscover will return the mail.contoso.com namespace for the Exchange Web Service URL. CAS2013 will then proxy the request to an Exchange 2010 Client Access server located in Site3. For the Orange User, Autodiscover will return the mail-region.contoso.com namespace for the Exchange Web Service URL. Note: Let’s assume that Site3 contains Exchange 2007 servers as well. In this scenario, the Autodiscover response will provide back the legacy.contoso.com namespace for the Exchange Service URL. CAS2007 in Site1 will proxy the request to an Exchange 2007 Client Access server that exists within the mailbox’s site, which will retrieve the data from the Exchange 2007 Mailbox server. This is why you cannot remove Exchange 2007 from the Internet-facing Active Directory site as long as Exchange 2007 exists in non-Internet facing Active Directory sites. Offline Address Book Like with Exchange Web Services, Autodiscover will provide the Offline Address Book URL. For the Red User, Autodiscover will return the mail.contoso.com namespace for the Offline Address Book URL. CAS2013 will then proxy the request to any Client Access server within the local site that is a web distribution point for the Offline Address Book in question. For the Purple User, Autodiscover will return the legacy.contoso.com namespace for the Offline Address Book URL. For the Blue User, Autodiscover will return the mail.contoso.com namespace for the Offline Address Book URL. CAS2013 will then proxy the request to any Client Access server within the local site that is a web distribution point for the Offline Address Book in question. For the Orange User, Autodiscover will return the mail-region.contoso.com namespace for the Offline Address Book URL. Note: Let’s assume that Site3 contains Exchange 2007 servers as well. In this scenario, the Autodiscover response will provide back the legacy.contoso.com namespace for the Offline Address Book URL. This is why you cannot remove Exchange 2007 from the Internet-facing Active Directory site as long as Exchange 2007 exists in non-Internet facing Active Directory sites. How CAS2013 Picks a Target Legacy Exchange Server It’s important to understand that when CAS2013 proxies to a legacy Exchange Client Access server, it constructs a URL based on the server FQDN, not a load balanced namespace or the InternalURL value.But how does CAS2013 choose which legacy Client Access server to proxy the connection? When a CAS2013 starts up, it connects to Active Directory and enumerates a topology map to understand all the Client Access servers that exist within the environment. Every 50 seconds, CAS2013 will send a lightweight request to each protocol end point to all the Client Access servers in the topology map; these requests have a user agent string of HttpProxy.ClientAccessServer2010Ping (yes, even Exchange 2007 servers are targeted with that user string). CAS2013 expects a response - a 200/300/400 response series indicates the target server is up for the protocol in question; a 502, 503, or 504 response indicates a failure. If a failure response occurs, CAS2013 immediately retries to determine if the error was a transient error. If this second attempt fails, CAS2013 marks the target CAS as down and excludes it from being a proxy target. At the next interval (50 seconds), CAS2013 will attempt to determine the health state of the down CAS to determine if it is available. The IIS log on a legacy Client Access server will contain the ping events. For example: 2014-03-11 14:00:00 W3SVC1 DF-C14-02 157.54.7.76 HEAD /ecp - 443 - 192.168.1.42 HTTP/1.1 HttpProxy.ClientAccessServer2010Ping - - coe-e14-1.coe.lab 302 0 0 277 170 0 2014-03-11 14:00:00 W3SVC1 DF-C14-02 157.54.7.76 HEAD /PowerShell - 443 - 192.168.1.27 HTTP/1.1 HttpProxy.ClientAccessServer2010Ping - - coe-e14-1.coe.lab 401 0 0 309 177 15 2014-03-11 14:00:00 W3SVC1 DF-C14-02 157.54.7.76 HEAD /EWS - 443 - 192.168.1.134 HTTP/1.1 HttpProxy.ClientAccessServer2010Ping - - coe-e14-1.coe.lab 401 0 0 245 170 0 2014-03-11 14:00:00 W3SVC1 DF-C14-02 157.54.7.76 GET /owa - 443 - 192.168.1.220 HTTP/1.1 HttpProxy.ClientAccessServer2010Ping - - coe-e14-1.coe.lab 301 0 0 213 169 171 2014-03-11 14:00:01 W3SVC1 DF-C14-02 157.54.7.76 HEAD /Microsoft-Server-ActiveSync/default.eas - 443 - 192.168.1.29 HTTP/1.1 HttpProxy.ClientAccessServer2010Ping - - coe-e14-1.coe.lab 401 2 5 293 194 31 2014-03-11 14:00:04 W3SVC1 DF-C14-02 157.54.7.76 HEAD /OAB - 443 - 10.166.18.213 HTTP/1.1 HttpProxy.ClientAccessServer2010Ping - - coe-e14-1.coe.lab 401 2 5 261 170 171 If for some reason, you would like to ensure a particular CAS2010 is never considered a proxy endpoint (or want to remove it for maintenance activities), you can do so by executing the following cmdlet on Exchange 2010 (note that this feature does not exist on Exchange 2007): Set-ClientAccessServer <server> -IsOutofService $True IMAP & POP Coexistence All this discussion about HTTP-based clients is great, but what about POP and IMAP clients? Like the HTTP-based client counterparts, IMAP and POP clients are also proxied from the Exchange 2013 Client Access server to a target server (whether that be an Exchange 2013 Mailbox server or a legacy Client Access server). However, there is one key difference, there is no health-checking on the target IMAP/POP services. When the Exchange 2013 Client Access server receives a POP or IMAP request, it will authenticate the user and perform a service discovery. If the target mailbox is E2010, CAS2013 will enumerate the POP or IMAP InternalConnectionSettings property value for each Exchange 2010 Client Access server within the mailbox’s site. Therefore, it is important to ensure that the InternalConnectionSettings maps to the server's FQDN, and not a load-balanced namespace. If the target mailbox is E2007, CAS2013 will enumerate each Exchange 2007 Client Access server’s server FQDN within the mailbox’s site. CAS2013 will choose a server to proxy the request based on the incoming connection’s configuration. If the incoming connection is over an encrypted channel, CAS2013 will try to locate an SSL proxy target first, TLS next, plaintext lastly. If the incoming connection is over a plaintext channel, CAS2013 will try to locate a plaintext proxy target first, SSL next, TLS lastly. Important: Exchange 2013 Client Access servers do not validate that the POP or IMAP services are actually running on the target Client Access servers. It's important, therefore, to ensure that the services are running on every legacy Client Access server if you have POP or IMAP clients in your environment. Conclusion Hopefully this information dispels some of the myths around proxying and redirection logic for Exchange 2013 when coexisting with either Exchange 2007 or Exchange 2010. Please let us know if you have any questions. Ross Smith IV Principal Program Manager Office 365 Customer Experience Updates 3/14/2014: Added section on IMAP/POP coexistence 3/25/2014: Added detail that MBX2013 proxies to the CAS2007 MSAS InternalURL for 2007 EAS mailbox accessExchange 2013 Client Access Server Role
In a previous article, I discussed the new server role architecture in Exchange 2013. This article continues the series by discussing the Client Access server role. While this Exchange server role shares the same name as a server role that existed in the last two Exchange Server releases, it is markedly different. In Exchange 2007, the Client Access server role provided authentication, proxy/redirection logic, and performed data rendering for the Internet protocol clients (Outlook Web App, EAS , EWS , IMAP and POP ). In Exchange 2010, data rendering for MAPI was also moved to the Client Access server role. In Exchange 2013, the Client Access server (CAS) role no longer performs any data rendering functionality. The Client Access server role now only provides authentication and proxy/redirection logic, supporting the client Internet protocols, transport, and Unified Messaging. As a result of this architectural shift, the CAS role is stateless (from a protocol session perspective, log data that can be used in troubleshooting or trending analysis is generated, naturally). Session Affinity As I alluded to in the sever role architecture blog post, Exchange 2013 no longer requires session affinity at the load balancer. To understand this better, we need to look at how CAS 2013 functions. From a protocol perspective, the following will happen: A client resolves the namespace to a load balanced virtual IP address. The load balancer assigns the session to a CAS member in the load balanced pool. CAS authenticates the request and performs a service discovery by accessing Active Directory to retrieve the following information: Mailbox version (for this discussion, we will assume an Exchange 2013 mailbox) Mailbox location information (e.g., database information, ExternalURL values, etc.) CAS makes a decision on whether to proxy the request or redirect the request to another CAS infrastructure (within the same forest). CAS queries an Active Manager instance that is responsible for the database to determine which Mailbox server is hosting the active copy. CAS proxies the request to the Mailbox server hosting the active copy. The protocol used in step 6 depends on the protocol used to connect to CAS . If the client leverages the HTTP protocol, then the protocol used between the Client Access server and Mailbox server is HTTP (secured via SSL using a self-signed certificate). If the protocol leveraged by the client is IMAP or POP, then the protocol used between the Client Access server and Mailbox server is IMAP or POP. Telephony requests are unique, however. Instead of proxying the request at step 6, CAS will redirect the request to the Mailbox server hosting the active copy of the user’s database, as the telephony devices support redirection and need to establish their SIP and RTP sessions directly with the Unified Messaging components on the Mailbox server. Figure 1: Exchange 2013 Client Access Protocol Architecture In addition to no longer performing data rendering, step 5 is the fundamental change that enables the removal of session affinity at the load balancer. For a given protocol session, CAS now maintains a 1:1 relationship with the Mailbox server hosting the user’s data. In the event that the active database copy is moved to a different Mailbox server, CAS closes the sessions to the previous server and establishes sessions to the new server. This means that all sessions, regardless of their origination point (i.e., CAS members in the load balanced array), end up at the same place, the Mailbox server hosting the active database copy. Now many of you may be thinking, wait how does authentication work? Well for HTTP, POP, or IMAP requests that use basic, NTLM, or Kerberos authentication, the authentication request is passed as part of the HTTP payload, so each CAS will authenticate the request naturally. Forms-based authentication (FBA) is different. FBA was one of the reasons why session affinity was required for OWA in previous releases of Exchange – the reason being that that the cookie used a per server key for encryption; so if another CAS received a request, it could not decrypt the session. In Exchange 2013, we no longer leverage a per server session key; instead we leverage the private key from the certificate that is installed on the CAS. As long as all members of the CAS array share the exact same certificate (remember we actually recommend deploying the same certificate across all CAS in both datacenters in site resilience scenarios as well), they can decrypt the cookie. Proxy vs. Redirection In the previous section, I spoke about CAS proxying the data to the Mailbox server hosting the active database copy. Prior to that, CAS has to make a decision whether it will perform the proxy action or perform a redirection action. CAS will only perform a redirection action under the following circumstances: The origination request is telephony related. For Outlook Web App requests, if the mailbox’s location is determined to be in another Active Directory site and there are CAS2013 members in that site that have the ExternalURL populated, then the originating CAS will redirect the request unless the ExternalURL in the target site is the same as in the originating site (and the ExternalURL is the URL entered in the web browser by the user) – in which case CAS will proxy (this is the multiple site single namespace scenario). Figure 2: Exchange 2013 Client Access Proxy and Redirection Behavior Examples For OWA requests, if the mailbox version is Exchange 2007, then CAS2013 will redirect the request to CAS2007. Outlook Connectivity For those of you paying attention, you may have noticed I only spoke about HTTP, POP, and IMAP. I didn’t mention RPC/TCP as connectivity solution that CAS supports. And that is for a very specific reason – CAS2013 does not support RPC/TCP as a connectivity solution; it only supports RPC/HTTP (aka Outlook Anywhere). This architecture change is primarily to drive a stable and reliable connectivity model. To understand why, you need to keep the following tenets in the back of your mind: Remember CAS2013 is an authentication and proxy/redirection server. It does no processing of the data (no rendering or transformation). It simply proxies the request to MBX2013 using the client protocol. In this case HTTP. CAS2013 and MBX2013 are not tied together from a user affinity or geographical perspective. You can have CAS2013 in one datacenter authenticate the request and proxy the request to a MBX2013 server in another datacenter. To enable this we had to change the communication protocols used between server roles. Moving away from RPC to the client protocols that are more tolerant of throughput & latency over WAN /Internet connections. For a given mailbox, the protocol that services the request is always going to be the protocol instance on the Mailbox server that hosts the active copy of the database for the user’s mailbox. This was done to ultimately uncouple versioning and functionality issues we’ve seen in the past two generations (i.e., have to deploy CAS2010, HT2010, MBX2010 together to get certain functionality and upgrading one doesn’t necessarily give you new capabilities and may break connectivity). The last item is tied to this discussion of why we have moved away from RPC/TCP as a connectivity solution. In all prior releases the RPC endpoint was a FQDN . In fact, the shift to the middle tier for RPC processing in CAS2010 introduced a new shared namespace, the RPC Client Access namespace. By moving RPC Client Access back to the MBX2013 role, this would have forced us to use either the MBX2013 FQDN for the RPC endpoint (thus forcing an Outlook client restart for every database *over event) or a shared namespace for the DAG . Neither option is appropriate and adds to the complexity and support of the infrastructure. So instead, we changed the model. We no longer use a FQDN for the RPC endpoint. Instead we now use a GUID. The mailbox GUID, to be precise (along with a domain suffix to support multi-tenant scenarios). The mailbox GUID is unique within the (tenant) organization, so regardless of where the database is activated and mounted, CAS can discover the location and proxy the request to the correct MBX2013 server. Figure 3: RPC Endpoint Changes This architectural change means that we have a very reliable connection model – for a given session that is routed to CAS2013, CAS2013 will always have a 1:1 relationship with the MBX2013 server hosting the user’s mailbox. This means that the Mailbox server hosting the active copy of the user’s database is the server responsible for de-encapsulating the RPC stream from the HTTP packets. In the event a *over occurs, CAS2013 will proxy the connection to MBX2013 that assumes the responsibility of hosting the active database copy. Oh, and this means in a native Exchange 2013 environment, Outlook won’t require a restart for things like mailbox moves, *over events, etc. The other architectural change we made in this area is the support for internal and external namespaces for Outlook Anywhere. This means you may not need to deploy split-brain DNS or deal with all Outlook clients using your external firewall or load balancer due to our change in MAPI connectivity. Third-Party MAPI Products I am sure that a few of you are wondering what this change means for third-party MAPI products. The answer is relatively simple – these third-party solutions will need to leverage RPC/HTTP to connect to CAS2013. This will be accomplished via a new MAPI/CDO download that that has been updated to include support for RPC/HTTP connectivity. It will be released in the first quarter of calendar year 2013. To leverage this updated functionality, the third-party vendor will either have to programmatically edit the dynamic MAPI profile or set registry key values to enable RPC/HTTP support. I do also want to stress one key item with respect to third-party MAPI support. Exchange 2013 is the last release that will support a MAPI/CDO custom solution. In the future, third-party products (and custom in-house developed solutions) will need to move to Exchange Web Services (EWS) to access Exchange data. Namespace Simplification Another benefit with the Exchange 2013 architecture is that the namespace model can be simplified (especially for those of you upgrading from Exchange 2010). In Exchange 2010, a customer that wanted to deploy a site-resilient solution for two datacenters required the following namespaces: Primary datacenter Internet protocol namespace Secondary datacenter Internet protocol namespace Primary datacenter Outlook Web App failback namespace Secondary datacenter Outlook Web App failback namespace Primary datacenter RPC Client Access namespace Secondary datacenter RPC Client Access namespace Autodiscover namespace Legacy namespace Transport namespace (if doing ad-hoc encryption or partner-to-partner encryption) As I previously mentioned, we have removed two of these namespaces in Exchange 2013 – the RPC Client Access namespaces. Recall that CAS2013 proxies requests to the Mailbox server hosting the active database copy. This proxy logic is not limited to the Active Directory site boundary. A CAS2013 in one Active Directory site can proxy a session to a Mailbox server that is located in another Active Directory site. If network utilization, latency, and throughput are not a concern, this means that we do not need the additional namespaces for site resilience scenarios, thereby eliminating three other namespaces (secondary Internet protocol and both Outlook Web App failback namespaces). For example, let’s say I have a two datacenter deployment in North America that has a network configuration such that latency, throughput and utilization between the datacenters is not a concern. I also wanted to simplify my namespace architecture with the Exchange 2013 deployment so that my users only have to use a single namespace for Internet access regardless of where their mailbox is located. If I deployed an architecture like below, then the CAS infrastructure in both datacenters could be used to route and proxy traffic to the Mailbox servers hosting the active copies. Since I am not concerned about network traffic, I configure DNS to round-robin between the VIP s of the load balancers in each datacenter. The end result is that I have a site resilient namespace architecture while accepting that half of my proxy traffic will be out-of-site. Figure 4: Exchange 2013 Single Namespace Example Transport Early on I mentioned that the Client Access Server role can proxy SMTP sessions. This is handled by a new component on the CAS2013 role, the Front-End Transport service. The Front-End Transport service handles all inbound and outbound external SMTP traffic for the Exchange organization, as well as, can be a client endpoint for SMTP traffic. The Front-End Transport Service functions as a layer 7 proxy and has full access to the protocol conversation. Like the client Internet protocols, the Front-End Transport service does not have a message queue and is completely stateless. In addition, the Front-End Transport service does not perform message bifurcation. The Front-End Transport service listens on TCP25, TCP587, and TCP717 as seen in the following diagram: Figure 5: Front-End Transport Service Architecture The Front-End Transport service provides network protection – a centralized, load balanced egress/ingress point for the Exchange organization, whether it be POP/IMAP clients, SharePoint, other third-party or custom in-house mail applications, or external SMTP systems. For outgoing messages, the Front-End Transport service is used as a proxy when the Send Connectors (that are located on the Mailbox server) have the FrontEndProxyEnabled property set. In this situation, the message will appear to have originated from CAS2013. For incoming messages, the Front-End Transport service must quickly find a single, healthy Transport service on a Mailbox server to receive the message transmission, regardless of the number or type of recipients: For messages with a single mailbox recipient, select a Mailbox server in the target delivery group, and give preference to the Mailbox server based on the proximity of the Active Directory site. For messages with multiple mailbox recipients, use the first 20 recipients to select a Mailbox server in the closest delivery group, based on the proximity of the Active Directory site. If the message has no mailbox recipients, select a random Mailbox server in the local Active Directory site. Conclusion The Exchange 2013 Client Access Server role simplifies the network layer. Session affinity at the load balancer is no longer required as CAS2013 handles the affinity aspects. CAS2013 introduces more deployment flexibility by allowing you to simplify your namespace architecture, potentially consolidating to a single world-wide or regional namespace for your Internet protocols. The new architecture also simplifies the upgrade and inter-operability story as CAS2013 can proxy or redirect to multiple versions of Exchange, whether they are a higher or lower version, allowing you to upgrade your Mailbox servers at your own pace. Ross Smith IV Principal Program Manager Exchange Customer ExperienceThe Preferred Architecture
During my session at the recent Microsoft Exchange Conference (MEC), I revealed Microsoft’s preferred architecture (PA) for Exchange Server 2013. The PA is the Exchange Engineering Team’s prescriptive approach to what we believe is the optimum deployment architecture for Exchange 2013, and one that is very similar to what we deploy in Office 365. While Exchange 2013 offers a wide variety of architectural choices for on-premises deployments, the architecture discussed below is our most scrutinized one ever. While there are other supported deployment architectures, other architectures are not recommended. The PA is designed with several business requirements in mind. For example, requirements that the architecture be able to: Include both high availability within the datacenter, and site resilience between datacenters Support multiple copies of each database, thereby allowing for quick activation Reduce the cost of the messaging infrastructure Increase availability by optimizing around failure domains and reducing complexity The specific prescriptive nature of the PA means of course that not every customer will be able to deploy it (for example, customers without multiple datacenters). And some of our customers have different business requirements or other needs, which necessitate an architecture different from that shown here. If you fall into those categories, and you want to deploy Exchange on-premises, there are still advantages to adhering as closely as possible to the PA where possible, and deviate only where your requirements widely differ. Alternatively, you can consider Office 365 where you can take advantage of the PA without having to deploy or manage servers. Before I delve into the PA, I think it is important that you understand a concept that is the cornerstone for this architecture – simplicity. Simplicity Failure happens. There is no technology that can change this. Disks, servers, racks, network appliances, cables, power substations, generators, operating systems, applications (like Exchange), drivers, and other services – there is simply no part of an IT services offering that is not subject to failure. One way to mitigate failure is to build in redundancy. Where one entity is likely to fail, two or more entities are used. This pattern can be observed in Web server arrays, disk arrays, and the like. But redundancy by itself can be prohibitively expensive (simple multiplication of cost). For example, the cost and complexity of the SAN based storage system that was at the heart of Exchange until the 2007 release, drove the Exchange Team to step up its investment in the storage stack and to evolve the Exchange application to integrate the important elements of storage directly into its architecture. We recognized that every SAN system would ultimately fail, and that implementing a highly redundant system using SAN technology would be cost-prohibitive. In response, Exchange has evolved from requiring expensive, scaled-up, high-performance SAN storage and related peripherals, to now being able to run on cheap, scaled-out servers with commodity, low-performance SAS/SATA drives in a JBOD configuration with commodity disk controllers. This architecture enables Exchange to be resilient to any storage related failure, while enabling you to deploy large mailboxes at a reasonable cost. By building the replication architecture into Exchange and optimizing Exchange for commodity storage, the failure mode is predictable from a storage perspective. This approach does not stop at the storage layer; redundant NICs, power supplies, etc., can also be removed from the server hardware. Whether it is a disk, controller, or motherboard that fails, the end result should be the same, another database copy is activated and takes over. The more complex the hardware or software architecture, the more unpredictable failure events can be. Managing failure at any scale is all about making recovery predictable, which drives the necessity to having predictable failure modes. Examples of complex redundancy are active/passive network appliance pairs, aggregation points on the network with complex routing configurations, network teaming, RAID, multiple fiber pathways, etc. Removing complex redundancy seems unintuitive on its face – how can removing redundancy increase availability? Moving away from complex redundancy models to a software-based redundancy model, creates a predictable failure mode. The PA removes complexity and redundancy where necessary to drive the architecture to a predictable recovery model: when a failure occurs, another copy of the affected database is activated. The PA is divided into four areas of focus: Namespace design Datacenter design Server design DAG design Namespace Design In the Namespace Planning and Load Balancing Principles articles, I outlined the various configuration choices that are available with Exchange 2013. From a namespace perspective, the choices are to either deploy a bound namespace (having a preference for the users to operate out of a specific datacenter) or an unbound namespace (having the users connect to any datacenter without preference). The recommended approach is to utilize the unbound model, deploying a single namespace per client protocol for the site resilient datacenter pair (where each datacenter is assumed to represent its own Active Directory site - see more details on that below). For example: autodiscover.contoso.com For HTTP clients: mail.contoso.com For IMAP clients: imap.contoso.com For SMTP clients: smtp.contoso.com Figure 1: Namespace Design Each namespace is load balanced across both datacenters in a configuration that does not leverage session affinity, resulting in fifty percent of traffic being proxied between datacenters. Traffic is equally distributed across the datacenters in the site resilient pair, via DNS round-robin, geo-DNS, or other similar solution you may have at your disposal. Though from our perspective, the simpler solution is the least complex and easier to manage, so our recommendation is to leverage DNS round-robin. In the event that you have multiple site resilient datacenter pairs in your environment, you will need to decide if you want to have a single worldwide namespace, or if you want to control the traffic to each specific datacenter pair by using regional namespaces. Ultimately your decision depends on your network topology and the associated cost with using an unbound model; for example, if you have datacenters located in North America and Europe, the network link between these regions might not only be costly, but it might also have high latency, which can introduce user pain and operational issues. In that case, it makes sense to deploy a bound model with a separate namespace for each region. Site Resilient Datacenter Pair Design To achieve a highly available and site resilient architecture, you must have two or more datacenters that are well-connected (ideally, you want a low round-trip network latency, otherwise replication and the client experience are adversely affected). In addition, the datacenters should be connected via redundant network paths supplied by different operating carriers. While we support stretching an Active Directory site across multiple datacenters, for the PA we recommend having each datacenter be its own Active Directory site. There are two reasons: Transport site resilience via Shadow Redundancy and Safety Net can only be achieved when the DAG has members located in more than one Active Directory site. Active Directory has published guidance that states that subnets should be placed in different Active Directory sites when the round trip latency is greater than 10ms between the subnets. Server Design In the PA, all servers are physical, multi-role servers. Physical hardware is deployed rather than virtualized hardware for two reasons: The servers are scaled to utilize eighty percent of resources during the worst-failure mode. Virtualization adds an additional layer of management and complexity, which introduces additional recovery modes that do not add value, as Exchange provides equivalent functionality out of the box. By deploying multi-role servers, the architecture is simplified as all servers have the same hardware, installation process, and configuration options. Consistency across servers also simplifies administration. Multi-role servers provide more efficient use of server resources by distributing the Client Access and Mailbox resources across a larger pool of servers. Client Access and Database Availability Group (DAG) resiliency is also increased, as there are more servers available for the load-balanced pool and for the DAG. Commodity server platforms (e.g., 2U, dual socket servers with no more than 24 processor cores and 96GB of memory, that hold 12 large form-factor drive bays within the server chassis) are use in the PA. Additional drive bays can be deployed per-server depending on the number of mailboxes, mailbox size, and the server’s scalability. Each server houses a single RAID1 disk pair for the operating system, Exchange binaries, protocol/client logs, and transport database. The rest of the storage is configured as JBOD, using large capacity 7.2K RPM serially attached SCSI (SAS) disks (while SATA disks are also available, the SAS equivalent provides better IO and a lower annualized failure rate). Bitlocker is used to encrypt each disk, thereby providing data encryption at rest and mitigating concerns around data theft via disk replacement. To ensure that the capacity and IO of each disk is used as efficiently as possible, four database copies are deployed per-disk. The normal run-time copy layout (calculated in the Exchange 2013 Server Role Requirements Calculator) ensures that there is no more than a single copy activated per-disk. Figure 2: Server Design At least one disk in the disk pool is reserved as a hot spare. AutoReseed is enabled and quickly restores database redundancy after a disk failure by activating the hot spare and initiating database copy reseeds. Database Availability Group Design Within each site resilient datacenter pair you will have one or more DAGs. DAG Configuration As with the namespace model, each DAG within the site resilient datacenter pair operates in an unbound model with active copies distributed equally across all servers in the DAG. This model provides two benefits: Ensures that each DAG member’s full stack of services is being validated (client connectivity, replication pipeline, transport, etc.). Distributes the load across as many servers as possible during a failure scenario, thereby only incrementally increasing resource utilization across the remaining members within the DAG. Each datacenter is symmetrical, with an equal number of member servers within a DAG residing in each datacenter. This means that each DAG contains an even number of servers and uses a witness server for quorum arbitration. The DAG is the fundamental building block in Exchange 2013. With respect to DAG size, a larger DAG provides more redundancy and resources. Within the PA, the goal is to deploy larger DAGs (typically starting out with an eight member DAG and increasing the number of servers as required to meet your requirements) and only create new DAGs when scalability introduces concerns over the existing database copy layout. DAG Network Design Since the introduction of continuous replication in Exchange 2007, Exchange has recommended multiple replication networks for separating client traffic from replication traffic. Deploying two networks allows you to isolate certain traffic along different network pathways and ensure that during certain events (e.g., reseed events) the network interface is not saturated (which is an issue with 100Mb, and to a certain extent, 1Gb interfaces). However, for most customers, having two networks operating in this manner was only a logical separation, as the same copper fabric was used by both networks in the underlying network architecture. With 10Gb networks becoming the standard, the PA moves away from the previous guidance of separating client traffic from replication traffic. A single network interface is all that is needed because ultimately our goal is to achieve a standard recovery model despite the failure - whether a server failure occurs or a network failure occurs, the result is the same, a database copy is activated on another server within the DAG. This architectural change simplifies the network stack, and obviates the need to eliminate heartbeat cross-talk. Witness Server Placement Ultimately, the placement of the witness server determines whether the architecture can provide automatic datacenter failover capabilities or whether it will require a manual activation to enable service in the event of a site failure. If your organization has a third location with a network infrastructure that is isolated from network failures that affect the site resilient datacenter pair in which the DAG is deployed, then the recommendation is to deploy the DAG’s witness server in that third location. This configuration gives the DAG the ability to automatically failover databases to the other datacenter in response to a datacenter-level failure event, regardless of which datacenter has the outage. Figure 3: DAG (Three Datacenter) Design If your organization does not have a third location, then place the witness server in one of the datacenters within the site resilient datacenter pair. If you have multiple DAGs within the site resilient datacenter pair, then place the witness server for all DAGs in the same datacenter (typically the datacenter where the majority of the users are physically located). Also, make sure the Primary Active Manager (PAM) for each DAG is also located in the same datacenter. Data Resiliency Data resiliency is achieved by deploying multiple database copies. In the PA, database copies are distributed across the site resilient datacenter pair, thereby ensuring that mailbox data is protected from software, hardware and even datacenter failures. Each database has four copies, with two copies in each datacenter, which means at a minimum, the PA requires four servers. Out of these four copies, three of them are configured as highly available. The fourth copy (the copy with the highest Activation Preference) is configured as a lagged database copy. Due to the server design, each copy of a database is isolated from its other copies, thereby reducing failure domains and increasing the overall availability of the solution as discussed in DAG: Beyond the “A”. The purpose of the lagged database copy is to provide a recovery mechanism for the rare event of system-wide, catastrophic logical corruption. It is not intended for individual mailbox recovery or mailbox item recovery. The lagged database copy is configured with a seven day ReplayLagTime. In addition, the Replay Lag Manager is also enabled to provide dynamic log file play down for lagged copies. This feature ensures that the lagged database copy can be automatically played down and made highly available in the following scenarios: When a low disk space threshold is reached When the lagged copy has physical corruption and needs to be page patched When there are fewer than three available healthy copies (active or passive) for more than 24 hours By using the lagged database copy in this manner, it is important to understand that the lagged database copy is not a guaranteed point-in-time backup. The lagged database copy will have an availability threshold, typically around 90%, due to periods where the disk containing a lagged copy is lost due to disk failure, the lagged copy becoming an HA copy (due to automatic play down), as well as, the periods where the lagged database copy is re-building the replay queue. To protect against accidental (or malicious) item deletion, Single Item Recovery or In-Place Hold technologies are used, and the Deleted Item Retention window is set to a value that meets or exceeds any defined item-level recovery SLA. With all of these technologies in play, traditional backups are unnecessary; as a result, the PA leverages Exchange Native Data Protection. Summary The PA takes advantage of the changes made in Exchange 2013 to simplify your Exchange deployment, without decreasing the availability or the resiliency of the deployment. And in some scenarios, when compared to previous generations, the PA increases availability and resiliency of your deployment. Ross Smith IV Principal Program Manager Office 365 Customer ExperienceExchange Server 2016 Architecture
Exchange Server 2016 builds upon the architecture introduced in Exchange Server 2013, with the continued focus goal of improving the architecture to serve the needs of deployments at all scales. Building Block Architecture In Exchange Server 2016, there is a single building block that provides the client access services and the high availability architecture necessary for any enterprise messaging environment. Figure 1: Building Block Architecture In our continuing quest to improve the product’s capabilities and simplify the architecture and its deployment, we have removed the Client Access server (CAS) role and added the client access services to the Mailbox role. Even without the CAS role, the system maintains loose coupling in terms of functionality, versioning, user partitioning and geographical affinity. The Mailbox server role now: Houses the logic to route protocol requests to the correct destination endpoint. Hosts all of the components and/or protocols that process, render and store the data. No clients connect directly to the back-end endpoints on the Mailbox server; instead, clients connect client access services and are routed (via local or remote proxy) to the Mailbox server that hosts the active database that contains the user’s mailbox. Mailbox servers can be added to a Database Availability Group (DAG), thereby forming a high availability unit that can be deployed in one or more datacenters. DAGs in Exchange Server 2016 do have a few specific enhancements: DatabaseAvailabilityGroupIpAddresses is no longer required when creating a DAG. By default, the failover cluster will be created without an administrative access point, as this is the recommended best practice. Replay Lag Manager is enabled by default. (released in CU1) Lagged database copy play down can be delayed based on disk latency, thereby ensuring active users are not impacted. (released in CU1) Database failovers times are reduced by 33% when compared to Exchange Server 2013. (released in CU3) Removal of the separate CAS role does not affect how communication occurs between servers. Communication between servers still occurs at the protocol layer, effectively ensuring that every server is an island. For a given mailbox’s connectivity, the protocol being used is always served by the protocol instance that is local to the active database copy. Figure 2: Inter-server communication in Exchange 2016 The load balancer configuration is also not affected by this architectural change. From a protocol perspective, the following will happen: A client resolves the namespace to a load balanced virtual IP address. The load balancer assigns the session to a Mailbox server in the load balanced pool. The Mailbox server authenticates the request and performs a service discovery by accessing Active Directory to retrieve the following information: Mailbox version (for this discussion, we will assume an Exchange 2016 mailbox) Mailbox location information (e.g., database information, ExternalURL values, etc.) The Mailbox server makes the decision to proxy the request or redirect the request to another Mailbox server in the infrastructure (within the same forest). The Mailbox server queries an Active Manager instance that is responsible for the database to determine which Mailbox server is hosting the active copy. The Mailbox server proxies the request to the Mailbox server hosting the active copy. The protocol used in step 6 depends on the protocol used to connect to client access services. If the client request uses HTTP, then the protocol used between the servers is HTTP (secured via SSL using a self-signed certificate). If the protocol used by the client is IMAP or POP, then the protocol used between the servers is IMAP or POP. Telephony requests are unique. Instead of proxying the request at step 6, the Mailbox server will redirect the request to the Mailbox server hosting the active copy of the user’s database, as the telephony devices support redirection and need to establish their SIP and RTP sessions directly with the Unified Messaging services on the Mailbox server. Figure 3: Client Protocol Connectivity Now many of you may be thinking, wait how does authentication work? Well for HTTP, POP, or IMAP requests that use basic, NTLM, or Kerberos authentication, the authentication request is passed as part of the HTTP payload, so the Client Access services within each MBX server will authenticate the request naturally. Forms-based authentication (FBA) is different. FBA was one of the reasons why session affinity was required for OWA in previous releases of Exchange – the reason being that that the cookie used a per server key for encryption; so if another MBX received a request, it could not decrypt the session. With Exchange 2013, we stopped leveraging a per server session key and instead leveraged the private key from the certificate that was installed on the Client Access server. With Exchange 2016, we now leverage the organization certificate that is configured for the organization’s authorization configuration. This means that any Exchange 2016 server can decrypt the cookie. And yes, the Edge Transport server role will ship in Exchange Server 2016 (and at RTM, to boot!). All the capabilities and features you had in the Edge Transport server role in Exchange Server 2013, remain in Exchange Server 2016. Why did we remove the Client Access server role? The Exchange Server 2016 architecture evolves the building block architecture that has been refined over the course of the last several releases. With this architecture, all servers in the Exchange environment (excluding Edge Transport) are exactly the same—the same hardware, the same configuration, and so forth. This uniformity simplifies ordering the hardware, as well as performing maintenance and management of the servers. As with Exchange 2010 and in Exchange 2013, we continue to recommend role co-location as a best practice. From a cost perspective, the overall goal is to ensure that the architecture is balanced for CPU and disk. Having separate server roles can result in long-term cost disadvantages as you may purchase more CPU, disk, and memory resources than you will actually use. For example, consider a server that hosts only the Client Access server role. Many servers enable you to add a given number of disks in a very economical fashion—when you are deploying and using that number of disks, the cost is essentially zero. But if you deploy a server role that uses far less than the given number of disks, you’re paying for a disk controller that is either under-used or not used at all. This architecture is designed to enable you to have fewer physical Exchange servers in your environment. Fewer physical servers mean lower costs for a variety of reasons: Operational costs are almost always higher than the capital costs. It costs more to manage a server over its lifetime than it does to purchase it. You purchase fewer Exchange server licenses. This architecture only requires a license for one Exchange server and one operating system, while breaking out the roles required multiple Exchange server licenses and multiple operating system licenses. Deploying fewer servers has a trickle-down effect across the rest of the infrastructure. For example, deploying fewer physical servers may reduce the total rack and floor space required for the Exchange infrastructure, which in turn reduces power and cooling costs. This architecture ultimately distributes the load across a greater number of servers than deploying single-role servers because all Mailbox servers also handle client access because: You’re distributing the load across a greater number of physical machines, which increases scalability. During a failure event, the load on the remaining servers only increases incrementally, which ensures the other functions the server is performing aren’t adversely affected. The solution can survive a greater number of Client Access role (or service) failures and still provide service, which increases resiliency. Key Architectural Improvements Exchange Server 2016 also includes a number of architectural improvements, beyond the server role consolidation, including search enhancements, document collaboration improvements, and more. Search Improvements (Released in CU3): One of the challenging areas for on-premises environment was the amount of data that was replicated with each database copy in previous releases. In Exchange Server 2016, we have reduced bandwidth requirements between the active copy and a passive HA copy; to determine the bandwidth savings for your deployment configuration, you can leverage the Exchange Server Role Requirements Calculator. This was accomplished by enabling the local search instance to read data from its local database copy. As a result of this change, passive HA search instances no longer need to coordinate with their active counterparts in order to perform index updates. Another area of investment in search has been around decreasing the length of time to return search results, especially in online mode clients like OWA. This is accomplished by performing multiple asynchronous disk reads prior to the user completing the search term, which populates the cache with the relevant information, providing sub-second search query latency for online mode clients. Document Collaboration In previous releases of Exchange, OWA included document preview for Office and PDF documents, reducing the need to have a full fidelity client. SharePoint 2013 had a similar feature, however it used the Office Web Apps Server 2013 (now rebranded as Office Online Server) to accomplish this capability. Within Office 365, we also leverage Office Online Server to provide this capability, ensuring uniform document preview and editing capability across the suite. In Exchange Server 2016, we leverage Office Online Server to provide the rich document preview and editing capabilities for OWA. While this was a necessary change to ensure a homogenous experience across the Office Server suite, this does introduce additional complexity for environments that don’t have Office Online Server. The next generation of Office Online Server will not be supported for co-location with Exchange. Therefore, you must deploy a separate server farm infrastructure. This infrastructure will require unique namespaces, and will require session affinity to be maintained at the load balancer. While Exchange supports an unbound namespace model, the Office Online Server will require a bound namespace for each site resilient datacenter pair. However, unlike the bound namespace model within Exchange, Office Online Server will not require any namespace changes during a datacenter activation. Figure 4: Office Online Server Connectivity Extensibility (scheduled for a future CU): Office 365 introduced the REST APIs (Mail, Calendar, and Contact APIs), and now these APIs are available in Exchange Server 2016. The REST APIs simplify programming against Exchange by providing a familiar syntax that is designed with openness (e.g., open standards support JSON, OAUTH, ODATA) and flexibility (e.g., granular, tightly scoped permission to access user data). These APIs allow developers to connect from any platform, whether it be web, PC, or mobile. SDKs exist for.NET, iOS, Android, NodeJS, Ruby, Python, Cordova, and CORS for use in single page JavaScript web apps. What about Exchange Web Services (EWS)? All existing applications that leverage EWS will continue to work with Exchange Server 2016. We are, however, focusing new platform investments on the REST APIs and the apps for Office extensibility model. We expect to make significantly fewer investments in EWS so that we can focus our resources on investing in a single modern API that will, over time, enable most of the scenarios that our partners currently use EWS. Outlook Connectivity Introduced in Exchange Server 2013 Service Pack 1, MAPI/HTTP is the new standard in connectivity for Outlook. In Exchange Server 2016, MAPI/HTTP is enabled by default. In addition, Exchange Server 2016 introduces per-user control over this connectivity model, as well as, the ability to control whether the protocol (and Outlook Anywhere) is advertised to external clients. Note: Exchange Server 2016 does not support connectivity via the MAPI/CDO library. Third-party products (and custom in-house developed solutions) need to move to Exchange Web Services (EWS) or the REST APIs to access Exchange data. Coexistence with Exchange Server 2013 In Exchange Server 2013, the Client Access server role is simply an intelligent proxy that performs no processing/rendering of the content. That architectural tenet paid off in terms of forward coexistence. When you introduce Exchange Server 2016, you do not need to move the namespace. That’s right, the Exchange Server 2013 Client Access infrastructure can proxy the mailbox requests to the Exchange 2016 servers hosting the active database copy! For the first time ever, you get to decide when you move the namespace over to the new version. And not only that, you can even have load balancer pools contain a mix of Exchange Server 2013 and Exchange Server 2016. This means you can do a one-for-one swap in the load balancer pool – as you add Exchange 2016 servers, you can remove Exchange 2013 servers. Topology Requirements Exchange Server 2016 will only be supported on Windows Server 2012, Windows Server 2012 R2 and Windows Server 2016 operating systems. From an Active Directory perspective, Exchange Server 2016 will require: Windows Server 2008 or later Active Directory servers. Windows Server 2008 or higher Forest Functional Mode and Domain Functional Mode. Exchange Server 2016 will only support coexistence with Exchange Server 2010 SP3 RU11 and Exchange Server 2013 CU10. The Preferred Architecture During my session at Microsoft Ignite, I revealed Microsoft’s preferred architecture (PA) for Exchange Server 2016. The PA is the Exchange Engineering Team’s best practice recommendation for what we believe is the optimum deployment architecture for Exchange 2016, and one that is very similar to what we deploy in Office 365. While Exchange 2016 offers a wide variety of architectural choices for on-premises deployments, this architecture is our most scrutinized one ever. While there are other supported deployment architectures, they are not recommended. The Exchange 2016 PA is very similar to the Exchange 2013 PA. A symmetrical DAG is deployed across a datacenter pair with active database copies distributed across all physical servers in the DAG. Database copies are deployed on JBOD storage, with four copies per-disk. One of the copies is a lagged database copy. Clients connect to a unified namespace that is equally distributed across the datacenters in the site resilient pair. However, the Exchange 2016 PA differs in the following ways: Exchange’s unbound namespace model is load balanced across the datacenters in a layer 7 configuration that does not leverage session affinity. An Office Online Server farm is deployed in each datacenter, with each farm having a unique namespace (bound model). Session affinity is managed by the load balancer. The DAG is deployed without an administrative access point. The commodity dual-socket server hardware platform contains 20-24 cores and up to 96GB of memory, and a battery-backed write cache controller. All data volumes are formatted with ReFS (with the integrity feature disabled). For more information, please see the Exchange 2016 Preferred Architecture article. Summary Exchange Server 2016 continues in the investments introduced in previous versions of Exchange by reducing the server role architecture complexity, aligning with the Preferred Architecture and Office 365 design principles, and improving coexistence with Exchange Server 2013. These changes simplify your Exchange deployment, without decreasing the availability or the resiliency of the deployment. And in some scenarios, when compared to previous generations, the PA increases availability and resiliency of your deployment. Ross Smith IV Principal Program Manager Office 365 Customer Experience Updates 5/8/15: Added section on topology requirements. 7/8/15: Updated topology requirements. 10/1/15: Updated for RTM. 10/12/15: Added Preferred Architecture link. 3/15/16: Updated with deferred play down support in CU1. 9/21/16: Updated with read from passive support in CU3.Ask the Perf Guy: Sizing Exchange 2016 Deployments
Uh oh. You are probably thinking to yourself, here comes another one of those incredibly long blog posts about sizing. Thankfully, it’s not. If you want to fully understand the sizing process for Exchange 2016, you are certainly welcome to read the previous post that I did for Exchange 2013, as the overall process is effectively the same with one major caveat. Since we have eliminated the CAS role in Exchange 2016, you must follow the process for multi-role deployment sizing. Overall, the inputs to our sizing formulas stay the same from Exchange 2013 to Exchange 2016. This means that our IOPS requirements, memory requirements, and all of the other values provided in the Exchange 2013 sizing guidance should continue to be used for Exchange 2016. We are changing one set of inputs, however. Processor requirements We are slightly increasing the processor requirements for Exchange 2016 (compared to Exchange 2013) as this is a very new release, and we are still learning how it performs in production. This slight increase in CPU provides some additional headroom for unanticipated issues, and may be changed in the future as we learn more from our internal deployments as well as customer feedback. The same SPECint_rate2006 baseline value described in the Exchange 2013 guidance should continue to be used (33.75 per-core). Messages sent or received per mailbox per day Mcycles per User, Active DB Copy or Standalone Mcycles per User, Passive DB Copy 50 2.99 0.70 100 5.97 1.40 150 8.96 2.10 200 11.94 2.80 250 14.93 3.50 300 17.91 4.20 350 20.90 4.90 400 23.88 5.60 450 26.87 6.30 500 29.85 7.00 These changes are reflected in v7.8 and later in the calculator. System scalability The previously released guidance on maximum recommended cores and maximum memory size for Exchange 2013 is generally applicable to Exchange 2016 in terms of background and general scalability issues, however we have increased the recommended maximum memory size for currently supported versions of Exchange 2016 to 192GB. We recommend not exceeding the following sizing characteristics for Exchange 2016 servers. Recommended Maximum Processor Core Count 24 Recommended Maximum Memory 192 GB Note: Version 9.1 and later of the Exchange Server Role Requirements Calculator aligns with this guidance. Summary If you are at all familiar with the process for sizing the last couple of Exchange releases, you will be very comfortable with Exchange 2016. As with any new release, you should plan to roll-out in stages and monitor the health and performance of the solution carefully. We do expect that our performance and scalability guidance for Exchange 2016 will evolve over the lifespan of the product so watch the Exchange team blog for future updates. Jeff Mealiffe Principal PM Manager Office 365 Customer ExperienceDatabase Maintenance in Exchange 2010
Over the last several months there has been significant chatter around what is background database maintenance and why is it important for Exchange 2010 databases. Hopefully this article will answer these questions. What maintenance tasks need to be performed against the database? The following tasks need to be routinely performed against Exchange databases: Database Compaction Database Defragmentation Database Checksumming Page Patching Page Zeroing Database Compaction The primary purpose of database compaction is to free up unused space within the database file (however, it should be noted that this does not return that unused space to the file system). The intention is to free up pages in the database by compacting records onto the fewest number of pages possible, thus reducing the amount of I/O necessary. The ESE database engine does this by taking the database metadata, which is the information within the database that describes tables in the database, and for each table, visiting each page in the table, and attempting to move records onto logically ordered pages. Maintaining a lean database file footprint is important for several reasons, including the following: Reducing the time associated with backing up the database file Maintaining a predictable database file size, which is important for server/storage sizing purposes. Prior to Exchange 2010, database compaction operations were performed during the online maintenance window. This process produced random IO as it walked the database and re-ordered records across pages. This process was literally too good in previous versions – by freeing up database pages and re-ordering the records, the pages were always in a random order. Coupled with the store schema architecture, this meant that any request to pull a set of data (like downloading items within a folder) always resulted in random IO. In Exchange 2010, database compaction was redesigned such that contiguity is preferred over space compaction. In addition, database compaction was moved out of the online maintenance window and is now a background process that runs continuously. Database Defragmentation Database defragmentation is new to Exchange 2010 and is also referred to as OLD v2 and B+ tree defragmentation. Its function is to compact as well as defragment (make sequential) database tables that have been marked/hinted as sequential. Database defragmentation is important to maintain efficient utilization of disk resources over time (make the IO more sequential as opposed to random) as well as to maintain the compactness of tables marked as sequential. You can think of the database defragmentation process as a monitor that watches other database page operations to determine if there is work to do. It monitors all tables for free pages, and if a table gets to a threshold where a significant high percentage of the total B+ Tree page count is free, it gives the free pages back to the root. It also works to maintain contiguity within a table set with sequential space hints (a table created with a known sequential usage pattern). If database defragmentation sees a scan/pre-read on a sequential table and the records are not stored on sequential pages within the table, the process will defrag that section of the table, by moving all of the impacted pages to a new extent in the B+ tree. You can use the performance counters (mentioned in the monitoring section) to see how little work database defragmentation performs once a steady state is reached. Database defragmentation is a background process that analyzes the database continuously as operations are performed, and then triggers asynchronous work when necessary. Database defragmentation is throttled under two scenarios: The max number of outstanding tasks This keeps database defragmentation from doing too much work the first pass if massive change has occurred in the database. A latency throttle of 100ms When the system is overloaded, database defragmentation will start punting defragmentation work. Punted work will get executed the next time the database goes through that same operational pattern. There's nothing that remembers what defragmentation work was punted and goes back and executes it once the system has more resources. Database Checksumming Database checksumming (also known as Online Database Scanning) is the process where the database is read in large chunks and each page is checksummed (checked for physical page corruption). Checksumming’s primary purpose is to detect physical corruption and lost flushes that may not be getting detected by transactional operations (stale pages). With Exchange 2007 RTM and all previous versions, checksumming operations happened during the backup process. This posed a problem for replicated databases, as the only copy to be checksummed was the copy being backed up. For the scenario where the passive copy was being backed up, this meant that the active copy was not being checksummed. So in Exchange 2007 SP1, we introduced a new optional online maintenance task, Online Maintenance Checksum (for more information, see Exchange 2007 SP1 ESE Changes – Part 2). In Exchange 2010, database scanning checksums the database and performs post Exchange 2010 Store crash operations. Space can be leaked due to crashes, and online database scanning finds and recovers lost space. Database checksum reads approximately 5 MB per second for each actively scanning database (both active and passive copies) using 256KB IOs. The I/O is 100 percent sequential. The system in Exchange 2010 is designed with the expectation that every database is fully scanned once every seven days. If the scan takes longer than seven days, an event is recorded in the Application Log : Event ID: 733 Event Type: Information Event Source: ESE Description: Information Store (15964) MDB01: Online Maintenance Database Checksumming background task is NOT finishing on time for database 'd:\mdb\mdb01.edb'. This pass started on 11/10/2011 and has been running for 604800 seconds (over 7 days) so far. If it takes longer than seven days to complete the scan on the active database copy, the following entry will be recorded in the Application Log once the scan has completed: Event ID: 735 Event Type: Information Event Source: ESE Description: Information Store (15964) MDB01 Database Maintenance has completed a full pass on database 'd:\mdb\mdb01.edb'. This pass started on 11/10/2011 and ran for a total of 777600 seconds. This database maintenance task exceeded the 7 day maintenance completion threshold. One or more of the following actions should be taken: increase the IO performance/throughput of the volume hosting the database, reduce the database size, and/or reduce non-database maintenance IO. In addition, an in-flight warning will also be recorded in the Application Log when it takes longer than 7 days to complete. In Exchange 2010, there are now two modes to run database checksumming on active database copies: Run in the background 24×7 This is the default behavior. It should be used for all databases, especially for databases that are larger than 1TB. Exchange scans the database no more than once per day. This read I/O is 100 percent sequential (which makes it easy on the disk) and equates to a scanning rate of about 5 megabytes (MB)/sec on most systems. The scanning process is single threaded and is throttled by IO latency. The higher the latency, the more database checksum slows down because it is waiting longer for the last batch to complete before issuing another batch scan of pages (8 pages are read at a time). Run in the scheduled mailbox database maintenance process When you select this option, database checksumming is the last task. You can configure how long it runs by changing the mailbox database maintenance schedule. This option should only be used with databases smaller than 1 terabyte (TB) in size, which require less time to complete a full scan. Regardless of the database size, our recommendation is to leverage the default behavior and not configure database checksum operations against the active database as a scheduled process (i.e., don’t configure it as a process within the online maintenance window). For passive database copies, database checksums occur during runtime, continuously operating in the background. Page Patching Page patching is the process where corrupt pages are replaced by healthy copies. As mentioned previously, corrupt page detection is a function of database checksumming (in addition, corrupt pages are also detected at run time when the page is stored in the database cache). Page patching works against highly available (HA) database copies. How a corrupt page is repaired depends on whether the HA database copy is active or passive. Page patching process On active database copies On passive database copies A corrupt page(s) is detected. A marker is written into the active log file. This marker indicates the corrupt page number and that page requires replacement. An entry is added to the page patch request list. The active log file is closed. The Replication service ships the log file to passive database copies. The Replication service on a target Mailbox server receives the shipped log file and inspects it. The Information Store on the target server replays the log file and replays up to marker, retrieves its healthy version of the page, invokes Replay Service callback and ships the page to the source Mailbox server. The source Mailbox server receives the healthy version of the page, confirms that there is an entry in the page patch request list, then writes the page to the log buffer, and correspondingly, the page is inserted into the database cache. The corresponding entry in the page patch request list is removed. At this point the database is considered patched (at some later point the checkpoint will advance and the database cache will be flushed and the corrupt page on disk will be overwritten). Any other copy of this page (received from another passive copy) will be silently dropped, because there is no corresponding entry in the page patch request list. On the Mailbox server where the corrupt page(s) is detected, log replay is paused for the affected database copy. The replication service coordinates with the Mailbox server that is hosting the active database copy and retrieves the corrupted page(s) and the required log range from the active copy’s database header. The Mailbox server updates the database header for the affected database copy, inserting the new required log range. The Mailbox server notifies the Mailbox server hosting the active database copy which log files it requires. The Mailbox server receives the required log files and inspects them. The Mailbox server injects the healthy versions of the database pages it retrieved from the active database copy. The pages are written to the log buffer, and correspondingly, the page is inserted into the database cache. The Mailbox server resumes log replay. Page Zeroing Database Page Zeroing is the process where deleted pages in the database are written over with a pattern (zeroed) as a security measure, which makes discovering the data much more difficult. With Exchange 2007 RTM and all previous versions, page zeroing operations happened during the streaming backup process. In addition since they occurred during the streaming backup process they were not a logged operation (e.g., page zeroing did not result in the generation of log files). This posed a problem for replicated databases, as the passive copies never had its pages zeroed, and the active copies would only have it pages zeroed if you performed a streaming backup. So in Exchange 2007 SP1, we introduced a new optional online maintenance task, Zero Database Pages during Checksum (for more information, see Exchange 2007 SP1 ESE Changes – Part 2). When enabled this task would zero out pages during the Online Maintenance Window, logging the changes, which would be replicated to the passive copies. With the Exchange 2007 SP1 implementation, there is significant lag between when a page is deleted to when it is zeroed as a result of the zeroing process occurring during a scheduled maintenance window. So in Exchange 2010 SP1, the page zeroing task is now a runtime event that operates continuously, zeroing out pages typically at transaction time when a hard delete occurs. In addition, database pages can also be scrubbed during the online checksum process. The pages targeted in this case are: Deleted records which couldn’t be scrubbed during runtime due to dropped tasks (if the system is too overloaded) or because Store crashed before the tasks got to scrub the data; Deleted tables and secondary indices. When these get deleted, we don’t actively scrub their contents, so online checksum detects that these pages don’t belong to any valid object anymore and scrubs them. For more information on page zeroing in Exchange 2010, see Understanding Exchange 2010 Page Zeroing. Why aren’t these tasks simply performed during a scheduled maintenance window? Requiring a scheduled maintenance window for page zeroing, database defragmentation, database compaction, and online checksum operations poses significant problems, including the following: Having scheduled maintenance operations makes it very difficult to manage 24x7 datacenters which host mailboxes from various time zones and have little or no time for a scheduled maintenance window. Database compaction in prior versions of Exchange had no throttling mechanisms and since the IO is predominantly random, it can lead to poor user experience. Exchange 2010 Mailbox databases deployed on lower tier storage (e.g., 7.2K SATA/SAS) have a reduced effective IO bandwidth available to ESE to perform maintenance window tasks. This is an issue because it means that IO latencies will increase during the maintenance window, thus preventing the maintenance activities to complete within a desired period of time. The use of JBOD provides an additional challenge to the database in terms of data verification. With RAID storage, it's common for an array controller to background scan a given disk group, locating and re-assigning bad blocks. A bad block (aka sector) is a block on a disk that cannot be used due to permanent damage (e.g. physical damage inflicted on the disk particles). It's also common for an array controller to read the alternate mirrored disk if a bad block was detected on the initial read request. The array controller will subsequently mark the bad block as “bad” and write the data to a new block. All of this occurs without the application knowing, perhaps with just a slight increase in the disk read latency. Without RAID or an array controller, both of these bad block detection and remediation methods are no longer available. Without RAID, it's up to the application (ESE) to detect bad blocks and remediate (i.e., database checksumming). Larger databases on larger disks require longer maintenance periods to maintain database sequentiality/compactness. Due to the aforementioned issues, it was critical in Exchange 2010 that the database maintenance tasks be moved out of a scheduled process and be performed during runtime continuously in the background. Won’t these background tasks impact my end users? We’ve designed these background tasks such that they're automatically throttled based on activity occurring against the database. In addition, our sizing guidance around message profiles takes these maintenance tasks into account. How can I monitor the effectiveness of these background maintenance tasks? In previous versions of Exchange, events in the Application Log would be used to monitor things like online defragmentation. In Exchange 2010, there are no longer any events recorded for the defragmentation and compaction maintenance tasks. However, you can use performance counters to track the background maintenance tasks under the MSExchange Database ==> Instances object: Counter Description Database Maintenance Duration The number of seconds that have passed since the maintenance started for this database. If the value is 0, maintenance has been finished for the day. Database Maintenance Pages Bad Checksums The number of non-correctable page checksums encountered during a database maintenance pass Defragmentation Tasks The count of background database defragmentation tasks that are currently executing Defragmentation Tasks Completed/Sec The rate of background database defragmentation tasks that are being completed You'll find the following page zeroing counters under the MSExchange Database object: Counter Description Database Maintenance Pages Zeroed Indicates the number of pages zeroed by the database engine since the performance counter was invoked Database Maintenance Pages Zeroed/sec Indicates the rate at which pages are zeroed by the database engine How can I check whitepace in a database? You will need to dismount the database and use ESEUTIL /MS to check the available whitespace in a database. For an example, see http://technet.microsoft.com/en-us/library/aa996139(v=EXCHG.65).aspx (note that you have to multiply the number of pages by 32K). Note that there is a status property available on databases within Exchange 2010, but it should not be used to determine the amount of total whitespace available within the database: Get-MailboxDatabase MDB1 -Status | FL AvailableNewMailboxSpace AvailableNewMailboxSpace tells you is how much space is available in the root tree of the database. It does not factor in the free pages within mailbox tables, index tables, etc. It is not representative of the white space within the database. How can I reclaim the whitespace? Naturally, after seeing the available whitespace in the database, the question that always ensues is – how can I reclaim the whitespace? Many assume the answer is to perform an offline defragmentation of the database using ESEUTIL. However, that's not our recommendation. When you perform an offline defragmentation you create an entirely brand new database and the operations performed to create this new database are not logged in transaction logs. The new database also has a new database signature, which means that you invalidate the database copies associated with this database. In the event that you do encounter a database that has significant whitespace and you don't expect that normal operations will reclaim it, our recommendation is: Create a new database and associated database copies. Move all mailboxes to the new database. Delete the original database and its associated database copies. A terminology confusion Much of the confusion lies in the term background database maintenance. Collectively, all of the aforementioned tasks make up background database maintenance. However, the Shell, EMC , and JetStress all refer to database checksumming as background database maintenance, and that's what you're configuring when you enable or disable it using these tools. Figure 1: Enabling background database maintenance for a database using EMC Enabling background database maintenance using the Shell: Set-MailboxDatabase -Identity MDB1 -BackgroundDatabaseMaintenance $true Figure 2: Running background database maintenance as part of a JetStress test My storage vendor has recommended I disable Database Checksumming as a background maintenance task, what should I do? Database checksumming can become an IO tax burden if the storage is not designed correctly (even though it's sequential) as it performs 256K read IOs and generates roughly 5MB/s per database. As part of our storage guidance, we recommend you configure your storage array stripe size (the size of stripes written to each disk in an array; also referred to as block size) to be 256KB or larger. It's also important to test your storage with JetStress and ensure that the database checksum operation is included in the test pass. In the end, if a JetStress execution fails due to database checksumming, you have a few options: Don’t use striping Use RAID-1 pairs or JBOD (which may require architectural changes) and get the most benefit from sequential IO patterns available in Exchange 2010. Schedule it Configure database checksumming to not be a background process, but a scheduled process. When we implemented database checksum as a background process, we understood that some storage arrays would be so optimized for random IO (or had bandwidth limitations) that they wouldn't handle the sequential read IO well. That's why we built it so it could be turned off (which moves the checksum operation to the maintenance window). If you do this, we do recommend smaller database sizes. Also keep in mind that the passive copies will still perform database checksum as a background process, so you still need to account for this throughput in our storage architecture. For more information on this subject see Jetstress 2010 and Background Database Maintenance. Use different storage or improve the capabilities of the storage Choose storage which is capable of meeting Exchange best practices (256KB+ stripe size). Conclusion The architectural changes to the database engine in Exchange Server 2010 dramatically improve its performance and robustness, but change the behavior of database maintenance tasks from previous versions. Hopefully this article helps your understanding of what is background database maintenance in Exchange 2010. Ross Smith IV Principal Program Manager Exchange Customer ExperienceExchange 2013 Server Role Architecture
The Past: Exchange 2007 and Exchange 2010 In 2005, we set out developing Exchange 2007. During the preliminary planning of Exchange 2007, we realized that the server architecture model had to evolve to deal with hardware constraints, like CPU, that existed at the time and would exist for a good portion of the lifecycle of the product. The architectural change resulted in five server roles, three of which were core to the product: Edge Transport for routing and anti-malware from the edge of the organization Hub Transport for internal routing and policy enforcement Mailbox for storage of data Client Access for client connectivity and web services Unified Messaging for voice mail and voice access Figure 1: Server role architecture in Exchange 2007 & Exchange 2010 Our goal was to make these server roles autonomous units. Unfortunately, that goal did not materialize in either Exchange 2007 or Exchange 2010. The server roles were tightly coupled along four key dimensions: Functionality was scattered across all the server roles, thereby making it a requirement that Hub Transport and the Client Access server roles be deployed in every Active Directory site where Mailbox servers were deployed. This also necessitated a tight versioning alignment – a down-level Hub Transport or Client Access shouldn’t communicate with a higher version Mailbox server; in some cases this was enforced (e.g., Exchange 2007 Hub Transport servers cannot deliver messages to Exchange 2010 Mailbox servers), but in other cases they were not (e.g., an Exchange 2010 SP1 Client Access server can render data from an Exchange 2010 SP2 Mailbox server). The versioning restriction also meant that you could not simply upgrade a single server role to take advantage of new functionality – you had to upgrade all of them. The server roles were also coupled together from a geographical affinity perspective as well due to the middle tier server roles using RPC as the communication mechanism to the Mailbox server role. Similarly, from a user perspective, the server roles were also tightly coupled – a set of users that are served by a given Mailbox server, are always served by a given set of Client Access and Hub Transport servers. The functionality and versioning aspects are the key issues. To understand it better, let’s look at the following diagram: Figure 2: Inter-server communication across different layers of functionality As you can see from the above diagram, there are three layers: 1) Protocols 2) Business Logic and 3) Storage. If you are familiar with the OSI model, you might believe the protocol layer is similar to the application layer and that data has to flow from the application layer through the other layers to get to the physical layer (or vice versa). However, that is not the case in Exchange 2007 or Exchange 2010; a protocol can interact directly with the storage layer. For example, the transport instance on Server1 (Exchange 2010 SP1) can deliver mail to the Store service on Server2 (Exchange 2010 SP2). The reverse is also true, the store can submit mail via the Store Submission service on Server2 to the transport service running on Server1. In either scenario, content conversion happens on the lower version server (as content conversion in this example happens within the transport components). While a newer version interacting with an older version may not be problematic, the same cannot be said when the reverse is true as the older version simply does not know about any changes that exist in the newer version that may break functionality (hence, why we put in blockers in certain circumstances, or provide guidance on upgrade procedures). The end result is that in Exchange 2007 and Exchange 2010 deployments, the server roles are deployed as a single monolithic entity. The Present: Exchange Server 2013 When we started our planning for Exchange Server 2013, we focused on a single architectural tenet – improve the architecture to serve the needs of deployments at all scales, from the small 100 user shop to hosting hundreds of millions of mailboxes in Office 365. This single tenet drove us to make major architectural changes and investments across the entire product. This shift is in part, because unlike when we were designing Exchange 2007, we are no longer CPU bound; in fact there is so much readily available CPU on modern server hardware, that the notion of dedicated server roles no longer makes sense as hardware ultimately is wasted (hence the recommendation in Exchange 2010 to deploy multi-role servers). In Exchange Server 2013, we have two basic building blocks – the Client Access array and the Database Availability Group (DAG). Each provides a unit of high availability and fault tolerance that are decoupled from one another. Client Access servers make up the CAS array, while Mailbox servers comprise the DAG . Figure 3: New server role architecture simplifies deployments So what is the Client Access server in Exchange 2013? The Client Access server role is comprised of three components, client protocols, SMTP, and a UM Call Router. The CAS role is a thin, protocol session stateless server that is organized into a load balanced configuration. Unlike previous versions, session affinity is not required at the load balancer (but you still want a load balancer to handle connection management policies and health checking). This is because logic now exists in CAS to authenticate the request, and then route the request to the Mailbox server that hosts the active copy of the mailbox database. The Mailbox server role now hosts all the components and/or protocols that process, render and store the data. No clients will ever connect directly to the Mailbox server role; all client connections are handled by the Client Access server role. Mailbox servers can be added to a Database Availability Group, thereby forming a high available unit that can be deployed in one or more datacenters. Unlike the past two generations, these two server roles do not suffer from the same constraints: Functionality is not dispersed across both server roles. All data rendering now occurs on the Mailbox server that hosts the active mailbox database copy. As we will discuss in more detail in a future article, the Client Access server role is merely a proxy. Versioning is also decoupled as a result of moving the data rendering stack to a single server role. Client Access servers and the Mailbox servers no longer leverage the chatty RPC protocol for client sessions, thereby removing the geographical affinity issues. From a user perspective, users do not need to be serviced by the Client Access servers that are located within the same Active Directory site as the Mailbox servers hosting the user’s mailboxes. If we return to the layering diagram, we can see how this changes: Figure 4: Inter-server communication in Exchange 2013 Instead of allowing communication between servers to occur at any layer in the stack, communication must occur between servers at the protocol layer. This ensures that for a given mailbox’s connectivity, the protocol being used is always served by the protocol instance that is local to the active database copy. In other words, if my mailbox is located on Server1 and I want to send a message to a mailbox on Server2, the message must be sent from Server1’s transport components to the transport components on Server2; content conversion of the message then occurs on Server2 as the message is injected into the store. If I upgrade the Mailbox server with a service pack or cumulative update, then for a given mailbox hosted on that server, all data rendering and content conversions for that mailbox will be local, thus removing version constraints and functionality issues that arose in previous releases. Looking Ahead This article is the start of several articles focused on architecture and the investments we have made in Exchange Server 2013. Over the next several weeks: We will delve into more specifics with respect to the server roles. Discuss our investments with respect to storage. Discuss sizing Exchange Server 2013. Discuss load balancing and namespace planning. And for those of you that did not get a chance to attend MEC 2012, feel free to visit www.iammec.com/video and watch my technical architecture keynote. Conclusion Exchange Server 2013 introduces a new building block architecture that facilitate deployments at all scales. All core Exchange functionality for a given mailbox is always served by the Mailbox server where the mailbox’s database is currently activated. The introduction of the Client Access server role changes enables you to move away from complicated session affinity load balancing solutions, simplifying the network stack. From an upgrade perspective, all components on a given server are upgraded together, thereby virtually eliminating the need to juggle Client Access and Mailbox server versions. And finally, the new server role architecture aligns Exchange with being able to take advantage of hardware trends for the foreseeable future – core count will continue to increase, disk capacity will continue to increase, and RAM prices will continue to drop. Ross Smith IV Principal Program Manager Exchange Customer Experience
