Outlook Connectivity with MAPI over HTTP
Among the many new features delivered in Exchange 2013 SP1 is a new method of connectivity to Outlook we refer to as MAPI over HTTP (or MAPI/HTTP for short). We’ve seen a lot of interest about this new connection method and today we’ll give you a full explanation of what it is, what it provides, where it will take us in the future, and finally some tips of how and where to get started enabling this for your users. What is MAPI over HTTP? MAPI over HTTP is a new transport used to connect Outlook and Exchange. MAPI/HTTP was first delivered with Exchange 2013 SP1 and Outlook 2013 SP1 and begins gradually rolling out in Office 365 in May. It is the long term replacement for RPC over HTTP connectivity (commonly referred to as Outlook Anywhere). MAPI/HTTP removes the complexity of Outlook Anywhere’s dependency on the legacy RPC technology. Let’s compare the architectures. MAPI/HTTP moves connectivity to a true HTTP request/response pattern and no longer requires two long-lived TCP connections to be open for each session between Outlook and Exchange. Gone are the twin RPC_DATA_IN and RPC_DATA_OUT connections required in the past for each RPC/HTTP session. This change will reduce the number of concurrent TCP connections established between the client and server. MAPI/HTTP will generate a maximum of 2 current connections generating one long lived connection and an additional on-demand short-lived connection. Outlook Anywhere also essentially double wrapped all of the communications with Exchange adding to the complexity. MAPI/HTTP removes the RPC encapsulation within HTTP packets sent across the network making MAPI/HTTP a more well understood and predictable HTTP payload. An additional network level change is that MAPI/HTTP decouples the client/server session from the underlying network connection. With Outlook Anywhere connectivity, if a network connection was lost between client and server, the session was invalidated and had to be reestablished all over again, which is a time-consuming and expensive operation. In MAPI/HTTP when a network connection is lost the session itself is not reset for 15 minutes and the client can simply reconnect and continue where it left off before the network level interruption took place. This is extremely helpful for users who might be connecting from low quality networks. Additionally in the past, an unexpected server-side network blip would result in all client sessions being invalidated and a surge of reconnections being made to a mailbox server. Depending on the number of Outlook clients reconnecting, the re-establishing of so many RPC/HTTP connections might strain the resources of the mailbox server, and possibly extend the outage in scope (to Outlook clients connected to multiple servers) and time, caused by a single server-side network blip. Why MAPI over HTTP? You are probably asking yourself why the Exchange team would create a complete replacement for something so well-known and used. Let us explain. The original Outlook Anywhere architecture wasn’t designed for today’s reality of clients connecting from a wide variety of network types – many of these are not as fast or reliable as what was originally expected when Outlook Anywhere was designed. Consider connections from cellular networks, home networks, or in-flight wireless networks as a few examples. The team determined the best way to meet current connection needs and also put Exchange in the position to innovate more quickly was to start with a new simplified architecture. The primary goal of MAPI/HTTP is provide a better user experience across all types of connections by providing faster connection times to Exchange – yes, getting email to users faster. Additionally MAPI/HTTP will improve the connection resiliency when the network drops packets in transit. Let’s quantify a few of these improvements your users can expect. These results represent what we have seen in our own internal Microsoft user testing. When starting Outlook users often see the message “Connecting to Outlook” in the Outlook Status bar. MAPI/HTTP can reduce the amount of time a user waits for this connection. In the scenario when a user first launches Outlook the time to start synchronization improved to 30 seconds vs. 90 seconds for Outlook Anywhere for 70% of the monitored clients. Improvements are also delivered when clients are resuming from hibernation or simply re-connecting to a new network. Testing showed that 80% of the clients using MAPI/HTTP started syncing in less than 30 seconds vs. over 40 seconds for Outlook Anywhere clients when resuming from hibernation. This improvement was made possible as MAPI/HTTP implements a pause/resume feature enabling clients to resume using an existing connection rather than negotiating a new connection each time. Current sessions for MAPI/HTTP are valid for 15 minutes, but as we fine tune and expand this duration, these improvements will be even more noticeable. Improvements aren’t limited to end users. IT administrators will gain greater protocol visibility allowing them to identify and remediate situations faster and with more confidence. Due to MAPI/HTTP moving to a more traditional HTTP protocol payload, the ability to utilize already known tools common to HTTP debugging is a reality. IIS and HttpProxy logs will now contain information similar to other HTTP based protocols like Outlook Web App and be able to pass information via headers. At times in the past, certain debug procedures for RPC/HTTP were only available via proprietary internal Microsoft tools. This move should put all customers on the same level playing field as far as what tools are available for debug purposes. Exchange administrators also will find response returned by Autodiscover for MAPI/HTTP to Outlook is greatly simplified. The settings returned are just protocol version and endpoint URLs for Outlook to connect to Exchange mailbox and directory fr om inside or outside the customer’s corporate network. Outlook treats the URLs returned as opaque and uses as-is, minimizing the risk of connectivity breaking with future endpoint changes. Since MAPI/HTTP, like any other web protocol, simply sends an anonymous HTTP request to Exchange and gets back the authentication settings, there is no need for Autodiscover to advertise the authentication settings. This makes it easier to roll out changes in authentication settings for Outlook. The future MAPI/HTTP puts the Exchange team in position to innovate more quickly. It simplifies the architecture removing dependency on the RPC technologies which are no longer evolving as quickly as the customers demand. It provides the path for extensibility of the connection capabilities. A new capability that is on the roadmap for Outlook is to enable multi-factor authentication for users in Office 365. This capability is made possible with the use of MAPI/HTTP and is targeted to be delivered later this year. For a deeper look at this upcoming feature you can review the recent Multi-Factor Authentication for Office 365 blog post. This won’t stop with Office 365 MFA, but provides the extensibility foundation for 3 rd party identity providers. How does MAPI/HTTP work? Let’s walk through the scenario of an Outlook 2013 SP1 client connecting to Exchange Server 2013 SP1 after MAPI/HTTP has been enabled. The Outlook client begins with an Autodiscover POST request. In this request Outlook includes a new attribute that advertises the client is MAPI/HTTP capable with the attribute X-MapiHTTPCapability = 1. The Exchange server sees the request is coming from a MAPI/HTTP capable client and responds with the MAPI/HTTP information including the settings on how to connect to the mailbox using MAPI/HTTP. This assumes the MAPI/HTTP has been configured and enabled on the server. The Outlook client detects the new connection path and prompts the user to restart Outlook to switch to use the new connection. While the restart is pending Outlook will continue using Outlook Anywhere. We recommend you deploy the latest Office client updates to provide the best user experience. The updates remove the prompt and clients are allowed to make the transition at the next unprompted restart of Outlook. After the restart, Outlook now uses MAPI/HTTP to communicate with Exchange. What’s required? So now we have a clear set of advantages you can offer users, let’s review the requirements to enable MAPI/HTTP. Server Requirements: Use of MAPI/HTTP requires all Exchange 2013 Client Access Servers to be updated to Exchange Server 2013 SP1 (or later). The feature is disabled by default in SP1 so you can get the servers updated without anyone noticing any changes. If you are an Office 365 Exchange Online customer you won’t have anything to worry about on the service side of deployment. Client Requirements: Outlook clients must be updated to use MAPI/HTTP. Office 2013 SP1 or Office 365 ProPlus February update (SP1 equivalent for ProPlus) are required for MAPI/HTTP. It is recommend you deploy the May Office 2013 public update or the April update for Office 365 ProPlus to eliminate the restart prompt when MAPI/HTTP is enabled for users. Outlook 2010 was updated to support MAPI/HTTP in the January 2015 Public Update, and additional fixes for it were released in the April 2015 Public Update. Prior version clients will continue to work as-is using Outlook Anywhere. Outlook Anywhere is the supported connection method for those clients. We will not be backporting MAPI/HTTP to Outlook 2007 or earlier versions. How to get ready Part one of getting ready is to get the required updates to your servers and clients as described in the prior section. Part two of getting ready is evaluating potential impacts MAPI/HTTP might have on your on-premises servers. Again if you an Office 365 customer you can ignore this bit. When you implement MAPI/HTTP in your organization, it will have an impact on your Exchange server resources. Before you go any further you need to review the impacts to your server resources. The Exchange 2013 Server Role Requirements Calculatorhas been updated to factor in use of MAPI/HTTP. You need to use the most recent version of the calculator (v6.3 or later) before you proceed. MAPI/HTTP increases the CPU load in the Exchange Client Access Servers. This is a 50% increase over Exchange 2013 RTM, however it is still lower than Exchange 2010 requirements. As you plan be mindful that deploying in a multi-role configuration will minimize the impact to your sizing. Again use the calculator to review potential impacts this may have in your environment. This higher CPU use is due to the higher request rate with several short-lived connections, with each request taking care of authentication and proxying. To provide the best MAPI/HTTP performance you need to install .NET 4.5.1 on your Exchange 2013 servers. Installing .NET 4.5.1 will avoid long wait times for users thanks to a fix to ensure the notification channel remains asynchronous to avoid queued requests. The change in communication between Exchange and Outlook has a small impact on the bytes sent over the wire. The header content in MAPI/HTTP is responsible for an increase in bytes transferred. In a typical message communications we have observed an average packet size increase of 1.2% versus Outlook Anywhere for a 50 KB average packet. In scenarios of data transfers over 10 MB the increase in bytes over the wire is 5-10%. These increases assumes an ideal network where connections are not dropped or resumed. If you consider real world conditions you may actually find MAPI/HTTP data on the wire may be lower than Outlook Anywhere. Outlook Anywhere lacks the ability to resume connections and the cost of re-syncing items can quickly outweigh the increase from the MAPI/HTTP header information. Now deploy MAPI/HTTP Now that you have prepared your servers with SP1, updated your clients, and reviewed potential sizing impacts you are ready to get on with implementing MAPI/HTTP. It is disabled by default in SP1 and you must take explicit actions to configure and enable it. These steps are well covered in the MAPI over HTTPTechNet article. A few important things to remember in your deployment. Namespace: MAPI/HTTP is a new endpoint on CAS and can utilize both an internal namespace and an external namespace. For more information on how to properly plan your namespace design, see Namespace Planning in Exchange 2013 and Load Balancing in Exchange 2013. Certificates: The certificate used in Exchange will need to include both the internal and external MAPI/HTTP virtual directories to avoid any user certificate prompts, thus consider if the names exist on your certificates. Refer to the certificate planning post for additional help planning. MAPI/HTTP Configuration: Enabling MAPI/HTTP is an organizational configuration in Exchange, you won’t have the ability configure this for a subset of servers. If you require more specific control you can control the client behavior with a registry key. NOTE: If you require more specific control you can control the client behavior with a registry key on each client machine. This is not recommended or required but included if your situation demands this level of control. This registry entry prevents Outlook from sending the MAPI/HTTP capable flag to Exchange in the Autodiscover request. When you change the registry key on a client it will not take effect until the next time the client performs an Autodiscover query against Exchange. To disallow MAPI/HTTP and force RPC/HTTP to be used. HKEY_CURRENT_USER\Software\Microsoft\Exchange] “MapiHttpDisabled”=dword:00000001 To allow MAPI/HTTP simply delete the MapiHttpDisabled DWORD, or set it to a value of 0 as below. HKEY_CURRENT_USER\Software\Microsoft\Exchange] “MapiHttpDisabled”=dword:00000000 Connectivity: An important final consideration is to verify load balancers, reverse proxies, and firewalls are configured to allow access to the MAPI/HTTP virtual directories. At this time ForeFront Unified Access Gateway (UAG) 2010 SP4 is not compatible with MAPI/HTTP. The UAG team has committed to deliver support for MAPI/HTTP in a future update. How do I know it is working? There are a few quick ways to verify your configuration is working as expected. 1. Test with the Test-OutlookConnectivity cmdlet Use this command to test MAPI/HTTP connectivity: Test-OutlookConnectivity -RunFromServerId Contoso -ProbeIdentity OutlookMapiHttpSelfTestProbe This test is detailed in the MAPI over HTTPTechNet Article. 2. Inspect MAPI/HTTP server logs Administrators can review the following MAPI/HTTP log files to validate how the configuration is operating: Location Path CAS: %ExchangeInstallPath%Logging\HttpProxy\Mapi\HTTP Mailbox: %ExchangeInstallPath%Logging\MAPI Client Access\ Mailbox: %ExchangeInstallPath%Logging\MAPI Address Book Service\ 3. Check Outlook connection status on clients You can also quickly verify that the client is connected using MAPI/HTTP. The Outlook Connection status dialog can be launch by CTRL-right clicking the Outlook icon in the notification area and selecting Connection Status. Here are the few key fields to quickly confirm the connection is using MAPI/HTTP. Field Value Protocol HTTP (v/s RPC/HTTP for Outlook Anywhere) Proxy Server Empty Server name Actual server name (v/s GUID for Outlook Anywhere connections) Summary MAPI/HTTP provides a simplified transport and resulting architecture for Outlook to connect with Exchange. It enables improved user experiences to allow them faster access to mail and improves the resilience of their Outlook connections. These investments are the foundation for future capabilities such as multi-factor authentication in Outlook. It also helps IT support and troubleshoot client connection issues using standard HTTP protocol tools. As with all things new you must properly plan your implementation. Use the deployment guidanceavailable on TechNet and the updated sizing recommendations in the calculator before you start your deployment. With proper use it will guide you to a smooth deployment of MAPI/HTTP. Special thanks to Brian Day and Abdel Bahgat for extensive contributions to this blog post. Brian Shiers | Technical Product Manager MAPI/HTTP FAQ We collected a number of questions which frequently came up during the development, internal dogfooding, and customer TAPtesting of MAPI/HTTP. We hope these answer most of the questions you may have about MAPI/HTTP. Can MAPI/HTTP be enabled/disabled on a per-server basis? No, it is an organization-wide Exchange setting. The user experience mentioned during database failovers when one server is not yet MAPI/HTTP capable made the functionality to turn MAPI/HTTP on and off per server not a viable solution. Can MAPI/HTTP be enabled/disabled on a per-mailbox basis? No, there is not currently a Set-CasMailbox parameter to enable/disable MAPI/HTTP for a single mailbox. I updated the registry key to disable MAPI/HTTP on a client but the connection didn’t change? The registry entry simply controls what Outlook sends to Exchange about its MAPI/HTTP capability during an Autodiscover request. It does not immediately change the connection method Outlook is using nor will it change it with a simple restart of Outlook. Remember, the Autodiscover response Outlook gets only has MAPI/HTTP or RPC/HTTP settings in it so it has no way to immediately switch types. You must allow Outlook to perform its next Autodiscover request and get a response from Exchange after setting this registry entry before the change will take place. If you must attempt to speed along this process, there are two options. Set the registry entry as you wish, close Outlook, delete the Outlook profile, and then restart Outlook and go through the profile wizard. This should result in an immediate switch, but any settings stored in the Outlook profile are lost. Set the registry entry as you wish, close Outlook, delete the hidden Autodiscover response XML files in %LOCALAPPDATA%\Microsoft\Outlook, and restart Outlook. Restart Outlook once more to complete the switch. What happens if a user’s mailbox database is mounted on a MAPI/HTTP capable server and then a database failover happens to a non-MAPI/HTTP capable server? E.g. When not all mailbox servers in a DAG are MAPI/HTTP capable and MAPI/HTTP has already been enabled in the organization, and then a mailbox failover takes place between the SP1 (or later) and pre-SP1 servers. Additionally this could happen if you move a mailbox from a MAPI/HTTP capable mailbox server to a server that is not MAPI/HTTP capable. In the above example Outlook would fail to connect and when Autodiscover next ran the user would get an Outlook restart notification warning because MAPI/HTTP is no longer a viable connection method due to the mailbox being mounted on a pre-SP1 server. After the client restart the client profile would be back to utilizing RPC/HTTP. Note: While a mix of MAPI/HTTP capable and non-capable mailbox Exchange servers in the same DAG are supported in an environment with MAPI/HTTP enabled, it is very strongly not recommended due to the possible user experience outlined above. It is suggested the entire organization be upgraded to SP1 or later before enabling MAPI/HTTP in the organization. What if a user accesses additional mailboxes where MAPI/HTTP is not yet available? Outlook profiles can continue access additional resources using non-MAPI/HTTP connectivity methods even if the user’s primary mailbox utilizes MAPI/HTTP. For example a user can continue to access Legacy Public Folders or Shared Mailboxes on other Exchange servers not utilizing MAPI/HTTP. During the Autodiscover process Exchange will determine and hand back to Outlook the proper connectivity method necessary for each resource being accessed. If MAPI/HTTP becomes unreachable will a client fallback to RPC/HTTP? No, a user’s profile will never attempt to use RPC/HTTP if MAPI/HTTP becomes unavailable because the original Autodiscover response only contained one connection method to use. There is no fallback from MAPI/HTTP to RPC/HTTP or vice versa. Normal high availability design considerations should ensure the MAPI/HTTP endpoint remain accessible in the event of server or service failures. Is MAPI/HTTP replacing Exchange Web Services (EWS)? No, MAPI/HTTP is not a replacement for EWS and there are no plans to move current EWS clients to MAPI/HTTP. Is RPC/HTTP being deprecated as an available connection method? Over time this may take place as non-MAPI/HTTP capable Outlook versions age out of their product support lifecycle, but there are no immediate plans to remove RPC/HTTP as a valid connection method. What authentication methods does MAPI/HTTP support? A huge architectural improvement by moving to MAPI/HTTP is that MAPI/HTTP is abstracted from authentication. In short authentication is done at the HTTP layer, so whatever HTTP can do, MAPI/HTTP can use. Does moving the MAPI/HTTP negatively affect the Lync client at all? No, the Lync client uses the same profile as configured by Outlook and will connect via whatever connectivity method is in use by Outlook. Is there any kind of SDK/API for third party application usage of MAPI/HTTP? The MAPI/HTTP protocol is publically documented (PDF download) and has the same level of documentation support as RPC/HTTP. There are no plans to update the MAPI CDO library for MAPI/HTTP and third party companies are still encouraged to utilize Exchange Web Services as the long-term protocol for interaction with Exchange as discussed in Exchange 2013 Client Access Server Rolearticle. What are the client requirements for MAPI/HTTP? Use of MAPI/HTTP requires Outlook 2013 clients to obtain Office 2013 Service Pack 1 or the February update for Office 365 ProPlus clients. Outlook 2010 was updated to support MAPI/HTTP in the January 2015 Public Update, and additional fixes for it were released in the April 2015 Public Update. How does this affect publishing Exchange 2013 via ARR, WAP, TMG, UAG, B2M, ABC, BBD … Publishing Exchange 2013 with MAPI/HTTP in use does not change very much. You will need to ensure devices in front of Exchange handling user access to CAS are allowing access to the Default Web Site’s /mapi/ virtual directory. At this time UAG SP3 is not compatible with MAPI/HTTP even with all filtering options disabled. UAG plans to add support for MAPI/HTTP in a future update. Learn More Still want more information? Review the following sessions on this topic from Microsoft Exchange Conference 2014 Outlook Connectivity: Current and FutureOutlook Connectivity: Current and FutureOutlook Connectivity: Current and Future What's New in Outlook 2013 and Beyond What's New in Authentication for Outlook 2013Announcing OAuth 2.0 support for IMAP and SMTP AUTH protocols in Exchange Online
Ever since we announced our intention to disable Basic Authentication in Exchange Online we said that we would add Modern Auth (OAuth 2.0) support for the IMAP, POP and SMTP protocols. Today, we’re excited to announce the availability of OAuth 2.0 authentication for IMAP and SMTP AUTH protocols to Exchange Online mailboxes.Managing OAB in Exchange Server 2013
The Exchange team blog article OAB in Exchange Server 2013 introduced the new Offline Address Book (OAB) generation and distribution architecture in Exchange Server 2013. Take a few moments to visit the article if you haven’t seen it yet or re-visit it for a quick refresher. The OAB management and administration is different in Exchange 2013 because of architecture changes. Additionally, the new Exchange Admin Center does not currently have options for managing OABs. This means that, at this time, you will need to use Exchange Management Shell for OAB-related tasks. This article takes you through commonly performed tasks in OAB administration and has a couple of real life scenarios to help understand the tasks better. Note: If you are in multi-forest Active Directory domain environment, make sure the Shell session has ViewEntireForest is enabled, else some of the commands in the article won’t return any output. Command to enable ViewEntireForest: Set-ADServerSettings -ViewEntireForest $true Creating a new OAB Creating a new OAB in Exchange 2013 no longer uses the -Server parameter. In order to create a new OAB, you should only specify the address lists to be required. The following example creates OAB for address list named “Global Address List FAB” New-OfflineAddressBook -Name OAB-FAB -AddressLists "Global Address List FAB" Identify the OAB generation server(s) The arbitration mailboxes in Exchange Server 2013 are assigned certain “Persisted capability” that defines the purpose/function of the arbitration mailbox. An arbitration mailbox with Persisted Capability “OrganizationCapabilityOABGen” is responsible for OAB generation. We will refer this mailbox as “Organization Mailbox” throughout the article. Exchange Server 2013 mailbox server hosting the Organization Mailbox will generate all OAB’s defined in the environment. For a non-DAG environment, use following command to identify the OAB Generation servers: Get-Mailbox -Arbitration | where {$_.PersistedCapabilities -like "*oab*"} | ft name,servername For a DAG environment, identifying OAB generation server(s) is a two-step process. Step1: Identify the mailbox database hosting organization mailbox with OAB Gen capability. Use the following command to list the arbitration mailboxes with persisted capability of OABGen and database on which this mailbox is hosted: Get-Mailbox -Arbitration | where {$_.PersistedCapabilities -like "*oab*"} | ft name,database Step2: Identify the mailbox server where the database hosting organization mailbox is mounted Use following command to identify active copy of mailbox database: Get-MailboxDatabaseCopyStatus db1 The server where database status is “mounted” is the current OAB generation server. Change the OAB generation server There are two methods of changing the OAB generation server. Move mailbox Move the organization mailbox to a mailbox database on a server intended to be designated as OAB Generation server. Example: DB1 is a single copy database present on the server Exch1 and hosts the organization mailbox. DB2 is mailbox database present on Exch2. The following command can be used to move the organization mailbox to DB2 and make Exch2 the OAB generation server. Get-Mailbox -Arbitration -database db1| where {$_.PersistedCapabilities –like “*oab*”} | New-MoveRequest -TargetDatabase db2 This method is more suited for environments that have single copy of mailbox database hosting the Organization Mailbox. Activate the mailbox database on another server This method is suited for environments that have multiple copies of the mailbox database hosting Organization Mailbox. Example: DB1 hosts the Organization Mailbox and has copies on servers Exch1 and Exch2. DB1 is currently active on Exch1. The following command can be used to activate DB1 on Exch2 and therefore make it the OAB generation server: Move-ActiveMailboxDatabase DB1 -ActivateOnServer Exch2 Note: Review guidelines mentioned in “Placement of Organization Mailbox” below before changing the OAB Generation server. Creating a new Organization Mailbox Administrators can create additional Organization Mailboxes for fault tolerance or for serving users in a geographically disbursed Exchange deployment. Creating a new Organization Mailbox is a two step process: Step1: Create a new arbitration mailbox New-Mailbox -Arbitration -Name "OAB Seattle" -Database DB2Seattle -UserPrincipalName oabs@contoso.com –DisplayName “OAB Mailbox for Seattle” Step2: Enable OABGen capability Set-Mailbox -Arbitration oabs -OABGen $true -MaxSendSize 1GB Note: Review guidelines mentioned in “Placement of Organization Mailbox” below before creating additional organization mailboxes. Changing the OAB Generation Schedule The OAB Generation till Exchange Server 2010 was based on a “Schedule” set on OAB properties. You might see a “Schedule” defined when viewing properties of Exchange 2013 OAB. But, the Exchange Server 2013 OAB generation does not take place according to the “Schedule” defined on OAB properties: Instead, Exchange Server 2013 OAB Generation takes place according to OABGeneratorWorkCycle and OABGeneratorWorkCycleCheckpoint properties configured at the Mailbox Server. Example: The values in the above screenshot mean OAB is generated once every day. Which Mailbox Server processed the OAB download request? The Exchange Server 2013 CAS role proxies the OAB download request to an appropriate Mailbox role server. The CAS role maintains log of each request it handles in the log files, present in folder %ExchangeInstallPath%\Logging\HttpProxy\OAB\ These log files are an excellent tool to identify which mailbox server the CAS chose to serve the request. Information of some important fields from the log file: Field Description UrlStem Useful to identify the OAB being downloaded and also if this was a full download or incremental download AuthenticatedUser Name of the User requesting the OAB AnchorMailbox DN of Organization Mailbox identified as the closest to serve the OAB request ServerHostName CAS Server Name handling the request HttpStatus Status code for Proxy action ProxyAction Action CAS Server performed for the request, it will be mostly “Proxy” for Exchange 2013 OAB TargetServer Name of Mailbox role server to which request was proxied The log file can be imported in Excel for better readability. Example: Forcing the OAB Generation The Exchange Server 2013 OAB generation can be forced to start immediately by two methods. Method 1: Update-OfflineAddresBook Below command will force OAB generation of an OAB named "Default Offline Address Book" across all organization mailboxes. Update-OfflineAddressBook "default offline address book" Note: This command initiates an RPC request to each mailbox server hosting an active organization mailbox. Method 2: Restart the Mailbox Assistant service. The Microsoft Exchange Mailbox Assistant service on Mailbox Role is responsible for generating OAB. Restarting this service generates all OAB’s defined in the environment on a specific mailbox server, if it’s hosting an active organization mailbox. Placement of Organization Mailbox Exchange Server 2013 CAS role proxies the OAB download request to a “nearest” mailbox server hosting an active Organization Mailbox. It can proxy the request in round robin fashion if it finds more than one organization mailbox active in same AD site. Prior to CU5, this will result in frequent full OAB downloads and is therefore, not recommended. Hence, current guidance is to plan organization mailbox placement such that you will have one organization mailbox active in an AD site. This applies to creating a new organization mailbox as well as to creating copies of mailbox database that hosts an organization mailbox. Prior to CU5, customers should only deploy a single OAB generation mailbox per Exchange organization to prevent users from accessing different OAB generation mailboxes and requiring a full OAB download. With CU5 and later, customers can assign OABs to specific OAB generation mailboxes and not have to worry about accidentally triggering full OAB downloads due to accessing different OAB generation mailboxes. For more information, please see the article, OAB Improvements in Exchange 2013 Cumulative Update 5. Scenarios The following scenarios discuss a real life situation to further explain the new OAB management methods. Scenario 1: Create a new Organization Mailbox Contoso has Exchange Server 2013 Mailbox & CAS role servers deployed at Dallas and Seattle sites. John, the Exchange Admin for Contoso, analyzes the http proxy log files on CAS servers and finds the OAB download request for Seattle users is going to Dallas servers. On further investigation, John finds he has just one Organization Mailbox present in Dallas, hence OAB download requests of all the users are going to Dallas server. He decides to create a new Organization Mailbox at Seattle site with following commands: Step1: Create a new Arbitration Mailbox New-Mailbox -Arbitration -Name "OAB Seattle" -Database DB2Seattle -UserPrincipalName oabs@contoso.com –DisplayName “OAB Mailbox for Seattle” Step2: Enable the Arbitration Mailbox with OABGen capability Set-Mailbox -Arbitration oabs -OABGen $true -MaxSendSize 1GB Scenario 2: Customize OAB Generation Schedule Ben is an administrator of Exchange 2013 deployment at Tail Spin Toys. The default OAB generation schedule does not suit them and they want to generate OAB approximately every fourth hour of the day. Ben will use following command to change properties of the mailbox servers that will be hosting the Organization Mailbox. Set-MailboxServer Exch1 -OABGeneratorWorkCycle 01.00:00:00 -OABGeneratorWorkCycleCheckpoint 04:00:00 After a couple of days, John analyzes Event ID 17002 in application log and makes sure the OAB is generated every four hours. Hopefully, you find this post useful! Let us know your feedback below! Bhalchandra Atre Updates 5/15/14: Updated Placement of Organization Mailbox section to align with current guidance.Demystifying Hybrid Free/Busy: what are the moving parts?
Hybrid Free/Busy is one of those things that many people do not fully understand. If everything works well, the complexity is hidden from view and people working in various parts of organization can seamlessly work together. But if things go wrong… you will appreciate deeper understanding of what makes it work. This is why we wanted to create the blog post series on the subject. In this article, we will discuss how Free/Busy works in an Exchange Hybrid configuration. In next blog post, you will learn what we (Microsoft Support) see as the most common problems along with how we go about diagnosing those (often) complex issues. Hope you like reading, because there is a lot to cover! So, what is Free/Busy? Free/Busy is a feature that allows you to see when others are free (their calendar shows availability), busy (their calendar shows them as busy), or even Out of Office, or Something Else (tentative or working away) so that you can find an appropriate time for your meetings. Calling it all “Free/Busy/OOF/Something-Else” didn’t sound so cool to marketing hence “Free/Busy”. In a Hybrid deployment, we usually have some mailboxes in Exchange On-Premises and some mailboxes in Exchange Online (users are in different premises) and this has to work there too. One of the most important parts in Hybrid Configurations is the Federation Trust and many features, including Free/Busy can rely upon this. A quick overview of a hybrid deployment with a focus on Federation Trust components In Contoso – On-Premises side we have on-premises Exchange Servers and mailboxes. We also have a federation trust with the Microsoft Federated Gateway (MFG) - now called Azure Authentication System. A Federation trust is not set by default for Exchange On-Premises organizations and we can either create it manually or run the Hybrid Configuration Wizard (HCW) which will do this for us. If you don’t have already a federation trust established on-premises (usually for purposes to share F/B with another on-premises organization) and you plan for a Hybrid deployment, then we strongly recommend you run HCW to automatically create the Federation Trust. In Contoso – Cloud Side there are Office 365 Exchange Online servers, your Office 365 tenant and mailboxes migrated from on-premises. A Federation trust with the MFG is automatically created for cloud-only-based Exchange organizations whom do not have a Hybrid relationship to an on-premises organization. If you run Get-FederationTrust cmdlet in Exchange Online PowerShell (see here how to connect to Exchange Online PowerShell) you would see two trusts: “WindowsLiveId” (Consumer Instance of Microsoft Federation Gateway) and “MicrosoftOnline” (Business Instance of Microsoft Federation Gateway) . Note: As a troubleshooting tip, you might want to make sure the Application Identifier is “260563” and the Application Uri is “outlook.com” for both; in case you have a different App ID (292841) and a different App URI (outlook.live.com) for a Cloud trust, this means your tenant has an old reference pointing to MFG and most probably Free/Busy from on-premises to cloud would fail with a quite generic 401 Unauthorized error. If you were to find yourself in a such situation, please open a support case with Microsoft to get it resolved. About Free/Busy lookup directionality To guide you to the correct troubleshooting steps we first need to determine what direction Free/Busy queries are failing in. Understanding how Free/Busy works in general and the direction that lookups are failing can greatly simplify the troubleshooting steps. Simply said, there are 2 possible Free/Busy directions (referring to our example above): When Joe looks up Jane’s Free/Busy, the Free/Busy direction is On-premises to Cloud. When Jane looks up Joe’s Free/Busy, the Free/Busy direction is Cloud to On-Premises Now that you are familiar with the Free/Busy directions, we should take a moment to discuss how it all works. Let me set the stage for the below diagram: joe@contoso.com has his mailbox in Exchange on-premises and jane@contoso.com has been moved to Exchange Online. In Exchange on-premises, joe@contoso.com is a Mailbox User. Because Directory Synchronization is a requirement for Hybrid Deployments, joe@contoso.com (mailbox) is synced to the cloud and represented as a Mail Enabled User (MEU) object in Cloud with Target Address (TA) of joe@contoso.com. jane@contoso.com was once a Mailbox User in Exchange on-premises. Her mailbox was then migrated to Exchange Online. Jane now has a Mailbox User in Exchange Online and is represented as a Remote Mailbox (RM) object in Exchange On-Premises. Jane has a Primary SMTP address (SMTP: jane@contoso.com) and a secondary smtp address (smtp: jane@contoso.mail.onmicrosoft.com). Notice how we refer to a primary email address (SMTP) and to a secondary email address (smtp). According to TechNet, the difference between primary and secondary addresses is that the primary address serves as the return e-mail address. When mail is received from a recipient, the primary address determines which address the mail appears to have come from. Recipients can receive mail sent to any of the addresses associated with them (primary or secondary). The Target Address (TA) of the on-premises Remote Mailbox object is represented as jane@tenant.mail.onmicrosoft.com (secondary smtp) and this is crucial for Autodiscover and email routing from on-premises to cloud. A successful Autodiscover query is important in the Free/Busy process as it provides us with the Availability Service URL of the user which is the External EWS URL of an Exchange Organization. This TechNet article gives us more information on this <domain>.mail.onmicrosoft.com secondary email address: The wizard (HCW) adds an accepted domain to the on-premises organization for hybrid mail flow and Autodiscover requests for the cloud organization. This domain, referred to as the coexistence domain, is added as a secondary proxy domain (contoso.mail.onmicrosoft.com) to any email address policies which have PrimarySmtpAddress templates for domains selected in the Hybrid Configuration wizard(contoso.com). By default, this domain is <domain>.mail.onmicrosoft.com. Table below illustrates the hybrid user objects discussed above as well as how to look them up. On-premises commands = Exchange Management Shell Cloud commands = Exchange Online PowerShell Typical Hybrid user objects configuration On-premises commands On-premises objects Corresponding cloud objects Cloud commands ON-PREM MAILBOX USER MAIL ENABLED USER IN CLOUD Get-mailbox Joe@contoso.com | FT PrimarySMTPaddress On-premises mailbox user with SMTP: Joe@contoso.com Cloud mail enabled user with SMTP: Joe@contoso.com Get-MailUser Joe@contoso.com | FT PrimarySMTPAddress (Get-mailbox Joe@contoso.com). EmailAddresses On-premises mailbox user has a secondary smtp address of @contoso.mail. onmicrosoft.com configured by HCW (Email Address Policies) Cloud mail enabled user has this secondary smtp Joe@contoso.mail. onmicrosoft.com (Get-mailuser Joe@contoso.com). EmailAddresses (Get-mailbox Joe@contoso.com). EmailAddresses On-premises mailbox user has a secondary smtp Joe@contoso.mail. onmicrosoft.com Cloud mail enabled user has an ExternalEmailAddress Joe@contoso.com Get-MailUser Joe@contoso.com |FT ExternalEmailAddress REMOTE MAILBOX ON-PREM MAILBOX USER IN CLOUD Get-RemoteMailbox Jane@contoso.com |FT primarySMTPaddress Remote mailbox SMTP: Jane@contoso.com Mailbox User SMTP: Jane@contoso.com Get-Mailbox Jane@contoso.com |FT PrimarySMTPAddress Get-RemoteMailbox Jane@contoso.com | FT RemoteRoutingAddress Remote mailbox TargetAddress: Jane@contoso.mail. onmicrosoft.com Mailbox User smtp: Jane@contoso.mail. onmicrosoft.com (Get-Mailbox Jane@contoso.com). EmailAddresses Now that we understand user objects and their relevant properties, we should come back to Free/Busy directions. Between the on-premises organization and cloud organization there are bidirectional Organization Relationships and/or bidirectional Intra Organization Connectors (for Exchange 2013 or later) that are created during Hybrid Configuration. The source of these Relationships / Connectors plays an important role in the F/B directionality. The Free/Busy directions are: Direction On-premises to Cloud You are logged in to Outlook on the Web or Outlook on Windows as an on-premises user (joe@contoso.com), you’re your mailbox hosted on your on-premises Exchange server. As an on-premises user you wish to have a meeting with the cloud user (jane@contoso.com) but will first have to check their availability for the meeting. Let’s walk through the process: 1. Joe creates a meeting request and adds Jane as an attendee. 2. The on-premises Exchange server in Contoso determines (usually based on Target Address of the mail-enabled user) that Jane is not a local mailbox and has a different domain name of contoso.mail.onmicrosoft.com set as the Target Address. 3. The Availability Service on the on-premises Exchange server (Client Access Server if Ex2010 or Mailbox Server if 2013/2016) in Contoso then checks to see if there is a path to query Jane’s Free/Busy information for Contoso’s cloud side. First, we check if we have an Intra-Organization Connector (1) with the domain name of contoso.mail.onmicrosoft.com (assuming the Exchange server is version 2013 or later). If an IOC does not exist, then we look to see if an Organization Relationship (2) is configured by looking for the domain name of contoso.mail.onmicrosoft.com in the list of any existing Organization Relationships. If neither an IOC nor an Organization Relationship for the domain contoso.mail.onmicrosoft.com exists, then we look for the domain name contoso.mail.onmicrosoft.com as an Availability Address Space (3). 4. Assuming we would have an Exchange 2010-only environment in the on-premises and we’ve ran HCW successfully, we should expect to see both an Organization Relationship as well as a Federation Trust which combined is the second option from step #3. The Target ApplicationURI in the Contoso on-premises Organization Relationship is set to outlook.com, which is an identifier for the Contoso Cloud organization trust in the Azure Authentication System. A request is then made to the Azure Authentication System for a delegation token that will be accepted by Contoso Cloud Organization. 5. Once the delegation token is received back from the Azure Authentication System, the Exchange server in Contoso on-premises sends an Autodiscover request to Exchange Online, and upon a successful Autodiscover response will send an EWS request to Exchange Online for Jane’s availability information. 6. The Exchange Online server validates the token provided by the Azure Authentication System and once confirmed, returns the requested Free/Busy data for Jane’s mailbox. Here is a flowchart which illustrates the Free/Busy lookup process: Direction Cloud to on-premises You are logged in to Outlook on the Web or Outlook on Windows as a cloud user (jane@contoso.com), whose mailbox is in Exchange Online. Jane wants to have a meeting with an on-premises user (joe@contoso.com), but must first check their availability for the meeting. 1. Jane creates a meeting request and adds Joe as an attendee 2. The Exchange Online servers in Contoso-cloud side organization determine (usually based on target address of a mail-enabled user) that Joe is not a local mailbox and has a domain name of contoso.com set as the target address. 3. The Availability Service on Exchange Online servers checks to see if there is a path for it to find the Free/Busy information for Contoso on-premises side organization. First, we check if we have an Intra-Organization Connector (1) with the domain name of contoso.com (assuming the Exchange server is 2013 or later on-premises). If an IOC does not exist, then we look to see if an Organization Relationship (2) is configured by looking for the domain name of contoso.com in the list of any existing Organization Relationships. If neither an IOC nor an Organization Relationship for the domain contoso.com exists, then we then look for the domain name contoso.com as an Availability Address Space (3). 4. Assuming we would have an Exchange 2010-only environment in the on-premises and we’ve ran HCW successfully, we should expect to see both an Organization Relationship as well as a Federation Trust which combined is the second option from step #3. The Target ApplicationURI in the Contoso Cloud Organization Relationship is set to FYDIBOHF25SPDLT.contoso.com, which is an identifier for the Contoso on-premises organization trust in the Azure Authentication System. A request is then made to the Azure Authentication System for a delegation token which will be accepted by Contoso on-premises organization. 5. Once the delegation token is received back from the Azure Authentication System, the Exchange Online server in Contoso cloud sends an Autodiscover request to the on-premises Exchange Servers and upon receipt of a successful Autodiscover response it will then send an EWS request for Joe’s availability. 6. The Contoso on-premises Exchange server validates the token provided by the Azure Authentication System and once confirmed, returns the requested Free/Busy data for Joe’s mailbox. Hybrid Free/Busy lookup order To summarize, the following order would be used when processing for Free/Busy lookups from cloud to on-premises: Look for IntraOrganizationConnector (OAUTH) If there is no IntraOrganizationConnector or if it is disabled, then look for Organization Relationship (DAUTH) If neither of these are found or they’re disabled, then look for Availability Address Space. Availability Address Space is normally only used for Lotus Notes organizations. If Free/Busy lookup is getting all the way down to using the final Availability Address Space option for cloud to on-premises lookups in a hybrid deployment, then this would suggest there are configuration issues which must be repaired. The order for Free/Busy lookups from on-premises to cloud is almost the same with some exceptions: If we have Exchange 2007 servers in coexistence with Exchange 2010/2013, we use Availability Address Space from Exchange 2007 to Exchange 2010/2013 and then Exchange 2010/2013 will use Org Relationship (Ex2010) or IntraOrganization Connector (Ex2013) to Cloud. If we have Exchange 2010 with Exchange 2013/2016 and OAuth is enabled, Exchange 2010 will use Availability Address Space to Exchange 2013/2016 and then 2013/2016 will use the IntraOrganization Connector to Cloud. If we have Exchange 2010 with Exchange 2013/2016 and OAuth is not enabled, Exchange 2010 will not send the request to the Exchange 2013/2016. Instead Exchange 2010 will use Organization Relationship to Cloud for Free/Busy. Exchange 2013 Cu5+ and Exchange 2016 organizations without coexisting with legacy Exchange servers will use by default IntraOrganization Connectors from On-Premises to Cloud (normal process) This table summarizes which components (Get-IntraorganizationConnector / Get-OrganizationRelationship / Get-AvailabilityAddressSpace) are present (YES) or not (NO) in the Exchange Organization depending on the Exchange Server versions on-premises in a Hybrid deployment. Free/Busy component matrix Exchange Hybrid Environment Intra Organization Connector (IOC) Organization Relationship Availability Address Space Pure Exchange 2016 Hybrid YES (created automatically by HCW) YES YES Pure Exchange 2013 (CU5+) Hybrid YES (created automatically by HCW) YES YES Pure Exchange 2010 Hybrid NO YES YES Mixed Exchange 2007 + Exchange 2010 Hybrid NO - Ex2010 NO - Ex2007 YES - Ex2010 NO - Ex2007 YES - Ex2010 YES - Ex2007 Mixed Exchange 2007 + Exchange 2013 Hybrid YES - Ex2013 (if created manually) NO - Ex2007 YES - Ex2013 NO - Ex2007 YES - Ex2013 YES - Ex2007 Mixed Exchange 2010 + Exchange 2013 Hybrid YES - Ex2013 (if created manually) NO - Ex2010 YES - Ex2013 YES - Ex2010 YES - Ex2013 YES - Ex2010 Mixed Exchange 2010 + Exchange 2016 Hybrid YES - Ex2016 (if created manually) NO - Ex2010 YES - Ex2016 YES - Ex2010 YES - Ex2016 YES - Ex2010 Mixed Exchange 2013 + Exchange 2016 Hybrid YES - Ex2016 (created automatically by HCW) YES - Ex2013 YES - Ex2016 YES - Ex2013 YES - Ex2016 YES - Ex2013 Note: OAuth is by default configured by HCW in Exchange 2013 CU5 and above (including Exchange 2016) organizations. If Exchange 2013 / Exchange 2016 servers coexist with Exchange 2010 or older then OAUTH is not configured by default by the HCW but can be manually configured. Note: Exchange 2013 CU5 is considered old and unlike wine, the finest CU is always the most recent one. You probably have heard about our n and n-1 supported version statement in Hybrid Deployments and can read more about it here and here. Which Free/Busy lookup method are you using? I won't go into all the details about the two types of authentication (OAUTH vs DAUTH) in this post, but I recommend reading this blog post at bedtime: Deep Dive: How Hybrid Authentication Really Works. As explained there, OAuth is an open-standards based model which is more preferred to a customized model. You can quickly determine if you are using OAuth or not by running: Get-IntraOrganizationConnector cmdlet in Exchange Management Shell (for F/B direction On-Premises to Cloud) and in Exchange Online PowerShell (for F/B direction Cloud to On-Premises). If you see Enabled = True, then you are using OAuth or the system should at least be trying to. Note: We also need to make sure that the Target Domain is present on the Intra Organization Connector or Organization Relationship when deciding if using OAUTH or DAUTH. I have created two flowcharts to show you the logic of using OAUTH vs DAUTH in a F/B lookup process for each F/B direction. The user’s setup remains still the same as shown in previous examples and I am reposting the diagram here so that you don’t need to scroll up. 1. Joe’s Exchange server deciding whether to use Oauth or Dauth : 2. Jane’s Exchange Online server deciding whether to use Oauth or Dauth Now that you know which method is being used (or at least which should be attempted) and we know the direction Free/Busy is failing, we can see if you have everything configured correctly. In most cases these configurations are handled by the HCW and should be accurate and you can re-run the HCW to get things back to a good configuration. If Exchange On-Premises > Exchange Online Free/Busy is failing for all users, you would first check Intra Organization Connector or the Organization Relationship from on-premises. If Exchange Online > Exchange On-Premises Free/Busy is failing for all users, you would first check Intra Organization Connector or the Organization Relationship from Exchange Online. Below, you will see a sample of the expected configuration for Intra Organization Connectors and Org Relationships from both sides (on-premises and cloud). This Baseline configuration was documented by my co-worker Ray Fong (Support Escalation Engineer) and I am very happy to have it when I troubleshoot Free/Busy issues. Exchange Online side Use this article to connect to Exchange Online PowerShell. INTRA ORGANIZATION CONNECTOR IN CLOUD Get-IntraOrganizationConnector | fl TargetAddressDomains,DiscoveryEndpoint,Enabled TargetAddressDomains : {contoso.com} DiscoveryEndpoint : https://autodiscover.contoso.com/autodiscover/autodiscover.svc * Enabled : True Note: In practice, the On-Premises Discovery Endpoint (Autodiscover) is more likely to be found in the format https://mail.contoso.com/autodiscover/autodiscover.svc because of the EWS External URL, so pay attention to this Autodiscover URL and replace it if needed with the autodiscover.yourdomain.tld on the IOC present in the Cloud Side (Reference Set-IntraOrganizationConnector) Get-IntraOrganizationConfiguration | fl OnPremiseTargetAddresses OnPremiseTargetAddresses : {contoso.com} TargetAddressDomains - This should be your federated domains. Example: contoso.com You can find the domains name by cross-checking Exchange Online's (Get-IntraOrganizationConfiguration).OnPremiseTargetAddresses TargetDiscoveryEndpoint - This should be the address of the On-Premises Autodiscover Endpoint. Example: https://autodiscover.contoso.com/autodiscover/autodiscover.svc/.If you paste the URL in IE, you should receive a regular windows authentication security prompt Enabled - This must be True. Exchange On-Premises side (use Exchange Management Shell) INTRA ORGANIZATION CONNECTOR IN ON-PREM Get-IntraOrganizationConnector | fl Name,TargetAddressDomains,DiscoveryEndpoint,Enabled Name : ExchangeHybridOnPremisesToOnline TargetAddressDomains : {contoso.mail.onmicrosoft.com} DiscoveryEndpoint : https://outlook.office365.com/autodiscover/autodiscover.svc Enabled : True TargetAddressDomains - This should be your tenant.mail.onmicroosft.com name. Example: 'contoso.mail.onmicrosoft.com' TargetDiscoveryEndpoint - This should be the address of EXO Autodiscover endpoint. Example: https://outlook.office365.com/autodiscover/autodiscover.svc Enabled - This must be True. Exchange Online side Use this article to connect to Exchange Online PowerShell. ORGANIZATION RELATIONSHIP IN CLOUD Get-OrganizationRelationship "Exchange Online to on premises Organization Relationship" | fl DomainNames,FreeBusy*,Target*,Enabled DomainNames : {contoso.com} FreeBusyAccessEnabled : True FreeBusyAccessLevel : LimitedDetails FreeBusyAccessScope : TargetApplicationUri : FYDIBOHF25SPDLT.contoso.com TargetSharingEpr : TargetOwaURL : TargetAutodiscoverEpr : https://autodiscover.contoso.com/autodiscover/autodiscover.svc/WSSecurity Enabled : True DomainNames - This should be your federated domains. Example: contoso.com You can find the domains name by cross-checking On-Premises' (Get-FederatedOrganizationIdentifier).Domains TargetAutodiscoverEPR - This should be the address of the On-Premises Autodiscover Endpoint. Example: https://autodiscover.contoso.com/autodiscover/autodiscover.svc/WSSecurity. If you paste the URL in IE, you should receive a regular windows authentication security prompt TargetSharingEPR - Ideally this is blank. If it is set, it should be the On-Premises Exchange servers EWS ExternalUrl endpoint. Example: https://server.contoso.com/EWS/Exchange.asmx You can find the URL by cross-checking On-Prem's Get-WebServicesVirtualDirectory ExternaUrl. If you paste the URL in IE with /WSSecurity at the end (https://server.contoso.com/EWS/Exchange.asmx/WSSecurity), you should receive a regular windows authentication security prompt TargetApplicationURI - This must match the ApplicationUrI from On-Prem. Example: FYDIBOHF25SPDLT.contoso.com You can find the value by cross-checking On-Prem's (Get-FederationTrust).ApplicationUri FreeBusyAccessEnabled - This must be True. FreeBusyAccessLevel - This should be either AvailabilityOnly or LimitedDetails. AvailabilityOnly: Free/Busy access with time only LimitedDetails: Free/Busy access with time, subject, and location FreeBusyAccessScope - This is typically blank. The FreeBusyAccessScope parameter specifies a security distribution group in the internal organization that contains users that can have their Free/Busy information accessed by an external organization. Enabled - This must be True. Exchange On-Premises side (use Exchange Management Shell) ORGANIZATION RELATIONSHIP IN ON-PREM Get-OrganizationRelationship "On Premises to Exchange Online Organization Relationship" | fl DomainNames,FreeBusy*,Target*,Enabled DomainNames : {contoso.mail.onmicrosoft.com} FreeBusyAccessEnabled : True FreeBusyAccessLevel : LimitedDetails FreeBusyAccessScope : TargetApplicationUri : Outlook.com TargetSharingEpr : TargetOwaURL : https://outlook.com/owa/contoso.onmicrosoft.com TargetAutodiscoverEpr : https://autodiscover-s.outlook.com/autodiscover/autodiscover.svc/WSSecurity Enabled : True DomainNames - This should be your tenant.mail.onmicrosoft.com name. Example: contoso.mail.onmicrosoft.com TargetAutodiscoverEpr - This should be a valid Exchange Online Autodiscover endpoint. Example: https://autodiscover-s.outlook.com/autodiscover/autodiscover.svc/WSSecurity or https://pod51038.outlook.com/autodiscover/autodiscover.svc/WSSecurity [<-- A pod address] You can find the value from TargetAutodiscoverEpr of Get-FederationInformaiton -DomainName tenantname.mail.onmicrosoft.com -BypassAdditionalDomainValidation | fl TargetSharingEPR - Ideally this is blank. If it is set, it should be Office 365 EWS endpoint. Example: https://outlook.office365.com/EWS/Exchange.asmx TargetApplicationURI - This must be outlook.com if this is for organization relationship of the cloud tenant. For non-cloud organization relationship, this must match (Get-FederationTrust).ApplicationUri of the On-Prem trust of the other organization. FreeBusyAccessEnabled - This must be True. FreeBusyAccessLevel - This should be either AvailabilityOnly or LimitedDetails. AvailabilityOnly: Free/Busy access with time only LimitedDetails: Free/Busy access with time, subject, and location FreeBusyAccessScope - This is typically blank. The FreeBusyAccessScope parameter specifies a security distribution group in the internal organization that contains users that can have their Free/Busy information accessed by an external organization. Enabled - This must be True. Both Autodiscover and EWS virtual directories must be enabled for WSSecurity authentication and/or OAuth. For example, if using OAuth, you should have OAuth listed as Authentication Methods, whereas if using only DAuth (Exchange 2010 for example), you would see only WSSecurity. Example of virtual directories authentication methods for an Exchange 2013 Hybrid Organization: Get-AutodiscoverVirtualDirectory -Server SERVER01 | fl Name,AdminDisplayVersion,*authentication* Name : Autodiscover (Default Web Site) AdminDisplayVersion : Version 15.0 (Build 1044.25) InternalAuthenticationMethods : {Basic, Ntlm, WindowsIntegrated, WSSecurity, OAuth} ExternalAuthenticationMethods : {Basic, Ntlm, WindowsIntegrated, WSSecurity, OAuth} Get-WebServicesVirtualDirectory -Server server01| fl Name,AdminDisplayVersion,*Authentication* Name : EWS (Default Web Site) AdminDisplayVersion : Version 15.0 (Build 1044.25) InternalAuthenticationMethods : {Ntlm, WindowsIntegrated, WSSecurity, OAuth} ExternalAuthenticationMethods : {Ntlm, WindowsIntegrated, WSSecurity, OAuth} As explained earlier, there are some situations where Free/Busy from on-premises to cloud is going via Availability Address Space. This would be the expected configuration for Availability Address Space in Exchange on-premises (Exchange Management Shell): Get-AvailabilityAddressSpace contoso.mail.onmicrosoft.com | fl ForestName, UserName, UseServiceAccount, AccessMethod, ProxyUrl, Name ForestName : contoso.mail.onmicrosoft.com UserName : UseServiceAccount : True AccessMethod : InternalProxy ProxyUrl : https://server01.contoso.com/EWS/Exchange.asmx Name : contoso.mail.onmicrosoft.com ForestName - The should be the tenant.mail.onmicrosoft.com domain name. This should also match the domain name of RemoteRoutingAddress of remote mailboxes. Example: contoso.mail.onmicrosoft.com UserName - This should be blank. UserServiceAccount - This must be True. AccessMethod - This should be InternalProxy. ProxyUrl - This should be the Exchange 2013/2016 Exchange Web Services Virtual Directory URL. The address could be the internal FQDN or load balancing EWS URL. Example: https://server01.contoso.com/EWS/Exchange.asmx We recommend you check the following requirements for inbound/outbound connectivity to and from Exchange servers in a Hybrid Deployment: Understanding Federated Delegation Office 365 URLs and IP address ranges If you have read this far – thank you! This concludes the Part 1 of this blog series. Onto troubleshooting next! Huge thanks to all that contributed to this blog post: Ray Fong, Nino Bilic, Tim Heeney, Greg Taylor and Brian Day. Mirela BuruianaDemystifying Hybrid Free/Busy: Finding errors and troubleshooting
EDIT 9/19/2023: This blog post has received significant update. In this second part of the Demystifying Hybrid Free/Busy, we will cover troubleshooting of Hybrid Free/Busy scenarios, more specifically – how and where to find an actual error that will indicate where the problem is. Before venturing forth, please make sure that you have seen Part 1 of this demystifying series! Here is the graphics we posted in the previous post; use this as a reference for users that we will be referring to when troubleshooting: Do you really have a Free/Busy issue? Usually when a user creates a new meeting in Outlook on the web (OWA) or Outlook, clicks on Scheduling Assistant, adds his or her colleague to the meeting, they try to see when the user is available to meet. If they see the hash marks \\\\\\\ instead of seeing if the other user is free or busy, there is an issue. Here, we do seem to have a bunch of Free/Busy issues: You can often see an error message by hovering over hash marks, however we usually find that the error is not very specific. Instead, we would need to take slightly more advanced steps to diagnose the issues by checking things like the Remote Connectivity Analyzer tool, Fiddler, F12 Network tab, Outlook logging or SARA tool. Where is the actual Free/Busy error message? First, we need to understand in which direction we have a lookup problem. Please see Part 1 for discussion of directionality. Sources of logs: Remote Connectivity Analyzer tool Outlook logging SARA tool OWA F12 Network Tab Fiddler – Outlook and OWA These steps are important for us to see the relevant message error for Free/Busy issues. Once we know the error message, it’s much easier to resolve the issue. Remote Connectivity Analyzer A few things to know about this tool: Source Mailbox: the user that will be requesting the free/busy information. This will be the user that is logged in Outlook or OWA and cannot see free/busy for other people. This is also called Requester or Organizer of the meeting. Authentication type for Source Mailbox: you will choose Modern Authentication Source Mailbox credentials: you will need to authenticate with the credentials of the Source Mailbox. The tool doesn’t support Basic Authentication for Exchange Online mailboxes because this is disabled in Exchange Online. While it is still used by Exchange On-premises environments, currently, if you select Basic Authentication for the on-premises source mailbox, the test will fail before doing the actual Free/Busy process. It works if your Exchange on-premises has enabled Modern Authentication for client protocols. In conclusion, Source Mailbox login needs to be using OAuth for this test to work, regardless of where it is hosted. Target Mailbox: the user that the Source Mailbox is requesting free/busy for. This is the Attendee of the meeting. The tool simulates Outlook’s way of querying Free/Busy. If you have a free/busy issue that is only happening in OWA but not in Outlook Desktop, then this test will likely not catch the error. To be able to perform the test, you must allow connectivity for the Remote Connectivity Analyzer tool’s IP addresses. These are part of the "Microsoft 365 Common and Office Online" ranges published in the Office 365 URLs and IP address ranges. The IPs for the Remote Connectivity Analyzer are part of the range specified as "Allow Required" (currently ID 46 in the documentation). Check https://testconnectivity.microsoft.com/Pages/ChangeList.htm for any future changes. Note that you can only insert one Target Mailbox email address per test. If you have errors for multiple target mailboxes, run multiple tests, for each user. Connectivity Test Results: With these 3 buttons on the top right corner, you can expand all the results and save them as XML or HTML files. Usually, support people appreciate these files a lot, so please do upload them in your support case workspace. When you expand the results, there are 3 important checks: Determining where the source mailbox is hosted (cloud or not). If the Mailbox is hosted in cloud, you will see something like this: IsOffice365Mailbox=True. The mailbox is hosted in Office 365. <ASURL>https://outlook.office365.com/EWS/Exchange.asmx</ASURL> If the Mailbox is not hosted in cloud, you will see something like this: IsOffice365Mailbox=False. The mailbox isn't hosted in Office 365. Determining where the target mailbox is hosted (cloud or not). Test Autodiscover for the Target Mailbox SMTP to retrieve External EWS url. Quick tip: on your side, in Windows PoweShell, you can also use the following commands to see the External EWS url of an user based on the Autodiscover call to Office 365, replace what is in Email= with your actual email addresses. Invoke-RestMethod -Uri "https://outlook.office365.com/autodiscover/autodiscover.json?Email=CLOUDUSER@CONTOSO.COM&Protocol=EWS" Invoke-RestMethod -Uri "https://outlook.office365.com/autodiscover/autodiscover.json?Email=ONPREMUSER@CONTOSO.COM&Protocol=EWS" Performing the Free/Busy Lookup. This will be Success or Failed. If it failed, look under the Additional details to see the error message. If success, be happy, maybe the issue is resolved, or not be happy as it might be an intermittent issue (which is harder to troubleshoot) or a local issue only (happening in your specific network, machine, Outlook version). In my case, I see that I have a NoFreeBusyAccessException, given by the Exchange on-premises server HHE1601. OUTLOOK Note: The Modern Outlook clients log Free/Busy information in Outlook ETL files and you won’t be able to see the Free/Busy error in plain text here. This was possible with Outlook 2010 logs, back in the old days. But this method is still useful, because you can provide the Outlook ETL log containing the error to Microsoft Support to parse it for you and help you fix it also. If you want to see the error for yourself, check the Fiddler method. For the Outlook F/B error, we need to first enable Outlook logging and after this we will need to reproduce the issue (\\\\\\). After repro, we will collect the Outlook logs. Steps: Enable Outlook logging: Follow this KB article and check the “Enable troubleshooting logging (this requires restarting Outlook)” option. Restart Outlook. Reproduce the issue for the non-working free/busy direction. Suppose Free/Busy direction not working is cloud to on-premises, you will be logged on as a cloud user (Source Mailbox), go to Calendar tab, New Meeting, Scheduling Assistant, add some on-premises users to a meeting until you see the hash marks (instead of Free/Busy information). You do not need to save or send a meeting request. Collect the Outlook-#####.etl log from %temp%\Outlook Logging folder (reference here). You would need to send the ETL file to Microsoft Support to get it analyzed as we are parsing this log with an internal tool. You might not know this, but Hybrid free/busy support cases are free of charge! Of course, you can still use the other methods (fiddler for Outlook/OWA or browser for OWA) to see Free/Busy error yourself, however we (Support) might ask you additionally to get this log as well for a further dive into the Free/Busy errors. SARA I would also like to mention that there is a Free/Busy troubleshooter in Beta version, incorporated into SARA tool (Microsoft Support and Recovery Assistant for Office 365) which you can download it from here : https://diagnostics.outlook.com/#/ Open SARA and select Outlook scenario, click Next, then select I’m having problems with my calendar, input email address and password of the source mailbox (cloud mailbox if direction not working is cloud > on-premises) and then select I can’t see when someone is free or busy. Due to the underlying complexity of it all, this is not a completely reliable way of determining the cause of free/busy issues in Hybrid Deployments, but it is a good start when troubleshooting. This F/B test from SARA covers mostly cloud to cloud scenarios but I recommend it here because it does connectivity and additional checks on tenant, licensing and Autodiscover. And sometimes it shows the underlying Free/Busy error message. Here are some screenshots with the SARA process: After the Office 365 readiness checks, the tool will ask you for the email address of the Target Mailbox: In the failed results, expand the Support Message and User Message: OWA / Outlook on the web F12 Network Tab Cloud OWA F12 Network tab You need to login to OWA as the source mailbox, hit F12 (Developer Tools for browser) and select the Network Tab. You would then lookup Free/Busy for the target mailbox (reproduce the issue). If the source mailbox is hosted in Cloud, to look for the F/B here, you can find the Search Icon and type there “GetSchedule” or find the Filter Icon and type “graphql”, then look at Response or Preview tab to see the error message by expanding GetSchedule until you reach to the error. (click thumbnail to view larger) If the Source Mailbox is hosted in Exchange On-Premises, you would look after GetUserAvailabilityInternal: Fiddler –Outlook or OWA You would need to download and install Fiddler tool from the internet, enable HTTPS decryption in Fiddler and then reproduce the Free/Busy issue in Outlook or OWA or both. Fiddler - Exchange Online Source Mailbox logged in Outlook desktop. Look for “GetUserAvailability” calls and then on the right side, you have Request on the top and Response on the bottom. Switch to XML tabs for a nicer view. In the Request you will see the attendees’ email addresses and, in the Response, you will have ResponseMessage with ResponseClass=Error or ResponseClass=Success. Fiddler – Exchange Online Source Mailbox logged in OWA. In Fiddler, you can check in the Request pane, under Raw tab the ClientRequestID, you can for example search after this specific value in your on-premises Exchange server logs: IIS W3SVC2 logs, HTTPProxy EWS logs and EWS logs (more information on these logs, location and extracts, later in the article). Example here from a lab: ClientRequestID: {72741DFF-A6AC-402B-991B-C6B5D56B1422} Date: Mon, 11 Sep 2023 19:01:25 GMT If you are fan of SQL language, you can use a tool like Log Parser Studio and search through these logs, for example, here is a query on the ClientRequestID from earlier: SELECT DateTime, ClientRequestID, RequestID, UserAgent, SoapAction, ErrorCode, GenericErrors, GenericInfo, FileName FROM '[LOGFILEPATH]' WHERE ClientRequestID LIKE '%{72741DFF-A6AC-402B-991B-C6B5D56B1422}%' You can also use findstr.exe utility to look for the client request id or other keywords like the requester’s email address or “CrossForest”. Example of command: findstr.exe /I /S "{72741DFF-A6AC-402B-991B-C6B5D56B1422}" *.log When troubleshooting Free/Busy issues, the following on-premises logs can be very useful, especially for Cloud to On-Premises Free/Busy direction. IIS logs Default Web Site (DWS) Path: %SystemDrive%\inetpub\logs\LogFiles\W3SVC1 Path example: C:\inetpub\logs\LogFiles\W3SVC1 Extract of Autodiscover and EWS log entries with IOC Enabled in IIS W3SVC1 logs: Autodiscover – OAUTH (autodiscover.svc without /WSSecurity) 2016-01-06 17:45:27 10.0.0.5 POST /autodiscover/autodiscover.svc &CorrelationID=<empty>;&ClientId=QNFNHKEEKYENCJITQQ&cafeReqId=7972d1fc-a9d9-44c6-8851-480d3601cbd7; 443 S2S~00000002-0000-0ff1-ce00-000000000000 132.245.65.28 ASAutoDiscover/CrossForest/EmailDomain//15.01.0361.007 200 0 0 109 EWS – OAUTH (exchange.asmx without /WSSecurity) 2016-01-06 17:45:27 10.0.0.5 POST /ews/exchange.asmx &CorrelationID=<empty>;&ClientId=WSIVGUUAUWWRFACJBWDA&cafeReqId=6ce8864c-74a0-4ad2-a3dc-7b69e0415403; 443 <unverified>actas1(sip:joe@contoso.com|smtp:joe@contoso.com|upn:joe@contoso.com) 132.245.65.28 ASProxy/CrossForest/EmailDomain//15.01.0361.007 200 0 0 703 Example of EWS entry with Organization Relationship Enabled in IIS W3SVC1 logs: EWS – DAUTH (exchange.asmx with /WSSecurity) 2016-01-06 18:04:41 10.0.0.5 POST /ews/exchange.asmx/WSSecurity &CorrelationID=<empty>;&ClientId=VOMGJKAWURSVKOXQLBVA&cafeReqId=18fd3a2e-7b1c-4828-8943-6b20912e2e44; 443 - 132.245.65.28 ASProxy/CrossForest/EmailDomain//15.01.0361.007 200 0 0 296 IIS logs Exchange BackEnd (BE) Path: %SystemDrive%\inetpub\logs\LogFiles\W3SVC2 Path example: C:\inetpub\logs\LogFiles\W3SVC2 Example of EWS entry with Organization Relationship Enabled (DAUTH) in IIS W3SVC2 logs: 2016-01-06 18:04:41 fe80::f17f:beef:a5e3:7d3c%25 POST /ews/exchange.asmx/WSSecurity - 444 - fe80::f17f:beef:a5e3:7d3c%25 ASProxy/CrossForest/EmailDomain//15.01.0361.007 200 0 0 93 HTTPProxy logs for Autodiscover Path: %ExchangeInstallPath%Logging\HttpProxy\Autodiscover Path example: C:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Autodiscover Example of Autodiscover entry with Organization Relationship Enabled (DAUTH) 2016-01-06T18:05:20.552Z,bcdfbed5-f11f-4250-a616-e38cb475cd3f,15,0,1104,2,,Autodiscover,autodiscover.contoso.com,/autodiscover/autodiscover.svc /WSSecurity,,,false,,contoso.com,Smtp~joe@contoso.com,ASAutoDiscover/CrossForest/EmailDomain/ /15.01.0361.007,132.245.65.28,exch-2013,200,200,,POST,Proxy,exch-2013.contoso.com,15.00.1104.000,IntraForest,AnchorMailboxHeader-SMTP,[…],BeginRequest=2016-01-06T18:05:20.192Z;CorrelationID=<empty>;ProxyState-Run=None;FEAuth=BEVersion-1941996624;NewConnection=fe80::f17f:beef:a5e3:7d3c%25&0; HTTPProxy logs for EWS Path: %ExchangeInstallPath%Logging\HttpProxy\Ews Path example: C:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Ews Example of EWS entry with Organization Relationship Enabled (DAUTH): 2016-01-06T18:04:41.490Z,4757ab2c-8ccc-4d1a-ae39-0780ecc8eabb,15,0,1104,2,{02CD833F-18AB-413A-83CB-0E86F4DA5362},Ews,mail.contoso.com,/ews/exchange.asmx/WSSecurity,,,false,,contoso.com, Smtp~joe@contoso.com,ASProxy/CrossForest/EmailDomain//15.01.0361.007,132.245.65.28,exch-2013,200,200,,POST,Proxy,exch-2013.contoso.com,15.00.1104.000,IntraForest,AnchorMailboxHeader-SMTP,[…],BeginRequest=2016-01-06T18:04:41.380Z; EWS logs Path: %ExchangeInstallPath%Logging\Ews Path example: C:\Program Files\Microsoft\Exchange Server\V15\Logging\Ews Example of EWS entry with Organization Relationship Enabled (DAUTH): 2016-01-06T18:04:41.490Z,4757ab2c-8ccc-4d1a-ae39-0780ecc8eabb,15,0,1104,2,{02CD833F-18AB-413A-83CB-0E86F4DA5362}, External,true,jane@contoso.mail.onmicrosoft.com,, ASProxy/CrossForest/EmailDomain//15.01.0361.007,Target=None;Req=Exchange2012/Exchange2013; ,132.245.65.28,exch-2013,exch-2013.contoso.com,GetUserAvailability,200,12150,,,,,,ebd34d71ac7342c19d947d881db4ad55,f866c73e-6c91-475e-bdec-0428bdeaa423,PrimaryServer; Requester=jane@contoso.mail.onmicrosoft.com; Failures=0 Event Viewer Application logs on Exchange Server References here and here. Example of Event ID 4002 for MSExchange Availability: Log Name: Application Source: MSExchange Availability Event ID: 4002 Task Category: Availability Service Level: Error Description: Process 4568: ProxyWebRequest CrossSite from S-1-5-21-391720751-1508397712-925700815-508779 to https://hybrid.contoso.com/ews/exchange.asmx failed. Caller SIDs: NetworkCredentials. The exception returned is Microsoft.Exchange.InfoWorker.Common.Availability.ProxyWebRequestProcessingException: System.Web.Services.Protocols.SoapException: You have exceeded the available concurrent connections for your account. Try again once your other requests have completed. at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall) IIS tracing for the error code in the IIS logs Reference here. Free/Busy errors and fixes Based on cumulative support team experience, we created a table (see the attachment to this post) with Free/Busy errors encountered so far and their possible resolutions. We cannot cover all possible scenarios and errors even though we have a good-sized list. This is meant to illustrate ways we can resolve specific errors and these suggestions might not work for you even if you have the same error. If you know the exact Free/Busy error that you get and checked configuration as discussed in part 1 of this series, this is already a tremendous progress, and this will help us resolve your issue faster. Of course, you can follow these suggestions on your own as most of the actions are harmless but if you don’t feel confident in troubleshooting on your own or you fear that actions are dangerous or irreversible, please contact us. Free/Busy Errors discussed in the attached document (FB_Errors_FixesV7): “An internal server error occurred. The operation failed” LID: 59916. 500 Internal Server error. "The remote user mailbox must specify the the explicit local mailbox in the header" "An error occurred when verifying security for the message" "Unable to connect to the remote server" “Autodiscover failed for email address <> with error ‘The request failed with HTTP status 404: Not Found’ ” “The request failed with HTTP status 401: Unauthorized - The user specified by the user-context in the token is ambiguous” LID: 43532 "An existing connection was forcibly closed by the remote host - An unexpected error occurred on a receive " "An existing connection was forcibly closed by the remote host - An unexpected error occurred on a send ” "Configuration information for forest/domain could not be found in Active Directory" "Proxy web request failed.,inner exception: The request failed with HTTP status 401: Unauthorized." "The response from the Autodiscover service at 'https://autodiscover/autodiscover.svc/WSSecurity' failed due to an error in user setting 'ExternalEwsUrl'. Error message: InvalidUser." LID: 33676 “The caller does not have access to free/busy data" LID: 47652 LID: 44348 “The request failed with HTTP status 403: Forbidden (The server denied the specified Uniform Resource Locator (URL). “ LID: 43532 “Unable to resolve e-mail address user@notes.domain.com to an Active Directory object” LID: 57660 “An error occurred when processing the security tokens in the message.” LID: 59916 “The cross-organization request for mailbox yyy@contoso.com is not allowed because the requester is from a different organization” LID: 39660 “The request failed with HTTP status 401: Unauthorized - Microsoft.Exchange.Security.OAuth.OAuth TokenRequestFailedException: Missing signing certificate “ “The application is missing a linked account for RBAC roles, or the linked account has no RBAC role assignments, or the calling users account is logon disabled” “The entered and stored passwords do not match“ “The password has to be changed.” “The password for the account has expired” or “Provision is needed before federated account can be logged in” “The request timed out” “The specified member name is either invalid or empty” “The result set contains too many calendar entries” LID: 54796 “The request failed with HTTP status 401: Unauthorized - The token has an invalid signature.” “The request failed with HTTP status 401: Unauthorized - Client assertion contains an invalid signature. [Reason - The key was not found., Thumbprint of key used by client: '<>’ “ “Proxy web request failed., inner exception: Response is not well-formed XML “ “Failed to communicate with https://login.microsoftonline.com/extSTS.srf., inner exception: Unable to connect to the remote server” “Autodiscover failed for E-Mail Address <> with error System.Net.WebException: The remote name could not be resolved: '<>'” “Failed to get ASURL. Error 8004010F” “Proxy web request failed. , inner exception: System.Net.WebException: The request failed with the error message: -- <head><title>Object moved” “The request was aborted: Could not create SSL/TLS secure channel.” “The user specified by the user-context in the token does not exist.";error_category="invalid_user“ “The hostname component of the audience claim value 'https://<>’ is invalid";error_category="invalid_resource“ “Proxy web request failed. , inner exception: System.Net.WebException: The request failed with HTTP status 503: Service Unavailable” “Proxy web request failed. , inner exception: System.Net.WebException: The request failed with HTTP status 504: Gateway Timeout.” Thanks to all that contributed to this content: Ray Fong, Nino Bilic, Tim Heeney, Greg Taylor and Brian Day. Mirela BuruianaExchange 2010 Cross-Forest Mailbox Moves
EDIT 8/12/2010: Added a note about the necessity to manually enable MSProxy in remote forest. We are seeing some trends where quite a few customers are migrating mailboxes to a new Exchange organization, in a different Active Directory (AD) forest. This blog post is aimed at helping to explain the fundamentals of what is required to move mailboxes across forests so that you can be prepared with the correct data, make better plans, and successfully perform a migration without encountering painful problems. The blog post doesn't cover how to setup and configure shared address space or Free/busy. After reading this blog post, you should have better understanding of: How to plan your migration by understanding your current forest configuration and your desired configuration. Different ways for you to synchronize user data between different AD forests. Networking and Administrator permissions required to perform a successful cross-forest mailbox move. The trends we are seeing currently show that companies are having more trouble understanding the different scenarios than performing the migration. There are several scenarios here, and Microsoft has tools, documentation, and scripts to assist in each one of them. There are many reasons companies choose to have multiple forests or maybe find themselves with multiple forests, requiring cross-forest moves of users and mailboxes. For instance: Companies that merge, are bought out, or have absorbed another company in some manner. Companies who want to start fresh and leave a lot of legacy issues behind. Companies that have subsidiaries; segment their environment by Department, Geography, or for Security considerations. The common Active Directory topologies that are supported in Exchange 2010 are as follows: Single forest, single Active Directory site Single forest, multiple Active Directory sites Multiple forest, multiple Active Directory sites Exchange deployment topologies vary due to organizational size and business complexity. Variations may include Single Forest, Resource Forest, Hybrid Forest, and Cross Forest topology. For purposes of discussion the following forest definitions will be used going forward: Forest Name Active Directory user object status Mailbox Status Exchange Forest Enabled User Object Mailbox Enabled Account Forest Enabled User Object No mailbox enabled objects Resource Forest Disabled User Object (linked to a separate enabled user object in an Account Forest) Mailbox Enabled Hybrid Forest Both 1.) AD Enabled Mailbox Enabled 2.) AD Disabled Mailbox Enabled Both mailbox enabled and disabled objects Most of the Cross-Org Move Mailbox scenarios are closely related to the Active Directory Forests involved in the migration. There are 3 major scenarios to be considered: 1. Move from Exchange Forest A to Exchange Forest B. This means that the user is a security principal in forest A and after he is moved to forest B, he is a security principal in forest B as well. This may be a hybrid-forest scenario, typical during inter-forest migrations, because the user is security principal in both. Hybrid is when there are both enabled and disabled users in the same forest. 2. Move from Account Forest to Exchange Resource Forest. Company is splitting Exchange off to its own forest. Maybe due to outsourcing it, complex business organization, or desire to de-couple the Exchange org (e.g. messaging services) from the other infrastructure. 3. Move from Exchange Resource Forest to Account Forest. This is the reverse of #2. Company is bringing Exchange back into the same forest for simplicity, to better integrate with OCS (though they are not required to be in the same forest), or collapsing/consolidating previously separate Exchange orgs into one user forest. Cross-forest is when all users from the same organization are only contacts or mail enabled user objects in the other forest. This is not referenced as a common scenario because it's usually in place between two separate legal entities and there would not be much movement (e.g. migrations) between them. Active Directory Forest Configuration examples: Below are some AD forest configuration examples. The forest scenarios don't necessarily imply there is a "move" or migration going on, some are long-term configurations. Resource Forest A Resource Forest scenario is a deployment that has at least one Exchange Resource Forest that hosts user mailboxes (but not active user accounts or enabled user accounts) and at least one other forest that hosts the AD user accounts. In other words, Exchange is installed into an AD forest which is separate from the "user account" AD forest. A one-way forest trust where the resource forest trusts the account forest is created. Each mailbox in the Exchange forest must have a corresponding user in the account forest, which is granted access to logon to the mailbox. This is referred to as a "Linked Mailbox". The user objects in the Exchange forest are never logged onto by an end user and are disabled. Hybrid Forest Typically this scenario is maintained initially for co-existence while migrating and decommissioning a forest. It is different from a typical cross-forest scenario because there may be both enabled and disabled users in both forests for the same organization. In some cases, an organization may actually need to maintain the Hybrid Forest scenario over the long-term. While this is a supported scenario, it comes with additional complexity that must be addressed: Mastering User and Exchange attributes occurs on both sides. A tool such as Forefront Identity Manager (FIM), is needed to maintain consistent data on both sides, including the GAL. Free/Busy and Public Folder access requires additional configuration, tools, and in some cases maintaining an Exchange 2007 server. (Please note that the IOREPL tool isn't currently supported with Exchange 2010 as a target server and in fact follows the Exchange 2003 Product support life cycle.) Free/Busy, over the long-term will be best managed using the new Federation services (Microsoft Federation Gateway) For more information refer to Understanding Federation Cross-forest Both forests contain mailboxes and user accounts and contacts. This type of configuration has user accounts always enabled and mailbox enabled, with a corresponding contact in the other forest. The following diagram depicts how different objects are represented in the corresponding forest: For more information on forests related to Cross Org migrations, refer to http://msexchangeteam.com/archive/2006/11/02/430289.aspx Three Migration paths you need to choose from: Depending on the current topology you have employed, you may find yourself planning to move users into the new forest and then following with moving their mailboxes as well. There are essentially three ways of planning to move your resources: A customized deployment in which you write ILM rules extension code to create the target Mail Enabled User (MEU). You should already have a custom ILM deployment for cross forest GALSync. Microsoft Identity Lifecycle Manager Service Pack 1 Feature Pack 1 (ILM 2007 SP1 FP1) GALSync Management Agent (MA) doesn't include several attributes now required in Exchange 2010, most importantly, msExchMailboxGUID. The out of the box GALSync MA cannot be used since it creates contact object instead of user object required for Online Mailbox Move. The ILM sample code demonstrates how to sync source mailbox as Mail Enabled Users (MEU). Note: Customers using "out of the box" GALSync MA may probably not know how to customize ILM. Use Prepare-MoveRequest.ps1 script to create the target MEU. It is important to note that the PrepareMoveRequest script works in conjunction with "out of the box" Exchange GALSync MA for ILM (or FIM). This means the script has built-in logic to convert target Mail Enabled Contact (MEC) created by ILM GALSync MA into the required MEU. Use Prepare-MoveRequest.ps1 script and then use ADMT to migrate the other attributes on the user object. Important Note: Our recommendation on working with ADMT is to rely on the PrepareMoveRequest script to create the local user object for mailbox move, and then use ADMT to migrate SIDHistory and password and merge this into the MEU created by PrepareMoveRequest.ps1 script. The point of doing ILM or the script first is to ensure the MEUs are all created with the correct msExch* attributes. This also ensures the following benefits: A correct GAL immediately for co-existence (short or long-term) Permissions for delegates and mailbox access are preserved during the move using the msExchMailboxGUID attribute. Since this is populated on the target object with PrepareMoveRequest.ps1 the permissions will be maintained in the cross-forest move. At this point it doesn't matter if ADMT is used to migrate/merge the user objects all at once or in "batches" of user objects. ADMT can be controlled better to ensure only merging of SIDhistory and certain other mandatory attributes if it's not already populated. Running ADMT first, without ensuring exclusions on msExch* attributes, can cause corrupted objects which the script cannot correctly convert with the -UseLocalObject switch. Important Note: When SP1 ships, we will support running ADMT first and then the PrepareMoveRequest script later. ILM and PrepareMoveRequest Scenarios broken-down: There are basically 5 steps involved with moving a mailbox across a forest in Exchange 2010. They are: Preparing Active Directory, Network Prerequisites, Administrator Permissions, Moving Mailboxes and Clean-up. Each of these steps is series of smaller steps that need to be taken in order to move a mailbox from one Exchange forest to and Exchange 2010 forest. The first step in Cross Forest mailbox moves is preparing Active Directory. In the target forest a mail enabled user account must be created with certain attributes. The method used for creating the target account and setting the mandatory attributes is up to the organization administrator. ADMT and ILM can be used to synchronize/pull over the attributes from the source forest. Exchange Provisioning using ILM 2007 If you deployed ILM for cross-forest global address list (GAL) synchronization, the recommended approach to creating the mail-enabled user is to use ILM 2007 Service Pack 1 (SP1) Feature Pack 1 (FP1) or Forefront Identity Manager 2010 (FIM) GALSync MA. We've created sample code that you can use to learn how to customize ILM to synchronize the source mailbox user and target mail user. For more information, including how to download the sample code, refer to this link. To deploy Exchange 2010 in a cross-forest topology, you must first install Exchange 2010 in the new forest. Then, provision the mail-enabled users representing the source mailboxes so that Exchange 2010 can move the mailbox and migrated users can see all addresses. Deployment steps: Deploy Exchange 2010 in a cross-forest topology with ILM 2007 FP1 SP1. Import and install the ILM sample code from Prepare Mailboxes for Cross-Forest Moves Using Sample Code. Note: The main purpose of the sample code is to encourage customers to customize, or add more functions to the sample code. The sample code is very basic and it only copies very basic attributes. Customers who rely on this sample code may find many attributes missing. Configure the Mail-Enabled User provisioning Management Agents for each forest. This allows the mailboxes in the source forest to be created as MEU in the target forest and ensure a common GAL. Create an SMTP Send connector in each forest and configure SMTP namespace sharing (http://technet.microsoft.com/en-us/library/bb676395.aspx). In each forest, enable the Availability service so that users in each forest can view free/busy data about users in the other forest. For more information, see Managing the Availability Service. Note: The Availability service is supported only for Outlook 2007 clients and newer. If Outlook 2003 clients still exist in one of the forests, the only solution will be to deploy Exchange 2007 first in the Exchange 2010 organization (because adding it late is not possible if Exchange 2010 is deployed first) and implement the IOREPL tool to replicate Free/Busy system public folders to the Exchange 2007 server. The Free/Busy system public folder replicas can then be replicated using PF replication to your Exchange 2010 server. IOREPL will not replicate a public/system folder directly to an Exchange 2010 server. For more information review: Exchange Provisioning using ILM 2007 and FIM 2010 http://technet.microsoft.com/en-us/magazine/ff472471.aspx Prepare-MoveRequest.ps1 It may be difficult for some customers to synchronize the prerequisite attributes for performing mailbox moves without using ILM. You may have some other solution in place that does not synchronize the required attributes, and does not allow customization. Small companies may not have a solution at all and simply wish to transition users from an existing forest (that is set to be obsolete) to a new, clean Exchange 2010 forest. To solve this problem, the PrepareMoveRequest script has been written to prepare the AD target object and synchronize the required attributes for cross-forest moves to work. The script creates the target MEU if necessary, or synchronizes an existing MEU when possible. The PrepareMoveRequest script prepares Exchange 2003, Exchange 2007, and Exchange 2010 mailbox users for migration to an Exchange 2010 forest. For more information about using the sample script, refer to the following link. The PrepareMoveRequest script supports 2 scenarios: Creating a brand new user in the local forest where the MBX will be moved to. A local recipient, either a MEU or MEC already exists, created by an external agent such as ILM - If the local forest object is a mail contact, the script will convert the mail contact to a mail user while persisting the contact's existing exchange-related attributes. If the local forest object is a MEU, the script will reuse this mail user and stamp the essential attributes on the local mail user object. The administrator must specify the -UserLocalObject switch in order to tell the script to use this scenario. Note: The scenario that the script doesn't support is that some external process created a local user object and relies on the script to copy all the attributes and links from the remote MBX to the local user. This is the ADMT scenario described after this scenario. In order to run New-MoveRequest cmdlet to move a mailbox from an Exchange 2003/2007/2010 source forest to an Exchange2010 target forest, the target forest must contain a valid MEU account with the set of AD attributes described in this section. These attributes are synchronized by the PrepareMoveRequest script. There are certain mandatory attributes that should be present on the target mail user for New-MoveRequest to run properly. These attributes are always set by the PrepareMoveRequest script, either as they are taken from the source MBX, or as determined by the script. The attributes are listed here http://technet.microsoft.com/en-us/library/ee861103.aspx. Process Overview: Run PrepareMoveRequest script first and then ADMT Prepare MEU To create the target mail enabled user account in an Exchange 2010 forest from the source mailbox enabled account in the source Exchange forest, the PrepareMoveRequest script must be executed in the target Exchange 2010 forest. The script pulls the mailbox enabled account attributes from the source forest. The script can be used to provision one target MEU account at a time, but can also take data that is passed by pipeline as input to provision MEUs in bulk. Since PrepareMoveRequest script relies on Update-Recipient task that exists only in Exchange Management Shell, all the below commands need to be run in Exchange Management Shell. Running in PowerShell will only result in error. Run the below commands in the target forest $Local = Get-Credential Input the target forest's Administrator Credentials in "Domain\User" and Password format. Note: The account used should have permissions to call Update-Recipient which is available only to Exchange Enterprise Admin. $Remote = Get-Credential Input the Source forest's Administrator Credentials in "Domain\User" and Password format. Note: Since the PrepareMoveRequest script will also update the source object's proxyAddresses to include the target object's legacyDN as X500 address, the account used to run this command should have Read and Write access for the source forest. Run the PrepareMoveRequest script in the target forest [PS] C:\>.\Prepare-MoveRequest.Ps1 -Identity "DN of a user from SourceForest" -RemoteForestDomainController "FQDN of Source DC" -RemoteForestCredential $Remote -LocalForestDomainController "FQDN of Target Forest DC" -LocalForestCredential $Local -TargetMailUserOU "Distinguished name of OU in TargetForest" -UseLocalObject Note 1: You can use the -Verbose flag to check which attributes have been set if you want to get a detailed list of the attributes that were touched. Note 2: You can use the -UseLocalObject parameter here. If the local matching object is found, then the local object will be used. Note: If the local matching object is found and UseLocalObject is not defined, the script will throw an error. If the local object doesn't exist, even if UseLocalObject is specified, the script will still create a new one. If you are sure that you didn't prepare local object before, you could remove this parameter to ensure accidental overriding. In the target forest, we get a new disabled mail-enabled user AD object created with some of the following Exchange attributes: legacyExchangeDN, mail, mailnickname, msExchmailboxGuid, proxyAddresses, X500, targetAddress, userAccountControl, userprincipalName SIDHistory is empty. This is expected because Exchange doesn't migrate SIDs. At this point all of the required attributes to perform a mailbox move have been synced into the target forest. Run ADMT in the target forest. Note: Currently the Active Directory Migration Tool (ADMT) v3.1 is not supported on Windows 2008 R2 Servers. If you plan to use ADMT v3.1, it must be installed on Windows 2008 server. Check the results in the target forest: The user should now have SIDHistory matching the objectSid of the source object (all other attributes are left untouched) Gotchas running ADMT first and then PrepareMoveRequest script: Currently, several customers are running ADMT first and then running the PrepareMoveRequest script. When a user is created via ADMT, the PrepareMoveRequest script doesn't work since there are no proxyAddresses for the script to match the source forest user with the target forest user. The recommended approach is to copy at least 1 proxy address using ADMT. However, if you use the -UseLocalObject parameter, the script will only copy the 3 mandatory parameters (msExchMailboxGUID, msExchArchiveGUID, msExchArchiveName). This is not very useful. Customers can simply copy these 3 themselves. Important Note: In SP1, we are adding the OverwriteLocalObject parameter. This is designed for the ADMT case. ADMT can copy the SIDhistory, password, and proxyAddresses, and the PrepareMoveRequest script can sync the other email attributes. In this case, it will copy attributes from source to target, so it's the opposite of UseLocalObject. ADMT and Exchange Attributes ADMT transfers Exchange attributes (e.g. homeMDB, homeMTA, showInAddressBook, msExch*) which make the target user look like a legacy mailbox in the target domain. This leaves the target account in an invalid state (e.g. homeMDB still points to the old forest) which is unexpected for the PrepareMoveRequest.ps1 script. To prevent this, Exchange attributes are excluded from ADMT. The PrepareMoveRequest.ps1 script can identify and match existing accounts in the target forest based on their SMTP address (proxyAddresses attribute). Note: It can also do this based on the MasterAccountSid, but this is only populated for accounts in a resource forest scenario. More precisely, the script will use the existing target accounts if the following are true: The target account has a value in proxyAddresses which matches one of the proxyAddresses of the source account. The target account is a mail enabled user i.e. you can retrieve it with the Get-Recipient command. For this to succeed, it needs to have mail attributes like 'mail', 'targetAddress' etc. You need to specify the -UseLocalObject parameter in the script If all these are true, the script will copy further attributes needed (especially msExchMailboxGUID) to the target account so that the move request can process the accounts. By default, ADMT 3.1 does NOT migrate "mail", "msExchMailboxGuid" and "proxyAddresses" attributes because of security reasons. This is documented in the below article under "System attribute exclusion list" Managing Users, Groups, and User Profiles http://technet.microsoft.com/en-us/library/cc974331(WS.10).aspx Important Note: When running ADMT second after ILM due to both forests having the same schema (attributes), unexpected Exchange attributes are brought over. This can cause issues. HomeMDB for example is brought over and causes the MEU to look like a legacy mailbox, and is unusable. To resolve the problem of ADMT being run first, and leaving the user in an invalid state for the PrepareMoveRequest.ps1 script, you can create the following VB script/ADMT COM object model to exclude all Exchange attributes from being migrated by ADMT. Set O = CreateObject("ADMT.Migration"). o.SystemPropertiesToExclude = " HomeMDB,HomeMTA,showInAddressBook,msExchHomeServerName, mail, proxyAddresses, msExch*" This allows update-recipient to find the target object and match it with the source account and merge the two together. For more information, refer to the below article: You will find that several custom attributes are missing when you use ADMT to migrate users between two forests http://support.microsoft.com/kb/937537 Network Prerequisites When mailboxes are moved from one Exchange 2010 forest to another Exchange 2010 forest, the process is handled through Exchange 2010 Client Access Servers using the MRSProxy service. The only port required to be open between the forests for MRSProxy to use HTTPS traffic is port 443. This works even if the source mailboxes are on 2003 or 2007 MBX servers as long as an Exchange 2010 CAS server exists in both organizations. Note: The whole forest doesn't need to be Exchange 2010 in order to use the MRSProxy. If there is at least one Exchange 2010 CAS in the forest (with access to the Mailbox Servers and AD), it can be used as the MRS Proxy for moves from a mostly Exchange 2003 or Exchange 2007 forest. This can be called the "Remote" scenario (or the "MRSProxy" scenario). By default, MRSProxy is disabled. To start MRSProxy on the Client Access server in the remote forest, you must modify the Client Access server's Web.config file. For more information refer to http://technet.microsoft.com/en-us/library/ee732395.aspx. If CAS servers are behind the NLB, you should do this on all servers that can take the load. If the mailbox is being moved from legacy Exchange forest then the mailbox replication service will need to have the same TCP ports open that is needed for a normal local mailbox move. Listed are the TCP ports that are needed for a local mailbox move. These ports will be needed to be open both ways for mailboxes to be moved. Note: This is more of the "Remote Legacy" scenario, but it can be used between two Exchange 2010 forests as well as between one Exchange 2010 forest and one Exchange 2003/2007 forest. TABLE: Port Protocol 808 (TCP) Mailbox Replication Service uses to communicate 53 (TCP) DNS 135 (TCP) RPC End Point 389 (TCP) LDAP 3268 LDAP 1024 > (TCP) if mailbox store is not statically configured then 1024 higher ports need to be open 88 (TCP) Kerberos 445 (TCP) Microsoft-DS Service 443 (TCP) Mailbox Replication Proxy service uses port 443 to communicate with other Exchange 2010 client access server via HTTPS. Also it is necessary for servers in both forests to successfully perform name resolution using DNS. For cross forest mailbox moves via the MRSProxy service, the source and target servers use certificates to encrypt the HTTPS traffic. The CAS Servers in the source and target forests must have installed a valid certificate that has been issued by a trusted certificate authority recognized by the server in the opposite forest. Administrator Permissions In order to move mailboxes across different Exchange forests the account used to initiate the move request in the target forest and the account used to access the mailbox and directory in the source forest must have the proper permissions. The permissions that are needed for the account in the source forest depend on the type of move. Remote The account must have the privileges made available by membership in the Recipient Administrators group. Remote Legacy The migration account must have the following permissions. Exchange Server Administrators role Exchange Recipient Administrators role Destination Forest Permissions In the target Exchange 2010 organization the account used to create and manage the move request must be a member of the Organization Management or Recipient Management role groups, or have the following RBAC roles assigned either directly or through group membership: Move Mailboxes role Mail Recipients role Mail Recipient Creation role Only the Move Mailbox role is required to have access to the New-MoveRequest command. However, the Mail Recipients and Mail Recipient Creation roles may also be required to creating and managing target accounts in preparation for mailbox moves. Moving the mailbox There are two methods to move a mailbox across forests using Exchange 2010. The method used depends on the type of cross forest move. Both Remote and Remote Legacy cross forest moves can be performed from the Exchange Management Shell, but only Remote moves can be performed from the Exchange Management Console. Exchange Management Console To create a new move request for a cross forest move using Exchange Management Console (EMC), the console must have a session open to both the target and source forests at the same time using the feature Add Exchange Forest. This makes it possible to maintain a connection to an Exchange 2010 server in the source forest, and an Exchange 2010 server in the target forest. With a connection to servers in both source and target organizations via the EMC, you will be able to identify a mailbox that is to be moved from the source forest, while initiating the move request on an Exchange 2010 server in the target forest. To initiate a cross forest move with the Exchange Management Console, navigate to the Mailboxes folder in the Recipient Configuration node of the source forest, select the mailbox(es) to be moved, and then select New Remote Move Request. This starts the New Remote Move Request. Exchange Management Shell To initiate a cross forest mailbox move in the Exchange Management Shell a New-MoveRequest command must be issued with Remote* parameters. Move requests issued without Remote* parameters are local moves within the same Exchange forest. The New-MoveRequest cmdlet requires certain attributes to be synchronized between the source MBX account and the target MEU account in order for the mailbox move to succeed. This is described in the previous steps. In the target domain, perform the move request by running the below cmdlet New-MoveRequest -Identity "Distinguished name of User in Target Forest" -RemoteLegacy -TargetDatabase "E2K10 Mailbox Database Name" -RemoteGlobalCatalog "FQDN of Source DC" -RemoteCredential $Remote -TargetDeliveryDomain "Target domain name" After the move completes, the proxyAddresses and targetAddress attributes should have changed in the target forest. If the accounts are disabled in the target forest, enable it, set a password and log into OWA and test. After Online Mailbox Move (OMM), the source object is changed from MBX to MEU and target object is changed from MEU to MBX For more information on performing cross forest moves in Exchange 2010, refer to Managing Move Requests Clean-up When the MRS completes the moving of mailbox data from the source forest to the destination forest it mailbox enables the target user account. If the user account is disabled it leaves the account disabled. The MRS mailbox disables the source account, and converts it into a MEU account with a target address that refers to the primary SMTP address of the target mailbox account. The New-MoveRequest takes the TargetDeliveryDomain parameter. This is what determines which targetAddress to stamp. MRS checks the list of proxyAddresses for one (not necessarily the primary SMTP) that matches the FQDN specified in the TargetDeliveryDomain. The MRS will stamp this address as the targetAddress on the MEU. We moved away from using the primary SMTP address because there is a need to maintain the primary STMP when moving mailboxes cross-forest since this is part of a user's identity. When the primary SMTP address is the same on both forests, mail flow becomes more difficult. If the source account is to be retired and removed from the source forest, the administrator must plan for this manual operation outside of the mailbox move operation. What's coming in Exchange 2010 SP1 As mentioned earlier, SP1, will include the PrepareMoveRequest script as part of the install. Additionally, we are fixing a couple of issues with that script: Requiring separate local and remote credentials to run the script. LegacyExchangeDN not set on the new user object after converting local contact to local user. When specifying TargetMailUserOU, we will only search OUs (instead of other object class). Common Issues The most common issues related to PrepareMoveRequest script are listed below. These are not relevant if you have deployed the customized ILM, or if you have already run PrepareMoveRequest. Not able to match source forest user with target forest user. This is mainly due to the fact the script relies on proxyAddresses to match objects, so the target forest user needs to have at least 1 proxy address that matches the source Inadequate AD permission to delete/add recipient objects. The script manipulates AD directly and invokes the Update-Recipient cmdlet at the end, so you need to have the appropriate permission to change AD and call Update-Recipient. Another thing you can check is whether the TargetMailUserOU is set correctly. The current script does not have good support for users created by ADMT. The updated PrepareMoveRequest script in SP1 will support a new parameter "OverwriteLocalObject" for users created by ADMT and it will copy attributes from the source forest user to the target user. "UseLocalObject" - This is the script logic where we assume ILM has already created the target forest MEC or MEU, and you want to keep the target forest attributes. So the script will convert the target forest MEU or MEC to the required MEU for MBX move. Finally, a few words of thanks: I had the privilege of working with several SMEs from the Product Group, Consulting, UE and Support who helped me visualize, plan, develop and complete this blog. I would like to call out Ian Liu (Program Manager) who was instrumental in sharing his vision and being accessible at all times while writing this blog. I also want to thank Daniel Talbot for his expertise on this subject and his many contributions to the blog. Other contributors that I'd like to express gratitude are Andrew Ehrensing and Huangjian Guo. Thanks to Ying Zhang, Ramon Infante, Jeff Kizner, Kweku Ako-Adjei , Ben Winzenz, Kristi Simmons, Bill Haenlin, Laura La Fleur, Nino Bilic, Jonathan Runyon and Ayla Kol for their review and feedback. Last but not the least, I'd like to thank William Rall for his innumerable thorough reviews and feedback that helped shape this blog. - Nagesh MahadevSupporting Windows Mail 8.1 in your organization
Windows 8.1 and Windows RT include a built-in email app named Windows Mail. Mail includes support for IMAP and Exchange ActiveSync (EAS) accounts. This article includes some key technical details of Windows Mail in Windows 8.1. (See Supporting Windows 8 Mail in your organization for Windows 8.0.) Use the information to help you support the use of Mail in your organization. Read this article start to finish, or jump to the topic that interests you. Use the reference links throughout the article for more information. NOTE Mail, Calendar, and People apps run on Windows 8.1 and Windows RT. Although this article discusses the Mail app, please note that much of the information in this article also applies to the Calendar, and People apps. When connected to a server that supports Exchange ActiveSync, the Calendar, and People apps may also display data that was downloaded over the Exchange ActiveSync connection. Protocol Support Mail lets users connect to any service provider that supports either of the following two protocols: Protocol Protocol versions & standards Functionality Exchange ActiveSync (EAS) EAS 2.5 EAS 12.0 EAS 12.1 EAS 14.0 EAS 14.1 Send and receive email Sync email, contacts & calendar ActiveSync Policies Remote Wipe IMAP + SMTP IMAP4 rev1 RFC 3501 SMTP RFC 5321 IMAP4 IDLE command (push email) RFC 2177 IMAP LIST Extension for Special Folders RFC 6154 XLIST extension of the LIST command (identifies special folders). Send and receive email only Contacts and calendar data not synchronized Microsoft Exchange does not support Public Folders via IMAP. See IMAP support in Exchange 2013. Post Office Protocol (POP) is not supported. NOTE All Windows Communications apps (Mail, Calendar, and People) can use the data that is synchronized using Exchange ActiveSync. After a user connects to their account in the Mail app, their contacts and calendar data is available in the other Windows Communications Apps and vice versa. Sync Configuration Mail can be configured to synchronize data at different times as follows: Push email (default) Polling at fixed intervals Manually If a push email connection can’t be established, it will automatically switch to poll at fixed intervals. Push Email Push email requires that accounts are either Exchange ActiveSync (which all support Push) or IMAP with the IDLE extension. Not all IMAP servers support IDLE, and it is supported only for the Inbox folder. When a push connection can’t be established, Mail will change to polling on 30 minute intervals. Push email on Exchange ActiveSync requires that HTTP connections must be maintained for up to 60 minutes, and IMAP IDLE requires TCP connections to be maintained for up to 30 minutes. Account Setup Features Windows 8.1 and Windows RT users can add email accounts to Mail using the Settings charm. The Settings charm is always available on the right side of the Windows 8.1 and Windows RT screen. (For more visual details about Charms & the Windows 8.1 user interface, see Search, share, print & more.) NOTE This section provides an overview of account setup in Mail. For step-by-step procedures for setting up an account, see What else do I need to know? at the end of this guide. To make it as easy as possible to add accounts, account setup only prompts the user to enter the email address and password for the account they want to set up. From that data, Mail attempts to automatically configure the account as follows: The domain portion of the email address is matched against a database of well-known service providers (such as Outlook.com). If it’s a match, its settings are automatically configured. The domain portion of the email address is used to discover the user's email settings using the Autodiscover. If automatic configuration fails, the user is prompted for additional details such as an email server name and domain name. Add an Exchange ActiveSync account Figure 1: Exchange ActiveSync (EAS) configuration in Windows Mail If automatic configuration fails, the following additional information is required to connect to a server via Exchange ActiveSync: Server address Domain Username Add an IMAP/SMTP account Figure 2: IMAP/SMTP configuration in Windows Mail The information required to connect to a server via IMAP/SMTP is: Email address Username Password IMAP email server IMAP SSL (if your IMAP server requires SSL encryption) IMAP port SMTP email server SMTP SSL (if your SMTP server requires SSL encryption) SMTP port Whether SMTP server requires authentication Whether SMTP uses the same credentials as IMAP (If not, user must also provide SMTP credentials) Security Features Mail provides administrators with some level of security through Exchange ActiveSync policies (Mobile Device Mailbox Policies in Exchange 2013). It doesn’t support any means of managing or securing PCs that are connected via IMAP. EAS includes support for certificate-based authentication and remote wipe. Exchange ActiveSync Policy Support Exchange ActiveSync devices can be managed using Exchange ActiveSync policies. Mail supports the following EAS policies. : Password required Allow simple password Minimum password length (to a maximum of 8 characters) Number of complex characters in password (to a maximum of 2 characters) Password history Password expiration Device encryption required (on Windows RT and editions of Windows that support BitLocker. See What's New in BitLocker for details about BitLocker improvements in Windows 8.1.) Maximum number of failed attempts to unlock device Maximum time of inactivity before locking Important If AllowNonProvisionableDevices is set to false in an EAS policy and the policy contains settings that are not part of this list, the device won’t be able to connect to the Exchange server. Getting into Compliance Most of the policies listed above can be automatically enabled by Mail, but there are certain cases where the user has to take action first. These are: Server requires device encryption: User has a device that supports BitLocker but BitLocker isn’t enabled. User must manually enable BitLocker. User has a Windows RT device that supports device encryption but it is suspended. User must reboot. User has a Windows RT device that supports device encryption, but it isn’t enabled. User must sign into Windows with a Microsoft account. An admin on this PC doesn’t have a strong password: All admin accounts must have a strong password before continuing. The user’s account doesn’t have a strong password: User must set a strong password before continuing. Windows 8 Picture Passwords and ActiveSync Policy If a Windows 8.x user uses a picture password and Exchange ActiveSync policy requires a password, the user will still need to create and enter a password in accordance with the policy. ActiveSync Policy v/s Group Policy on domain-joined Windows 8.1 devices If a Windows 8.1 PC is joined to an Active Directory domain and controlled by Group Policy, there may be conflicting policy settings between Group Policy and an Exchange ActiveSync policy. In the event of any conflict, the strictest rule in either policy takes precedence. The only exception is password complexity rules for domain accounts. Group policy rules for password complexity (length, expiry, history, number of complex characters) take precedence over Exchange ActiveSync policies – even if group policy rules for password complexity are less strict than Exchange ActiveSync rules, the domain account will be deemed in compliance with Exchange ActiveSync policy. Certificate-Based Authentication Communications applications can connect to a corporate Exchange service configured to require certificate-based authentication. User authentication certificates can be provisioned to Windows 8.1 devices by administrators or end-users can browse to certificate and install to user certificate storage. User can add and connect an email account using a certificate. (For account setup, password entry is required per standard account setup.) User may be prompted to give the Mail application permission to access their user certificate, and should accept the prompt to enable certificate usage. In cases where multiple certificates are available, the user can go to account Settings to select the desired certificate. Non-PIN protected software certificates are supported. Remote Wipe Mail supports the Exchange ActiveSync remote wipe directive, but unlike Windows Phone (which deletes all data on the device), Mail scopes the data deleted to the specified Exchange ActiveSync account for which the remote wipe command is issued. The user's personal data is not deleted. Additionally, attachments saved from that account are made inaccessible. For example, if a user has an Outlook.com account for personal use and a Contoso.com account for work use, a remote wipe directive from the Contoso.com server would impact Windows 8.1 and Windows Phone 7 as follows: Data Windows Phone 7 Windows 8.1 Mail Contoso.com email Deleted Deleted Contoso.com contacts Deleted Deleted Contoso.com calendars Deleted Deleted Contoso.com attachments Deleted Not deleted, but not accessible Outlook.com email Deleted Not deleted Outlook.com contacts Deleted Not deleted Outlook.com calendars Deleted Not deleted Outlook.com attachments Deleted Not deleted Other documents, files, pictures, etc. Deleted Not deleted Account Roaming To make it as easy as possible for users to have all of their accounts set up on all of their devices, Windows 8.1 uploads vital account information to the user’s Microsoft account. This information includes email address, server, server settings, and password. When a user signs into a new PC with their Microsoft account, their email accounts are automatically set up for them. Passwords are not uploaded from a PC for any accounts which are controlled by any Exchange ActiveSync policies. Users will have to enter their password to begin syncing a policy-controlled account on a new PC. If using client certificate authentication, the client certificate, and the certificate selection for an account will not be roamed. Users will have to select their desired client certificate to begin syncing a client certificate account on a new PC. Microsoft Accounts By default, users are required to have a Microsoft account, formerly known as Windows Live ID, to use the Windows Communications apps. This will usually be the Microsoft account that the user is signed into Windows with, but if they have not done so, they will be prompted to provide one before proceeding. If the Microsoft account is… Mail will… Outlook.com or Hotmail account Automatically sync email, Calendar and Contacts using Exchange ActiveSync Not an Outlook.com or Hotmail account (for example, dave@contoso.com) Prompt the user to provide password for their email account Can my organization remove the requirement for a Microsoft account? You can apply a Group Policy to a device to make a Microsoft Account optional for the Windows Communications apps. Note, the Group Policy setting is configured in Computer Configuration node in the Group Policy and applies to all users of the computer/device to which it's applied. The policy setting lets you control whether Microsoft accounts are optional for Windows Store apps that require an account to sign in. This policy only affects Windows Store apps that support it. Windows RT devices can use Local Group Policy. To apply the Group Policy setting: Launch GPEdit by opening the “run” prompt (Windows key + r), and entering GPEdit.msc Go to Computer Configuration > Administrative Templates > Windows Components > App runtime Select Allow Microsoft accounts to be optional to configure the policy If the Group Policy is applied and a Microsoft account is not used, the Communications apps will: Prompt the user for a work account (i.e. an Exchange ActiveSync account) password If account credentials are provided, use Exchange ActiveSync to synchronize email, Contacts and Calendar from the work account A user can add additional accounts if desired. You can use corporate firewalls or other mechanisms to block access to any consumer email services as needed. The following functionality will be unavailable to a user without a Microsoft Account: Windows Store Application Installs Account Settings roaming to additional devices Connectivity to additional 3rd party services (e.g. Social sites) Email communication from Microsoft regarding any updates to Microsoft Services Agreement. Data Consumption By default, Mail only downloads one month of email (up from 2 weeks in Windows 8.0). This is user configurable and can potentially download the user’s entire mailbox. For Exchange ActiveSync accounts, all contacts are downloaded and calendar events are downloaded only for three months behind the current date and 18 months ahead. Additionally, messages can be only partially downloaded to reduce bandwidth use as follows: Content On unmetered networks On metered networks Message bodies Truncated to the first 100KB or 20KB depending on folder and device conditions Truncated to the first 20KB. For more details see Engineering Windows 8 for mobile networks. Attachments Some attachments are downloaded automatically when device conditions allow. Attachments for messages in junk folder are not downloaded automatically. Never downloaded automatically. Embedded images in email messages are downloaded on-demand as the user reads them, and attachments which are not downloaded can be downloaded on-demand as the user attempts to open them. Mail downloads all folders for an account. Users can configure the period of email which is downloaded to adjust the size of data for an account. Mail does not enforce any limits on number and size of attachments users can send. Automatic Replies Mail allows users to view and set their automatic reply messages (aka Out of Office or OOF messages). There is a visual indication when auto-reply is enabled. Users can view and set automatic reply plain text content. For corporate accounts, separate internal and external auto-reply messages are supported. There is no date/time support for specifying start or end time for automatic replies. Enterprise Connectivity Authenticated Proxies The communications applications can connect over LAN or WiFi connections via authenticated proxies which use standard authentication methods including: NTLM, Digest, Negotiate, and Basic authentication. Any user credentials entered can be cached for the session, or remembered persistently. Self-Signed Certificates The communications applications warn the user with a prompt providing an option to connect anyway when trying to connect to services with common service certificate issues. See Self-Signed Certificates in Limitations below for details and recommendations. Limitations The following features are currently not supported by Mail: Direct mailbox connections using POP: Only EAS and IMAP protocols are supported. Note This does not mean that Windows 8.1 does not support POP . This post is about the Mail app. See Using email accounts over POP on Windows 8.1 and Windows RT 8.1 for workarounds. Opaque-Signed and Encrypted S/MIME messages When S/MIME messages are received in Mail, it displays an email item with a message body that begins with “This encrypted message can’t be displayed.” To view email items in the S/MIME format, users must open the message using Outlook Web App, Microsoft Outlook, or another email program that supports S/MIME messages. For more information, see Opaque-Signed and Encrypted S/MIME Message on MSDN. Self-Signed Certificates in Windows Mail 8.1 Users may experience connectivity errors when trying to connect to an Exchange server that uses a self-signed certificate or a certificate with other common issues. The user may receive the following error message. There’s a problem with a server’s security certificate. It might not be safe to connect to the server because… <details>. You can use one of the following options to resolve this issue. To resolve issue with self-signed certificates… Use this option if… Install a certificate signed by a trusted certification authority (CA) on the server You want Exchange to work for all clients without prompting You do not want your users to ignore or bypass certificate-related errors You want to avoid installing a self-signed certificate or a certificate signed by an untrusted CA on all devices Install the server’s self-signed certificate on the device You want to save the cost of a certificate signed by a trusted CA You want Exchange to work from Windows 8.1 devices that have the self-signed certificate installed. Instruct users to ignore common certificate issues You want to avoid the cost of a CA-signed certificate or do not want to install the server’s self-signed certificate on all devices Users are knowledgeable about certificate-related errors At the prompt, users can connect anyway to ignore common service certificate issues such as self-signed certificates, allowing the communications applications to use an encrypted connection to the email service with the certificate issue. If users choose to connect anyway and ignore the service certificate issues, their selection will be remembered, (can be viewed and changed any time via Settings for account). We recommend that users select Cancel when they receive a certificate-related error and contact the administrator to fix the issue (option 1). See Digital Certificates and SSL for more information. Install a server’s self-signed certificate on the device This enables Exchange to work for Windows 8.1 devices that have the certificate installed. Note The administrator must provide a certificate file (.cer). The certificate can be installed to the trusted root certificate authority store for either of the following options: For the current user This option does not require admin rights but must be completed for each user on the device. For the local device This option requires administrator rights and needs to be done only one time for a device. The user or the system administrator can use the .cer file to install the certificate. To do this, use one of the following methods: Use the command-line At an elevated command prompt, run the following command: certutil.exe -f -addstore root.cer NOTE The command installs the certificate for all users on the device. Use the Certificate Import Wizard Double-click the certificate file. A certificate dialog opens. Click Install Certificate. A Certificate Import Wizard window opens. Select the option to install the certificate for only the current user or for the local device. Select Place all certificates in the following store Click Browse to open the store selection dialog. Select Trusted Root Certification Authorities. Select the store, and then click Ok. You are returned to Certificate Import Wizard dialog, and the certificate store and certificate to be installed into that store are displayed. Troubleshooting Mail Client Connectivity If a Mail user can't successfully connect to an account, consider the following: Verify that the user is using the latest version of the Mail app. A user can check for updates to the Mail app by doing the following: from the Start screen, go to Store > Settings > App updates > Check for updates. To rule out any transient issues, the user can wait a few minutes and try again. Some cloud-based email services (for example, Microsoft Office 365) require that the user register their account before they can use email clients such as Mail. Office 365 users register their account when they sign in to the service for the first time. If the user is not an Office 365 user, the user registers their account when they sign in to their account using their Microsoft account or sign in to Outlook Web App. The user must sign out of Outlook Web App before they try to connect using Mail again. TIP The user will see the following message if they haven't registered their account: “We couldn’t find the settings for. Provide us with more info and we’ll try connecting again.” What else do I need to know? Set up your Office 365 or Exchange-based email in Windows 8 Mail Mail app for Windows: FAQs 2784275 How to configure an Exchange account and how to troubleshoot Exchange account connectivity issues in the Mail app in Windows 8 and Windows RT 2792112 80070057 error, and Windows Phone 8 cannot sync with Microsoft Exchange 2464593 Error 85010013, 8600C2B, or 86000C29 when you try to synchronize a Windows Phone-based device to an Exchange server You may also find Building the Mail app and Right from the Start: Delivering the best email experience on any tablet with Windows 8.1 on the Building Windows 8 blog of interest. Updates 10/21/2013: Added note about Windows 8.x picture passwords and Exchange ActiveSync policies. 10/21/2013: Added link to Using email accounts over POP on Windows 8.1 and Windows RT 8.1.Single Item Recovery in Exchange Server 2010
Introduction Users often delete data from their mailboxes that they later want recovered. Recovery of these deleted items is the most common reason for IT admins to recover data from traditional point-in-time backups today. With previous versions of Exchange Server, administrators implemented two solutions to enable single item recovery, dumpster and restores. Both had their issues, unfortunately. Exchange 2010 aims to reduce the complexity and administrative costs associated with single item recovery. Terminology The following definitions may be useful for understanding the content within this article: Store Soft Delete - this is when an item has been deleted from the Deleted Items folder and placed within dumpster. Store Hard Delete - This is when an item has been marked for purging out of the store. Hard Delete via Outlook (Shift-Delete) - This is when the end user performs a shift-delete on an item. This results in the item bypassing the deleted items folder and being directly placed in dumpster. Single Item Recovery in Previous Versions of Exchange The end user single item recovery functionality was enabled through the store via the store dumpster. Administrators configured the dumpster setting on a per database or per mailbox basis. The default deleted item retention window in Exchange 2003 is 7 days, while with Exchange 2007 the default is 14 days. The end user recovery process worked typically like this (see figure 1): User receives message. User deletes the message. The message is moved to the Deleted Items folder. The user empties the deleted items folder (or an automated process like Managed Folders or Records Management deletes items out of the deleted items folder on a scheduled basis). The user then realizes he requires the previously deleted item. Using Outlook the user accesses Recover Deleted Items. Assuming the deletion timestamp of the deleted item is still within the deleted item retention period, the user has two choices: The user can recover the item. The recovered item is placed back in the Deleted Items folder. The user can purge the item from recover deleted items. This results in the deletion of the message permanently from the user's mailbox. Figure 1. Dumpster in Previous Versions If the item's deletion timestamp is beyond the deleted item retention period, then the item is not available for end user recovery. Instead, the user must call Help Desk and request recovery of the item. This involves: The user knowing when he deleted the item. The Exchange administrator locates a backup that contains the item in question. The Exchange administrator successfully restores the database to a recovery storage group. The Exchange Administrator extracts the deleted item and provides it back to the user. This of course assumes that there is a policy that allows for single item restoration and that this process isn't a really low priority for the Exchange administrator. If the backup is invalid, or if there is no backup for the time period in question, then the deleted data is unrecoverable. The other issue that needs highlighting is the fact that in previous versions of Exchange there is no way to prevent the end user from purging the data from the Recover Deleted Items view. This poses a significant legal risk for organizations that must ensure compliance requirements are met. Dumpster Changes In previous releases of Exchange Server, dumpster is essentially a view stored per folder. Items in the dumpster (henceforth known as Dumpster 1.0) stay in the folder where they were soft-deleted (shift-delete or delete from Deleted Items) and are stamped with the ptagDeletedOnFlag flag. These items are special-cased in the store to be excluded from normal Outlook views and quotas. In addition, data with this flag cannot be searched or indexed. Note: Users can perform a shift-delete and cause a message to bypass the Deleted Items folder and go straight to dumpster. Prior to Outlook 2007, the Recover Deleted Items tool, by default, only exposed the dumpster view for the Deleted Items folder. By setting the DumpsterAlwaysOn registry key (http://support.microsoft.com/kb/886205) you can enable the Recover Deleted Items tool for all folders in the mailbox and thus expose dumpster data for any folder in the Exchange 2003 or Exchange 2007 mailbox. One of the key architectural changes in Exchange 2010 is to truly enable a litigation hold experience for customers that have legal compliance requirements. As a result, Exchange 2010 must meet these requirements: Exchange has to ensure that dumpster data moves with the mailbox. Dumpster data must be indexed and discoverable. Dumpster must have a quota. Exchange has to prevent purging of data from dumpster. Exchange has to track edits of certain content. Dumpster should be per mailbox and not per folder. In order to facilitate these requirements, Dumpster was re-architected. Unlike Dumpster 1.0, Dumpster 2.0 is no longer simply a view. Dumpster in Exchange 2010 is implemented as a folder called the Recoverable Items and is located within the Non-IPM subtree of the user's mailbox (note that this is a hidden section of the mailbox and is not exposed to the end user through any client interface). The folder has three sub-folders: Deletions Versions Purges The Deletions folder replaces the ptagDeletedOnFlag view that was displayed when a user accessed the Recover Deleted Items tool. When a user soft deletes or performs an Outlook hard delete against an item, the item is moved to the Recoverable Items\Deletions folder. When the user accesses Outlook/OWA Recover Deleted Items, the RPC Client Access service translates the request and returns the Recoverable Items\Deletions folder view. The Versions and Purges folders will be covered in the Single Item Recovery in Exchange 2010 section. By architecting Dumpster to be a folder, three of the requirements are immediately met: Dumpster data is now indexed and discoverable. Dumpster data can now be moved with the mailbox. Dumpster data is now stored on a per-mailbox basis rather than a per folder basis. From an end-user perspective, this means that deleted data is now easier to recover because Recover Deleted Items tool will now expose deleted data across the entire mailbox. To ensure that Denial of Service attacks by placing large quantities of data into dumpster are prevented, Dumpster 2.0 has configurable quota settings. These settings can be configured per database and per mailbox: RecoverableItemsWarningQuota - Sets the soft limit that defaults to 20 GB. When the Recoverable Items folder reaches that size, the Exchange administrator will be notified via an event log alert. This alert should fire at the time the mailbox reaches the limit and once daily afterward. By default, the mailbox is not configured with this property, meaning that database-level limits are used. At this limit items will begin to be deleted from the dumpster using the first-in / first-out (FIFO) principle - essentially, the oldest items in the dumpster are deleted first. For example, consider a mailbox that has a dumpster that is 20GB in size and the user deletes an additional 50MB worth of data. In this case, the oldest 50MB worth of data is deleted to make room for the new 50MB worth of data. RecoverableItemsQuota - Sets the hard limit that defaults to 30 GB. At that limit, soft deletes will fail. The Exchange administrator will be notified via an event log alert. This alert should fire at the time the mailbox reaches the limit and once daily afterward. By default, the mailbox is not configured with this property, meaning that database-level limits are used. Note: Exchange 2010 includes capability for each mailbox to also maintain an archive mailbox as well. There is a dumpster for both the primary mailbox and the archive mailbox. Data deleted in the primary mailbox is placed in the primary mailbox dumpster, while data deleted in the archive mailbox is placed in the archive mailbox dumpster. Single Item Recovery in Exchange 2010 But how does Exchange 2010 meet the other two requirements, ensuring data is not either accidentally or maliciously purged and that versions are tracked? Exchange 2010 now includes two mechanisms to meet those requirements: Short-term preservation of data Long-term preservation of data Short-Term Preservation of Data Exchange 2010 includes the ability to ensure that data within the mailbox is preserved for a period of time. This feature can be enabled enabled on a per mailbox basis by running the following cmdlet: Set-Mailbox -SingleItemRecoveryEnabled $true Note: When enabling this feature, you will be notified that it could take up to 60 minutes for single item recovery to take effect. This is an approximation due to Active Directory replication. Be sure to evaluate your Active Directory replication topology with respect to administering recipients to understanding how Active Directory replication may impact changes in your environment. The time period by which the deleted data is maintained is based on the deleted item retention window. The default time period is 14 days in Exchange 2010 and is configurable per database or per mailbox. The following cmdlets let you alter this behavior: For the mailbox database: Set-MailboxDatabase -DeletedItemRetention For the mailbox: Set-Mailbox -RetainDeletedItemsFor Note: Regardless, of whether you have Single Item Recovery enabled, calendar items are maintained in the Recoverable Items folder structure for 120 days. Long-term data preservation via litigation hold will disable the expiration of the items. At this point you may be thinking, how is this any different than in previous versions of Exchange? With short-term preservation deleted items will still be moved into the Recoverable Items folder structure. However, the data cannot be purged until deletion timestamp is past the deleted item retention window. Even if the end user attempts to purge the data, the data is retained. Consider this example by a malicious user: User sends or receives a message that is legally incriminating. User deletes the message. The message is moved to the Deleted Items folder. The user empties the deleted items folder. The user accesses the Recover Deleted Items functionality in Outlook or OWA. The user then selects the item and deletes the item. At this point the user believes he has removed the incriminating evidence. And this is a key strength in this work flow as the end user's actions are not interrupted or prevented; in other words, the end user's work flow is not impaired with single item recovery enabled. However, the message was not purged from the mailbox store. Instead the message was moved from the Recoverable Items\Deletions folder to the Recoverable Items\Purges folder. All store hard-deleted items end up in this folder when single item recovery is enabled. The Recoverable Items\Purges folder is not visible to the end user, meaning that they do not see data retained in this folder in the Recover Deleted Items tool. When the message deletion timestamp has exceeded the deleted item retention window, Records Management will purge the item. See Figure 2 for a visual representation of this behavior. Figure 2. Dumpster 2.0 Not only does short term preservation prevent purging of data before the deleted item retention window has expired, but it also enables versioning functionality. Essentially when an item is changed, a copy-on-write is performed to preserve the original version of the item. The original item is placed in the Recoverable Items\Versions folder. This folder is not exposed to the end user. What triggers a copy-on-write? For messages and posts (IPM.Note* and IPM.Post*), copy-on-write will capture changes in the subject, body, attachments, senders/recipients, and sent/received dates. For other types of items, copy-on-write will occur for any change to the item except for moves between folders and read/unread status changes.\ Drafts will be exempt from copy-on-write to prevent excessive copies when drafts are auto-saved. The data stored in the Recoverable Items\Versions folder is indexed and discoverable by compliance officers. Long-Term Preservation of Data Customers sometimes require mechanisms by which data is maintained for longer periods of time, say indefinitely. This is typically due to litigation hold that occurs when the organization is undergoing a lawsuit. With Exchange 2010, litigation hold can be enabled via the Exchange Control Panel or by setting the property LitigationHoldEnabled via the Set-Mailbox cmdlet. Note: When enabling this feature, you will be notified that it could take up to 60 minutes for single item recovery to take effect. This is an approximation due to Active Directory replication. Be sure to evaluate your Active Directory replication topology with respect to administering recipients to understanding how Active Directory replication may impact changes in your environment. When litigation hold is enabled, records management purging of dumpster data ceases. Consider the following four cases: When the end user deletes an item from the "Deleted Items" folder or shift-deletes Outlook/OWA from any folder (soft delete), the item is moved to Recoverable Items\Deletions folder, where it can be recovered in the Outlook /OWA "Recover Deleted Items" view. If the end user purges data from the "Recover Deleted Items" view (hard delete from the Recoverable Items\Deletions folder), the item will be moved to the Recoverable Items\Purges folder. When an end user modifies an item, a copy of the original item is placed in the Recoverable Items\Versions folder based on the criteria discussed in the Short-Term Preservation of Data section. Also, when litigation hold is enabled, the FIFO deletion at warning limit is ignored . When a user's Recoverable Items folder exceeds the warning quota for recoverable items (as specified by the RecoverableItemsWarningQuota parameter), an event is logged in the Application event log of the Mailbox server. When the folder exceeds the quota for recoverable items (as specified by the RecoverableItemsQuota parameter), users won't be able to empty the Deleted Items folder or permanently delete mailbox items. Also copy-on-write won't be able to create copies of modified items. Therefore, it's critical that you monitor the Recoverable Items quotas for mailbox users placed on litigation hold. Recovering Purged Data Data that is stored in the Recoverable Items\Purges folder is not accessible or discoverable by the end user. However the data is indexed and discoverable for those that have the proper access role in the Exchange organization role. Role Based Access Control (RBAC) provides the Discovery Management role to allow secure search access to non-technical personnel, without providing elevated privileges to make any operational changes to Exchange Server configuration. Compliance officers or other Exchange administrators with the Discovery Management role can leverage the Exchange Control Panel (ECP) to perform discovery searches using an easy-to-use search interface. When a single item recovery request is received by help desk, the following actions can be taken: Either Help Desk or a member of the Discovery Management role utilizes the Exchange Control Panel to target a search against the user's mailbox to locate the data in question. The framework to perform the search allows the administrator to use Advanced Query Syntax. The recovered data is then extracted and placed in the Discovery Mailbox in a folder that bears the user's name and the date/time that the search was performed. The administrator then opens the Discovery Mailbox via OWA or Outlook and verifies that the content recovered is the content requested by the end user. The administrator then leverages the Exchange Management Shell to perform an export-mailbox against the discovery mailbox targeting the end user's mailbox. The data is then placed back into the end user's mailbox. Help Desk can close the ticket. Backups and Single Item Recovery Naturally after understanding the features included in Exchange 2010, a logical follow up question is "Do I still need backups for single item recovery?" The answer depends on your backup requirements and your capacity planning. Today many customers minimize the deleted item retention window, yet they maintain long backup retention time periods (from 14 days to several months to years). Let's consider a customer that currently maintains backups for 90 days and only retains deleted items within Exchange for 5 days. This customer is performing backup restores on a weekly basis to recover deleted items for end users. If the customer moved to Exchange 2010 they could move that process into Exchange by simply increasing their mailboxes capacity for dumpster: Users send/receive 100 messages per work day and have an average message size of 50KB Single Item Recovery is enabled and the deleted retention window is configured to be 90 days 10% of items are edited Mailbox capacity calculations 5 work days * 100 emails = 500 emails / week For Purges: 500 emails / week * 13 weeks = 6500 emails / retention period 6500 emails * 50KB ? 318MB For Versions: 500 emails / week * 13 weeks = 6500 emails / retention period 6500 emails * .1 = 650 emails 650 emails * 50KB ? 32MB Total Space Required per mailbox: 350MB By increasing each mailbox's capacity by a minimum of 350MB, backups are no longer needed for single item recovery. Single item recovery can be maintained and performed within Exchange. But let's not stop there. What if the requirement is that items must be recoverable for 1 year? Assuming the same assumptions used in the previous example with the exception that deleted item retention is now configured for 365 days, each mailbox needs an additional minimum 1.4GB of space. Ultimately, if the storage subsystem is planned and designed appropriately and mailbox resiliency features are leveraged, traditional point-in-time backups can be relegated to a disaster recovery mechanism, if they are even needed at all. Conclusion Exchange 2010 introduces the concept of single item recovery. Single Item Recovery prevents purging of data and provides versioning capability (the ability to retain the unaltered form of the item). By default this data is retained until the age of the deleted item has exceeded the deleted item retention window. In addition, Exchange 2010 enables long-term preservation of data for litigation hold scenarios by preventing the purging of data all together. The following table summarizes the behavior in Exchange 2010: Feature State Soft-deleted items kept in dumpster Modified (versions) and store hard-deleted items kept in dumpster User can purge items from dumpster MRM automatically purges items from dumpster Single item recovery disabled Yes No Yes Yes, 14 days by default and 120 days for calendar items Single item recovery enabled Yes Yes No Yes, 14 days by default and 120 days for calendar items Litigation hold enabled Yes Yes No No - Ross Smith IVAsk the Perf Guy: How big is too BIG?
We’ve seen an increasing amount of interest lately in deployment of Exchange 2013 on “large” servers. By large, I mean servers that contain significantly more CPU or memory resources than what the product was designed to utilize. I thought it might be time for a reminder of our scalability recommendations and some of the details behind those recommendations. Note that this guidance is specific to Exchange 2013 – there are many architectural differences in prior releases of the product that will impact scalability guidance. In a nutshell, we recommend not exceeding the following sizing characteristics for Exchange 2013 servers, whether single-role or multi-role (and you are running multi-role, right?). Recommended Maximum Processor Core Count 24 Recommended Maximum Memory 96 GB Note: Version 7.5 and later of the Exchange Server 2013 Role Requirements Calculator aligns with this guidance and will flag server configurations that exceed these guidelines. As we have mentioned in various places like TechNet and our Preferred Architecture, commodity-class 2U servers with 2 processor sockets are our recommended server type for deployment of Exchange 2013. The reason for this is quite simple: we utilize massive quantities of these servers for deployment in Exchange Online, and as a result this is the platform that we architect for and have the best visibility into when evaluating performance and scalability. You might now be asking the fairly obvious follow up question: what happens if I ignore this recommendation and scale up? It’s hard, if not impossible, to provide a great answer to this question, because there are so many things that could go wrong. We have certainly seen a number of issues raised through support related to scale-up deployments of Exchange in recent months. An example of this class of issue appears in the “Oversizing” section of Marc Nivens’ recent blog article on troubleshooting high CPU issues in Exchange 2013. Many of the issues we see are in some way related to concurrency and reduced throughput due to excessive contention amongst threads. This essentially means that the server is trying to do so much work (believing that it has the capability to do so given the massive amount of hardware available to it) that it is running into architectural bottlenecks and actually spending a great deal of time dealing with locks and thread scheduling instead of handling transactions associated with Exchange workloads. Because we architect and tune the product for mid-range server hardware as described above, no tuning has been done to get the most out of this larger hardware and avoid this class of issues. We have also seen some cases in which the patterns of requests being serviced by Exchange, the number of CPU cores, and the amount of physical memory deployed on the server resulted in far more time being spent in the .NET Garbage Collection process than we would expect, given our production observations and tuning of memory allocation patterns within Exchange code. In some of these cases, Microsoft support engineers may determine that the best short-term workaround is to switch one or more Exchange services from the Workstation Garbage Collection mode to Server Garbage Collection mode. This allows the .NET Garbage Collector to manage memory more efficiently but with some significant tradeoffs, like a dramatic increase in physical memory consumption. In general, each individual service that makes up the Exchange server product has been tuned as carefully as possible to be a good consumer of memory resources, and wherever possible, we utilize the Workstation Garbage Collector to avoid a dramatic and typically unnecessary increase in memory consumption. While it’s possible that adjusting a service to use Server GC rather than Workstation GC might temporarily mitigate an issue, it’s not a long-term fix that the product group recommends. When it comes to .NET Garbage Collector settings, our advice is to ensure that you are running with default settings and the only time these settings should be adjusted is with the advice and consent of Microsoft Support. As we make changes to Exchange through our normal servicing rhythm, we may change these defaults to ensure that Exchange continues to perform as efficiently as possible, and as a result, manual overrides could result in a less optimal configuration. As server and processor technology changes, you can expect that we will make adjustments to our production deployments in Exchange Online to ensure that we are getting the highest performance possible at the lowest cost for the users of our service. As a result, we anticipate updating our scalability guidance based on our experience running Exchange on these updated hardware configurations. We don’t expect these updates to be very frequent, but change to hardware configurations is absolutely a given when running a rapidly growing service. It’s a fact that many of you have various constraints on the hardware that you can deploy in your datacenters, and often those constraints are driven by a desire to reduce server count, increase server density, etc. Within those constraints, it can be very challenging to design an Exchange implementation that follows our scalability guidance and the Preferred Architecture. Keep in mind that in this case, virtualization may be a feasible option rather than a risky attempt to circumvent scalability guidance and operate extremely large Exchange servers. Virtualization of Exchange is a well understood, fairly common solution to this problem, and while it does add complexity (and therefore some additional cost and risk) to your deployment, it can also allow you to take advantage of large hardware while ensuring that Exchange gets the resources it needs to operate as effectively as possible. If you do decide to virtualize Exchange, remember to follow our sizing guidance within the Exchange virtual machines. Scale out rather than scale up (the virtual core count and memory size should not exceed the guidelines mentioned above) and try to align as closely as possible to the Preferred Architecture. When evaluating these scalability limits, it’s really most important to remember that Exchange high availability comes from staying as close to the product group’s guidance and Preferred Architecture as possible. We want you to have the very best possible experience with Exchange, and we know that the best way to achieve that is to deploy like we do. Jeff Mealiffe Principal PM Manager Office 365 Customer Experience
