I’ve recently had an unusual situation I wanted to share. A client of mine had a geographically stretched Exchange 2010 DAG cluster that crashed really bad, the original “active” servers had been lost beyond repair… luckily the databases were replicated to another location, so the data was saved. In addition the client was in between a migration from Exchange 2007 to Exchange 2010 (the 2007 servers were not effected from the disaster..).
Just for the sake of explaining a little more, the original “active” servers should have been restored with the setup.com /m:recoverserver , but due to the nature of the failure those servers and their names are gone and were no longer required. Those failed Exchange 2010 DAG member servers were completely deleted from Active Directory using ADSIEdit.
To recover the Exchange 2010 environment I’ve done a few steps, following which the Exchange DAG was online and service was restored.
Restored the DAG to the DR site (evict nodes from the cluster, modify the quorum, leverage AlternativeWitnessServer): Restore-DatabaseAvailabilityGroup
Created a new ClientAccessArray in the new AD site
Modified all databases with Set-MailboxDatabase so the new CAS array is now the RpcClientAccessServer
Made sure all databases are active within the our new site and on the correct servers with Move-ActiveMailboxDatabase
Removed the lost database copies on the lost DAG members with Remove-MailboxDatabseCopy
Forcibly removed the lost DAG members from the DAG: Remove-DatabaseAvailabilityGroupServer -ConfigurationOnly
Following the actions above, service was restored, and all was good, until we encountered an issue with users located on the Exchange 2007, they reported that they could not retrieve any free/busy information from other users (which were all located on Exchange 2010 databases).
A quick troubleshooting showed that configuration was fine (URL’s were set correctly, networking access was fine, permissions were okay etc..), so I’ve enabled the troubleshooting log on an outlook client while logged on as an 2007 user. Looking at the xxxx-xxx-AS.log (availability service logs) generated from outlook, I was able to extract the root cause:
<FreeBusyResponse><ResponseMessage ResponseClass="Error"><MessageText>Unable to find a Client Access server that can serve a request for an intraforest mailbox <Jhon Doe>;SMTP:Jhon.Doe@Contoso.com., inner exception: The server MBX2.contoso.com was not found in the topology.</MessageText><ResponseCode>ErrorServiceDiscoveryFailed</ResponseCode><DescriptiveLinkKey>0</DescriptiveLinkKey><MessageXml><ExceptionType xmlns="http://schemas.microsoft.com/exchange/services/2006/errors">Microsoft.Exchange.InfoWorker.Common.Availability.ServiceDiscoveryFailedException</ExceptionType><ExceptionCode xmlns="http://schemas.microsoft.com/exchange/services/2006/errors">5021</ExceptionCode></MessageXml></ResponseMessage><FreeBusyView><FreeBusyViewType xmlns="http://schemas.microsoft.com/exchange/services/2006/types">None</FreeBusyViewType></FreeBusyView></FreeBusyResponse>
The availability service on the Exchange 2007 server was trying to locate the users’ using its msExchHomeServerName value which pointed to a deleted server, one of the original “active” DAG members that was lost ! Looking at the attributes values of John Doe (per my example above) reviles that the values of homeMTA and msExchHomeServerName were pointing to non existing values, the homeMTA clearly shows a deleted server value, and the msExchHomeServerName points to a server that no longer exists. here’s an example of what I saw:
homeMTA points to a deleted server value – CN=Microsoft MTA\0DEL:msExchHomeServerName points to a deleted server
I’ve wrote a small PowerShell script that helps update the values for all affected users using a LDAP filter and the Get-User cmdlet from the Active Directory Module and the Set-Mailbox -ConfigurationOnly cmdlet.
Use this script on your own risk, make sure to always double check your self before running on a production environment.
Make sure you modify the LDAP filter $filter and the MBX2 per your configuration.
The conclusion from this case was very interesting to me, the scenario we had here was a “typical” cross site activation of an Exchange 2010 DAG, but due to the nature of the failure, we were forced to re-home the mailboxes as if we were using Database Portability! (excluding the actual database change of course). See the links below for more about Database Potability.
If you you have any MAC users working against your Exchange 2007 or Exchange 2010 servers, you’ve probably already solved this issue, so this is just for future reference and general knowledge.
Edit the web.config file , search for maxRequestLength
Change the value from its default 13280 to 51200
IISReset to make sure configurations take place
Notes:
maxAllowedContentLenght value is entered as Bytes, calculate 50MB Base64 encoded size: =((1024*50)*1024)*4/3
maxRequestLenght value is entered as Kilo Bytes, calculate 50MB =1024*50
This week I’ve been called to help with an Exchange server 2010 that was creating tremendous amounts of log files for a specific database, with regards to the blog post I’ve mentioned, ExMon – Exchange Server User Monitor came to the rescue, real fast.
Fired it up with an interval of 15 minutes, and located the user that is responsible for the issue, note the screen shot, sorting by “Log Bytes” the top user created 800 MB in 15 minutes !!
From this point forward it was easy to solve this issue, disabling both MAPI And Active Sync feature for the user, and detected the cause.
Exchange 2010 SP2 introduces the Hybrid Configuration Wizard which provides you with a streamlined process to configure a hybrid deployment between on-premises and Office 365 Exchange organizations. Hybrid deployments provide the seamless look and feel of a single Exchange organization and offer administrators the ability to extend the feature-rich experience and administrative control of an on-premises organization to the cloud. For more information, see Understanding the Hybrid Configuration Wizard.
Address Book Policies
Exchange 2010 SP2 introduces the address book policy object which can be assigned to a mailbox user. The ABP determines the global address list (GAL), offline address book (OAB), room list, and address lists that are visible to the mailbox user that is assigned the policy. Address book policies provide a simpler mechanism to accomplish GAL separation for the on-premises organization that needs to run disparate GALs. For more information, see Understanding Address Book Policies.
Cross-Site Silent Redirection for Outlook Web App
With Exchange 2010 SP2, you can enable a silent redirection when a Client Access server receives a client request that is better serviced by a Client Access server located in another Active Directory site. This silent redirection can also provide a single sign-on experience when forms-based authentication is enabled on each Client Access server. For more information, see Understanding Proxying and Redirection.
Mini Version of Outlook Web App (OMA is back !)
The mini version of Outlook Web App is a lightweight browser-based client, similar to the Outlook Mobile Access client in Exchange 2003. It’s designed to be used on a mobile operating system. The mini version of Outlook Web App provides users with the following basic functionality:
Access to e-mail, calendar, contacts, tasks and the global address list.
Access to e-mail subfolders.
Compose, reply to, and forward e-mail messages.
Create and edit calendar, contact, and task items.
In Exchange 2010 SP1, if you wanted to move mailboxes from on-premises to Outlook.com or to another forest, you had to enable MRSProxy on the remote Client Access server. To do this, you had to manually configure the web.config file on every Client Access server. In Exchange 2010 SP2, two parameters have been added to the New-WebServicesVirtualDirectory and Set-WebServicesVirtualDirectory cmdlets so that you don’t have to perform the manual configuration: MRSProxyEnabled and MaxMRSProxyConnections. For more information, see Start the MRSProxy Service on a Remote Client Access Server.
Mailbox Auto-Mapping
In Exchange 2010 SP1, Office Outlook 2007 and Outlook 2010 clients can automatically map to any mailbox to which a user has Full Access permissions. If a user is granted Full Access permissions to another user’s mailbox or to a shared mailbox, Outlook, through Autodiscover, automatically loads all mailboxes to which the user has full access. However, if the user has full access to a large number of mailboxes, performance issues may occur when starting Outlook. Therefore, in Exchange 2010 SP2, administrators can turn off the auto-mapping feature by setting the value of the new Automapping parameter to $false on the Add-MailboxPermission cmdlets. For more information, see Disable Outlook Auto-Mapping with Full Access Mailboxes.
Multi-Valued Custom Attributes
Exchange 2010 SP2 introduces five new multi-value custom attributes that you can use to store additional information for mail recipient objects. The ExtensionCustomAttribute1 to ExtensionCustomAttribute5 parameters can each hold up to 1,300 values. You can specify multiple values as a comma-delimited list.The following cmdlets support these new parameters:
Set-DistributionGroup
Set-DynamicDistributionGroup
Set-Mailbox
Set-MailContact
Set-MailPublicFolder
Set-RemoteMailbox
Litigation Hold
In Exchange 2010 SP2, you can’t disable or remove a mailbox that has been placed on litigation hold. To bypass this restriction, you must either remove litigation hold from the mailbox, or use the new IgnoreLegalHold switch parameter when removing or disabling the mailbox. The IgnoreLegalHold parameter has been added to the following cmdlets:
Great stuff coming with Exchange 2010 SP2, along with features already mentioned at the Exchange Team Blog – Announcing Exchange 2010 Service Pack 2 , a major schema update will support some new stuff.
Quoting Michal smith’s blog, here’s some key points to mention:
The Mail-Recipient class has now gained the Company and Department attributes.
This means that Groups (both security groups and distribution groups) and Contacts (mail contacts) can now be assigned values to the Company and Department attributes.
From a technical perspective, the Mail-Recipient class is a system auxiliary class, for both the Group and Contact classes, and all attributes present in Mail-Recipient are available in them.
The ms-Exch-Custom-Attributes class has gained 35 new custom attributes, from ms-Exch-Extension-Attribute-16 to ms-Exch-Extension-Attribute-45, and ms-Exch-Extension-Custom-Attribute-1 through ms-Exch-Extension-Custom-Attribute-5.
This means that Contacts, Groups, Users, Public Folders, Dynamic Distribution Lists, and Recipient Policies all now have a huge number of new attributes that can be assigned arbitrary values by an organization. This is welcome news to organizations who are using many or most of the current custom attributes and are wary to extend the schema themselves.
From a technical perspective, the ms-Exch-Custom-Attributes class is an auxiliary class for all the named classes above.
Many new attributes and classes were added to provide support for Address Book Policies and to enhance access to various address lists, global address lists, and offline address lists maintained by Exchange.
The master class is ms-Exch-Address-Book-Mailbox-Policy.
There are several new attributes and one new class (ms-Exch-Coexistence-Relationship) that are probably designed to support the Hybrid Coexistence Wizard and to overall simplify the process of configuring hybrid coexistence with Exchange Online.
There is a new class (ms-Exch-ActiveSync-Device-Autoblock-Threshold) and a number of new attributes that are within that class that appear to be designed to support automatic throttling of ActiveSync devices.
Update 9/Jun/2015 – Thanks to Josh Davis for the feedback, I’ve added a note about making sure to include both drives (if EDB and LOG files are separated).
Update 21/Oct/2013 – This article suggests that you cannot sustain downtime or interruption for your users while battling with deleting log files or restoring your working backup solution. If you can sustain a downtime (should be around minutes or so) the easiest method will be to enable Circular Logging on your database / storage group – see more here – http://technet.microsoft.com/en-us/library/bb331958%28v=exchg.141%29.aspx#UTL
I often get calls and questions regarding backups and Exchange Server, since ever this issue is not always working as required or as you would expect, but that’s off-topic 🙂
One of the most common stories is that without a working Exchange Server backup when you perform massive mailbox moves, transaction logs will get piled and fill up the volume or disk that they reside in. and then panic starts, “hey my databases were dismounted…” then of course the administrator realizes that the space on the log drive or volume has indeed ran out and now he needs to figure out what to delete.. and here’s where this post comes in…
So how can you delete or purge Exchange server logs without any risk ? well, in simple – you cannot, because the whole idea of restoring an Exchange or for this matter any transactional database requires you to have a first – “full” backup of the database itself and all transaction logs that were generated since the the date of the database creation date, or the last “successful” “full backup”.
Now here’s a nice method to “fake” a “full backup” or an on-demand transaction logs purge when you see you will be soon out of space, using the Exchange VSS writers and the diskshadow utility (available with Server 2008 or 2008 R2) . This procedure also “proves” that a VSS backup for your Exchange Server will work fine.
note: This method was tested on an Exchange server with Locally Attached Disks, not storage attached LUNs.
Use this method on on your risk. You should preform a “Full Backup” right after this process is done.
This example will show you how to purge the logs for a database that is located on Drive D, the log files of the databases are also located in Drive D. we will “fake backup” drive D and this will trigger the logs to be purged.
Note: If you have separated your log files and database file in different drives, or you want to include additional databases in the “backup” you must include the additional drives in the process, so in the example below, you will “Add volume e:” after “Add volume drive d:” and so on…
Open Command prompt
Launch Diskshadow
Add volume d:
(optional, add one line for each additional drive to include) Add volume X:
Begin Backup
Create
End Backup
At this step you should notice the following events in the application log indicating that the backup was indeed successful and logs will now be deleted.
Here’s some screenshots from the process:
The Diskshadow example screenshot.
ESE – Event ID 2005 – Starting a Full Shadow Copy Backup
MSexchangeIS – Exchange VSS Writer preparation.
ESE Event ID 224 – Logs are now purged 🙂
MSExchangeIS Event ID 9780 – Backup is now complete.
side note: although this example was tested against Exchange 2010, it should work just as fine with Exchange 2013 & 2007.
Since the first migrations of Exchange 2003 to Exchange 2010 we’ve seen a really annoying error within Outlook 2003, 2007 and Outlook 2010 when trying to De-commission legacy servers, specifically when moving all public folders replicas including the EFORMS REGISTRY system folder and it’s children folders. once the organizational forms ( respectively you might see a different folder name within your organizatino ) is replicated only to the Exchange 2010 – a log / error message will be created in the Sync Issues upon an Outlook Send/Receive operation:
11:56:54 Synchronizing Forms
11:56:54 Downloading from server ‘public folder server‘
11:56:54 Error synchronizing folder
11:56:54 [80004005-501-4B9-560]
11:56:54 The client operation failed.
11:56:54 Microsoft Exchange Information Store
Notice: Use this method at your own risk ! This method is for organizations that do not use Forms !
Many posts and different resolutions were recommended, my original “fix” for this issue was to not replicate the organizational forms folder to the new Exchange 2010 public folder when starting to De-commission the Exchange 2003 server, practically “leaving it behind”.
I recently handled a situation where the Exchange 2003 server was already removed, and the forms folder was already replicated to Exchange 2010, and the error was already in place. I could not use Exchange 2003 System Manager to remove the replica, Exchange Management Shell or EXFolders. You cannot really leave an empty replica list within the tools.
Open MFCMapi, click the session menu, select the logon and display store table option.
Double click public folders, expend the public root tree, expend NON_IPM_SUBTREE, expand EFORMS REGISTRY.
Locate and select the organizational forms folder.
Scroll the property list to find the PR_REPLICA_LIST entry – double click it and clear the value inside – clear means delete the values inside the property. Setting PR_REPLICA_LIST to NULL actually leaves us with an empty replica list – which “solves” this issue.
Note that when you click to apply the change of the PR_REPLICA_LIST the property list will immediately shrink, this is normal 🙂
Exit Outlook, wait and see that indeed the Sync Issues folder does not include a new log with the 80004005-501-4B9-560 error.
Use this method at your own risk ! and again – this method is for organizations that do not use Forms !
So you want to use the Calendar Repair Assistant (CRA) with Exchange 2010 SP1, you’ve ran a few powershell commands, but nothing happens ?
You’ve missed a change Exchange 2010 SP1 introduced two new settings for Set-MailboxServer related to the Calendar Repair Assistant:
-CalendarRepairWorkCycle and -CalendarRepairWorkCycleCheckpoint
These parameters work together. The CalendarRepairWorkCycle parameter specifies the time span in which all mailboxes on the specified server will be scanned by the CRA. For example, if you specify seven days for this parameter, the CRA will process all mailboxes on this server every seven days. Calendars that have inconsistencies will be flagged and repaired according to the interval specified by the CalendarRepairWorkCycleCheckpoint parameter. For example, if you specify one day for this parameter, the CRA will query every day for new mailboxes that require processing.
To have you exchange server schedule a daily repair schedule at 23:00 PM, while making sure this task runs each day (Cycle), and searches for new mailboxes to process every 12 hours (CycleCheckpoint) run the following:
Wanted to share with you a situation I’ve encountered with Exchange 2010 SP1.
The subject mail system suffered from an extensive downtime. The mailbox server role had it’s databases unavailable due to a storage outage, but as any major mail system – SMTP does not just “stop”.. and after a long 24+ hours of downtime, there were quite a few messages that were waiting to be delivered to the system’s recipients… counted ruffly around 5000+
Once the storage system issue was solved, and the mailbox databases were back up, the queue viewer showed that all 20 databases had around 100+ messages to deliver, which triggered the Exchange 2010 SP1 Store Driver throttling, more “verbose” information was also found at the Hub Transport connectivity log @ TransportRolesLogsConnectivity directory showing the exact error is:
432-4.3.2 STOREDRV; mailbox server is too busy
In a few words, throttling makes sure that a single client or a “specific issue” could effect the whole mail system, and it works in several aspects with Exchange 2010, some might be client, protocol and server role throttling.
In our case, the Mailbox Store engine throttling was triggered due the overwhelming messages per seconds, per recipient and the connection the hub transport servers was issuing to the mailbox server role.
Now, we could have “accept” this by-design behavior, but once the service was backup, it’s expected to have all queues zeroed-out, that is “where’s all the emails from today??!”
So, the goal was – let’s turn the throttling off , in regards to the Hub Transport <> Mailbox Server connections and once all queues are empty we will turn it on, sounds logical ? well, after quite a few searches I’ve noticed that the documentation is missing…
The Hub Transport throttling ( advanced ) settings are controlled in the throttling configuration on the edgetransport.exe.config file under the Bin directory, but the setting to disable the throttling all together is nowhere to be found.. the only references found were: <add key="RecipientThreadLimit" value="2" />
<add key="MaxMailboxDeliveryPerMdbConnections" value="3" />
even setting these values to ridicules numbers did not help, we still had 100+ messages waiting at each database delivery queue.
Only after a more furious search I’ve stumbled upon the following “hidden” magic setting: <add key="MailboxDeliveryThrottlingEnabled" value="False" />
Restarted the Hub Transport service, waited a few seconds and… here’s the result:
🙂
Of course, once all queues were zeroed-out, took around 5 minutes, I’ve enabled throttling to the original value “True” and restarted the Hub Transport services again.
