432-4.3.2 STOREDRV and Store Driver throttling

Hi again,

Wanted to share with you a situation I’ve encountered with Exchange 2010 SP1.

The subject mail system suffered from an extensive downtime. The mailbox server role had it’s databases unavailable due to a storage outage, but as any major mail system – SMTP does not just “stop”.. and after a long 24+ hours of downtime, there were quite a few messages that were waiting to be delivered to the system’s recipients… counted ruffly around 5000+

Once the storage system issue was solved, and the mailbox databases were back up, the queue viewer showed that all 20 databases had around 100+ messages to deliver, which triggered the Exchange 2010 SP1 Store Driver throttling, more “verbose” information was also found at the Hub Transport connectivity log @ TransportRolesLogsConnectivity directory showing the exact error is:

432-4.3.2 STOREDRV; mailbox server is too busy

In a few words, throttling makes sure that a single client or a “specific issue” could effect the whole mail system, and it works in several aspects with Exchange 2010, some might be client, protocol and server role throttling.

In our case, the Mailbox Store engine throttling was triggered due the overwhelming messages per seconds, per recipient and the connection the hub transport servers was issuing to the mailbox server role.

Now, we could have “accept” this by-design behavior, but once the service was backup, it’s expected to have all queues zeroed-out, that is “where’s all the emails from today??!”

So, the goal was – let’s turn the throttling off , in regards to the Hub Transport <> Mailbox Server connections and once all queues are empty we will turn it on, sounds logical ? well, after quite a few searches I’ve noticed that the documentation is missing…

The Hub Transport throttling ( advanced ) settings are controlled in the throttling configuration on the edgetransport.exe.config file under the Bin directory, but the setting to disable the throttling all together is nowhere to be found.. the only references found were:
<add key="RecipientThreadLimit" value="2" />
<add key="MaxMailboxDeliveryPerMdbConnections" value="3" />

even setting these values to ridicules numbers did not help, we still had 100+ messages waiting at each database delivery queue.

Only after a more furious search I’ve stumbled upon the following “hidden” magic setting:
<add key="MailboxDeliveryThrottlingEnabled" value="False" />

Restarted the Hub Transport service, waited a few seconds and… here’s the result:

Hub-Transport-Throttling-Disabled

🙂

Of course, once all queues were zeroed-out, took around 5 minutes, I’ve enabled throttling to the original value “True” and restarted the Hub Transport services again.

Hope this helps you out!

Links for the enthusiasts:

Exchange 2010 SP1 Store Driver throttling – Tony Redmond’s blog

Understanding Message Throttling – Technet

Understanding the EdgeTransport.exe.Config File – Technet

Microsoft iSCSI Software Target 3.3 is now available freely!

This week Microsoft has made the iSCSI Software Target available for free download !

These are obviously great news for all cluster / shared-storage-in-need people like me 🙂 read all about it at Jose Barreto’s Blog post:

Microsoft iSCSI Software Target 3.3 for Windows Server 2008 R2 available for public download

More quick links:

Enjoy !!

Office 2007 Cumulative Update for February 2011 is now released

To follow my previous blog “Watch out from latest outlook updates !”

the Exchange team has announced that the Office 2007 Cumulative Update for February 2011 is now available.

The update adds Personal Archives support in Outlook 2007.
For more details about the hotfix, see KBA 2475891: Description of the Office Outlook 2007 hotfix package (Outlook-x-none.msp): February 22, 2011. and also fixes quite a lot of issues that were presented from the original outlook December hotfix, Autodiscover issues, POP3 Authentication issues and few more..

As always ! test your hotfix !

Happy Archiving 🙂

Authentication pop ups and annoyances with Exchange 2007 / 2010 and Outlook Anywhere

Hi again,

This issue has came up too much, so I wanted to blog something short about this.

Prerequisites:

  • Update (Added June 29th 2013) – If using Exchange 2013, check out Exchange 2013 Outlook Anywhere Considerations for some additional specific Exchange 2013 issues.
  • Exchange 2007 or 2010
  • Outlook 2003 / 2007 / 2010
  • Windows XP / 7 / etc..
  • Outlook Anywhere ( RPC over HTTP ) enabled – with Basic Authentication or NTLM Authentication
  • Autodiscover – working correctly 😉

So, you’ve got it all configured, you enabled Outlook Anywhere, configured ISA 2006 / TMG / UAG to publish the Outlook Anywhere (or not), you published Autodiscover records an all is working great !

BUT ! you have this annoying user credentials pop ups, and users are going nuts ! and so do you !@ ( enough sarcasm ) it may work for a while, and then you are prompted again for user and password, or even worse – it might not work at all…

Here’s what can go wrong in bullets because we have a few different issues that might cause troubles..

  • Outlook Anywhere is configured to use NTLM authentication:
    • Solution 1 – Configure MSSTD or the Certificate Principle Name correctly (see below)
    • Solution 2 – Configure your clients local security policy, in specific – LmCompatiblilityLevel to 2 or 3
    • Solution 3 – If you try to pull NTLM with ISA / TMG / UAG, either configure “Kerberos Constrained Delegation” – check links below for the white-paper from Microsoft” or change the publishing rule to apply to “All Users” and in the Authentication Delegation tab choose the option “No delegation, but client may authenticate directly”
  • SSL Certificates issues
    • Outlook Anywhere was enabled for – mail.company.com (ExternalHostName), but you have a wildcard certificate or the certificate subject name does not match mail.company.com
    • Solution – Configure MSSTD or the Certificate Principle Name correctly (see below)
  • Outlook Anywhere continuously keep being configured automatically !%

So what’s that MSSTD or Certificate Principle Name ? well it’s a method Outlook can verify that the server you are connecting to indeed holds the correct SSL certificate subject name before sending credentials to.. well yeah it ain’t that secure.

Microsoft Exchange Proxy Settings

This setting is actually being configured automatically since Exchange 2007 and continue to be with Exchange 2010.

So here’s what you can do with it – all examples follow the Set Outlook Provider cmdlet syntax:CertPrincipalName

    • You have a wildcard certificate – Run this command:

Set-OutlookProvider EXPR -CertPrincipalName msstd:*.company.com

    • You have a differnet subject name on your SSL certificate then the ExternalHostName you configured for Outlook anywhere on your CAS server

Set-OutlookProvider EXPR -CertPrincipalName msstd:correctsubject.company.com

    • You don’t want that “only connect to proxy servers that have this principle name in their certificate” check box marked at all ! 🙂

Set-OutlookProvider EXPR -CertPrincipalName none

New feature with Exchange 2010 – The Set-OutlookProvider cmdlet now allows Outlook 2010 clients to connect exclusively through RPC over HTTP (Outlook Anywhere) before trying RPC over TCP connections when connecting over the Internet. !

This means you can control the check box “On fast network, connect using HTTP first, then connect using TCP/IP”, here’s the two options:

    • Always connect using HTTP (mark “on fast networks”) :

Set-OutlookProvider EXPR -OutlookProviderFlags:ServerExclusiveConnect

    • User TCP/IP first then HTTP (default):

Set-OutlookProvider EXPR -OutlookProviderFlags:None

This should cover it, no more pop ups and hopefully Outlook Anywhere and you will be friends again !

ilantz

Credits (or links) :

When, if and how do you modify Outlook Providers?

Set-OutlookProvider

Publishing Outlook Anywhere Using NTLM Authentication With Forefront TMG or Forefront UAG

Exchange 2013 Outlook Anywhere Considerations

Watch out from latest outlook updates !

Hey everyone,

There has been a growing concern about the issues from latest Outlook 2007 / 2010 updates:

http://blogs.office.com/b/microsoft-outlook/archive/2010/12/17/issues-with-the-recent-update-for-outlook-2007.aspx

Microsoft has released the updated http://support.microsoft.com/kb/2412171 KB, but seems not all trouble were solved, mainly the AutoDiscover issue, since http://support.microsoft.com/kb/2479671 was released followed by the “original” 2412171KB numerous issues have been identified with outlook features.

I’d advice to stay away from these updates until all is clear.

Update – Office 2007 Cumulative Update for February 2011 is now released

Ilantz

Enable or disable POP3 and IMAP4 by group membership in Exchange 2007

Hi everyone !

My good friend podlisk has finally got his blog up and running, check out the script for this highly required task for Exchange 2007 / 2010 🙂

http://podlisk.wordpress.com/2011/01/13/enable-or-disable-pop3-and-imap4-by-group-membership-in-exchange-2007

Enjoy !

Configure Static “fake” server names for RPC over HTTPS

Hi !

This came up with a request to “fake” exchange server names, which are actually old DE-commissioned servers. users are using RPC over HTTP , and the exchange profile they use had those server names as the actual mailbox server.

This issue might show up when performing cross forest migration or removing servers, while manipulating name resolving using DNS CNAME records, etc..

With Exchange 2010 and Exchange 2007 Outlook Anywhere settings are applied automatically when you enable the outlook anywhere feature on a CAS server, proxy names in the registry ( HKEY_LOCAL_MACHINESOFTWAREMicrosoftRpcRpcProxy) are automatically entered, that is all back-end  servers that were enabled for RPC over HTTP (2003) and all mailbox servers 2007 / 2010.

So by default RpcProxy will only answer for existing  mailbox servers, we want to add our own “old” , “fake” exchange server names.

Here’s how to manipulate the RpcProxy entry in the Exchange server make it stick.

Use at your own risk!

Under each CAS you will enable for Outlook Anywhere follow these steps:

  1. Configure “PeriodicPollingMinutes” to 0 , this will stop automatic settings overwrite – removing the static entries you will add later.Locate the value  PeriodicPollingMinutes, under HKEY_LOCAL_MACHINESYSTEMCurrentControlSetservicesMSExchangeServiceHostRpcHttpConfigurator

    2.  

  2. Configure the “fake” names, for example “email2.fake.com” will be the fake mailbox server we will add.
    Append “;email2.fake.com:6001-6002;email2.fake.com:6004” to the value of “ValidPorts_AutoConfig_Exchange”
    The String Value is under : HKEY_LOCAL_MACHINESOFTWAREMicrosoftRpcRpcProxyFor example:

    EX2010:6001-6002;EX2010:6004;EX2010.test.lab:6001-6002;
    EX2010.test.lab:6004;ex2k3:6001-6002;ex2k3:6004; ex2k3.test.lab:6001-6002;ex2k3.test.lab:6004;
    email2.fake.com:6001-6002;email2.fake.com:6004

    3.  

  3. Restart the services: MSExchangeServiceHost and MSExchangeProtectedServiceHost

    4.  

  4. IISReset

Done !

Now configure outlook to use the email2.fake.com server and configure Outlook Anywhere to verify it works.

FPSMC Agent Installation Error

Forefront Protection Server Managment Console 2010 was latly been released, see the FSS blog entry

So a quick install, reveled some issues with the Deplay Agent task on some servers.. failing with this error:

Failed to deploy the Agent. Could not connect to net.tcp://ex-cas.contoso.com:8816/PushInstaller. The connection attempt lasted for a time span of 00:00:21.0157595. TCP error code 10060: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 192.168.5.20:8816.

Quick workaround, just configure the Firewall State for the specific failed servers for the domain profile to off 🙂 or, configure the inbound port 8816 from the FPSMC consle server to that server..

Just a heads up for anyone who sees this.

Happy holidays !

Ilantz

Exchange Calendar Update Tool – Extract Mailboxes from Exchange 2010 fails

Every year at December, we at Israel ( and at some other points of the year, over the world.. ) have to rebase some calendar appointments..

This entry is not about daylight saving bashing 😉 but just a note to anyone that will use the Exchange Calendar Update Tool against Exchange 2010 mailboxes and servers.

I did not had enough time to actually find out why and what is the appropriate fix for this, but here’s a workaround for the error and the empty result when extracting the mailboxes from the servers..

If you will examine the logs in the msextmz extract log, when trying to search for the mailboxes on the required servers, you will notice that the output will be empty, and zero mailboxes will be reported.

needless to say that this obviously eliminates the possibility for extracting timezones from the mailboxes – i will not cover this issue, because in Israel we need to rebase the appointments just to reflect the current daylight saving durations..

Any way here’s the error:

[20-Dec-2010 12:51:56][3684]:HrProcessMailboxTable:Please log on to a profile with administrator privileges.
[20-Dec-2010 12:51:56][3684]:HrProcessMailboxTable:Unable open mailbox table for server /o=Contoso/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=EX-2010.  Error 0x80004005.
[20-Dec-2010 12:51:56][3684]:HrProcessMailboxTable:Returning Error 0x80004005

You can easily report the mailboxes from powershell using:

Get-mailbox -ResultSize:unlimited -RecipientTypeDetails usermailbox | select ServerLegacyDN, LegacyExchangeDN | Export-Csv mailboxes.csv

Then use excel to export the data and match it with the format for the update tool which should be like this:

ServerLegacyDN <TAB> LegacyExchangeDN <TAB> TimeZone

Save that to a TXT, watch the formatting and tabs ! remove all the csv hyphens,commas etc..

 

Hope this will be fixed anytime soon, or a clarification will be published..

until then, good luck !

and Happy Holidays !

ilantz

 

Some Links:

Using the Exchange Calendar Update Tool to address daylight saving time changes for Exchange Server

December 2010 DST Cumulative Update for Windows operating systems