TCP/IP KeepAlive, Session Timeout, RPC Timeout, Exchange, Outlook and you

Update June 21th, 2016 following feedback and a (true golden) blog post by the Exchange Team – Checklist for troubleshooting Outlook connectivity in Exchange 2013 and 2016 (on-premises) I’ve updated the recommended values for the timeout settings, and shortened the article overall for better reading. Do read the post in general, and in topic – check the CAS & Load Balancer configuration paragraphs.


Hi Again,

This post will spotlight networking considerations that are mostly overlooked. I’ve gathered a few of these issues that might brought you here searching for an answer:

  • Outlook is retrieving data from the Microsoft Exchange Server
  • The connection to Microsoft Exchange is unavailable. Outlook must be online or connected to complete this action
  • Sent items are stuck in Outbox or delayed
  • Outlook freezes or stuck when sending a message
  • Event ID 3033 regarding Exchange Server ActiveSync complaining about the most recent heartbeat intervals used by clients
  • Other strange / weird issues “but PING works! / telnet to the port works great!” – my personal favorite

The mentioned issues or symptoms could take place in any network environment, thus more common in complex network setups where multiple devices are protecting / route network traffic. Some typical configurations examples could be one of the following:

  • Outlook Anywhere or RPC over HTTP is being used, servers are protected or published by ISA / TMG / UAG / F5 / Juniper or any other reverse proxy / publishing solutions
  • Exchange servers are located behind a firewall, router or other network device
  • Clients / Remote clients are located behind a firewall, router or other network device (just to be clear on that…)
  • Exchange servers are being load-balanced with an external physical / virtual appliance

If you’ve read this post up until here and got disappointed because the above does not fit your issue, I’d like to suggest reviewing other RPC troubleshooting topics that might help Troubleshooting Outlook RPC dialog boxes – revisited or Outlook RPC Dialog Box Troubleshooting

Exchange Server traditionally (2000 to 2010) used MAPI over RPC to communicate “natively”, RPC is known to be “sensitive” and that’s why Exchange Server 2013 and beyond allows only Outlook Anywhere (RPC over HTTP) connections from clients which in my opinion is a great change that will simplify future deployments.

Client<>Server connections in general remains active while data “flows” , mails are sent/received etc. but when the connection is Idle, we might have a situation that it will be terminated. Here comes the term KeepAlive – a “dummy” packet that makes sure the connection remain active while no data is flowing and idle.

Here’s my “how-to” suggestion:

  • Configure the RPC timeout on Exchange servers to make sure that components which use RPC will trigger a keep alive signal within the time frame you would expect
    reg add "HKLM\Software\Policies\Microsoft\Windows NT\RPC" -v "MinimumConnectionTimeout" -t REG_DWORD -d 120
  • Consider modifying the server TCP/IP KeepAlive to reduce the chance of “IDLE” connections being terminated – (Default is Two hours – The recommended value is 30 minutes , and no less then 15 minutes) – this controls the OS TCP behavior with idle connections, could greatly improve responsiveness and scalability – http://support.microsoft.com/kb/314053/EN-US
  • Make sure that you are aware of any router, firewall or any other network device that is placed between your clients and your servers. Once you do – note their session timeout, session TTL or session ageing setting for the relevant protocol and port! (this could be tricky, so do not treat this lightly)

The trick for success here is that timeout settings should be configured without overlapping one another while following the client access “path” – for example – Client > FW > Load Balancer > Server:

  • FW timeout TCP/IP timeout – 40 minutes
  • Load Balancer – TCP/IP timeout – 35 minutes
  • Server – TCP/IP timeout – 30 minutes

If additional network devices are placed between the server and your clients, make sure that session timeout settings continue to be configured accordingly.
With today’s security measures, network security has become much more complex. A typical corporate network will implement many different network appliances or software based solutions to secure data, restrict access, prevent attacks and unwanted traffic.
Bottom line – don’t think you are done with network considerations just because “ping works” or an email comes with a statement like “your port is now open”.

I hope this post will benefit others as this issue was and will probably remain common with Exchange and other client / server services.

Don’t get timed out 🙂
Ilantz

Additional useful links and sources of data:

Windows 8 Wakes Up From Sleep or Hibernation Unexpectedly

Hi Again,

I’ve upgraded my desktop to windows 8 lately and since the upgrade I’ve noticed that each time the computer enters sleep mode or hibernation it keeps turning on my itself mysteriously and no apparent reason.

Well…no more!! Here’s the actual line of events that led me to the solution:

  1. Went through some event viewer entries, specifically looking at Power-Troubleshooter and Kerner-General source that did not provide me with anything…
    Event ID 1 Source Power-Troubleshooter Wake Source Unknown
  2. Double checked that no one is touching the mouse or keyboard… 🙂
  3. Made sure that the “Allow wake timers” option is not enabled for the active power scheme
    Allow Wake Timers Set To Disabled
  4. Disabled the “Allow this device to wake up the computer” option on the network card adapter Power Management settings tab – you can query all devices that are allowed using the following command (cmd not PowerShell):
    powercfg -devicequery wake_armedAllow This Device To Wake The Computer Disabled

Only after being frustrated again from the computer still waking up with no apparent reason I’ve noticed that it keeps waking up at around specific times, which led me to the conclusion that it’s probably a schedule task that was waking the computer up !
Seems like there is a Media Center task names mcupdate_scheduled that was causing all the trouble !

Wake The Computer To Run This Task Enabled

So, I’ve written a small PowerShell script to disable the “wake the computer to run this task” option from all scheduled tasks at once, and that did the trick!
This script should work fine with Windows 8 or Server 2012 and might serve as an example for manipulating scheduled tasks with PowerShell.

Get-ScheduledTask | ? { $_.Settings.WakeToRun -eq $true -and $_.State -ne "Disabled"} | % { $_.Settings.WakeToRun = $false; Set-ScheduledTask $_ }

Now my computer sleeps and hibernates without waking up ! ZzzzzzzZzzzzZzzzz

Ilantz

Additional Links:

http://superuser.com/questions/503786/windows-8-desktop-wakes-up-immediately-after-sleep-due-to-keyboard-mouse/522628

http://www.howtogeek.com/127818/how-to-stop-windows-8-waking-up-your-pc-to-run-maintenance/

Exchange RBAC Knowledge base

Hi Again, due to popular demand, here is my small RBAC “knowledge base”, this should apply just fine to Exchange 2010 and Exchange 2013.

Fast and Furious how-to and cool examples:

http://www.mikepfeiffer.net/2010/11/7-useful-one-liners-when-managing-rbac-in-exchange-2010/

http://blogs.technet.com/b/heyscriptingguy/archive/2012/01/13/use-powershell-and-rbac-to-control-access-to-exchange-server-cmdlets.aspx

http://www.opsvault.com/how-to-create-custom-recipient-management-groups-using-exchange-2010-rbac/

http://rbac.codeplex.com/ – a free GUI tool for managing RBAC

Learn and Understand RBAC:

http://blogs.technet.com/b/exchange/archive/2009/11/16/3408825.aspx – RBAC and the Triangle of Power

http://technet.microsoft.com/en-us/library/dd298183.aspx – Understanding Role Based Access Control

http://help.outlook.com/en-us/140/dd207272.aspx – Built-in RBAC Roles for Exchange Online

Hope you find this information useful !

ilantz

Office Client and Office Server 2013 product line was released!

Hello Everyone,

A huge release by Microsoft, all client and server office products are out ! Login to your Technet / MSDN Subscriptions and start downloading 🙂

Lync Server 2013

Exchange Server 2013

Office Web Apps 2013

Office Professional Plus 2013 (x64 and x86)

SharePoint Server 2013

Visio Professional (x64 and x86)

Project Professional 2013 (x64 and x86)

Enjoy !

ilantz

Exchange 2010 DAG failover with lost members and homeMTA and msExchHomeServerName values

Hi Again,

I’ve recently had an unusual situation I wanted to share. A client of mine had a geographically stretched Exchange 2010 DAG cluster that crashed really bad, the original “active” servers had been lost beyond repair… luckily the databases were replicated to another location, so the data was saved. In addition the client was in between a migration from Exchange 2007 to Exchange 2010 (the 2007 servers were not effected from the disaster..).

Just for the sake of explaining a little more, the original “active” servers should have been restored with the setup.com /m:recoverserver , but due to the nature of the failure those servers and their names are gone and were no longer required. Those failed Exchange 2010 DAG member servers were completely deleted from Active Directory using ADSIEdit.

To recover the Exchange 2010 environment I’ve done a few steps, following which the Exchange DAG was online and service was restored.

  1. Restored the DAG to the DR site (evict nodes from the cluster, modify the quorum, leverage AlternativeWitnessServer): Restore-DatabaseAvailabilityGroup
  2. Created a new ClientAccessArray in the new AD site
  3. Modified all databases with Set-MailboxDatabase so the new CAS array is now the RpcClientAccessServer
  4. Made sure all databases are active within the our new site and on the correct servers with Move-ActiveMailboxDatabase
  5. Removed the lost database copies on the lost DAG members with Remove-MailboxDatabseCopy
  6. Forcibly removed the lost DAG members from the DAG: Remove-DatabaseAvailabilityGroupServer -ConfigurationOnly

Following the actions above, service was restored, and all was good, until we encountered an issue with users located on the Exchange 2007, they reported that they could not retrieve any free/busy information from other users (which were all located on Exchange 2010 databases).

A quick troubleshooting showed that configuration was fine (URL’s were set correctly, networking access was fine, permissions were okay etc..), so I’ve enabled the troubleshooting log on an outlook client while logged on as an 2007 user. Looking at the xxxx-xxx-AS.log (availability service logs) generated from outlook, I was able to extract the root cause:

<FreeBusyResponse><ResponseMessage ResponseClass="Error"><MessageText>Unable to find a Client Access server that can serve a request for an intraforest mailbox <Jhon Doe>;SMTP:Jhon.Doe@Contoso.com., inner exception: The server MBX2.contoso.com was not found in the topology.</MessageText><ResponseCode>ErrorServiceDiscoveryFailed</ResponseCode><DescriptiveLinkKey>0</DescriptiveLinkKey><MessageXml><ExceptionType xmlns="http://schemas.microsoft.com/exchange/services/2006/errors">Microsoft.Exchange.InfoWorker.Common.Availability.ServiceDiscoveryFailedException</ExceptionType><ExceptionCode xmlns="http://schemas.microsoft.com/exchange/services/2006/errors">5021</ExceptionCode></MessageXml></ResponseMessage><FreeBusyView><FreeBusyViewType xmlns="http://schemas.microsoft.com/exchange/services/2006/types">None</FreeBusyViewType></FreeBusyView></FreeBusyResponse>

The availability service on the Exchange 2007 server was trying to locate the users’ using its msExchHomeServerName value which pointed to a deleted server, one of the original “active” DAG members that was lost !  Looking at the attributes values of John Doe (per my example above) reviles that the values of homeMTA and msExchHomeServerName were pointing to non existing values, the homeMTA clearly shows a deleted server value, and the msExchHomeServerName points to a server that no longer exists. here’s an example of what I saw:

homeMTA points to a deleted server value - CN=Microsoft MTA\0DEL:
homeMTA points to a deleted server value – CN=Microsoft MTA\0DEL:
msExchHomeServerName points to a deleted server
msExchHomeServerName points to a deleted server

I’ve wrote a small PowerShell script that helps update the values for all affected users using a LDAP filter and the Get-User cmdlet from the Active Directory Module and the Set-Mailbox -ConfigurationOnly cmdlet.

Use this script on your own risk, make sure to always double check your self before running on a production environment.

$filter = "(&(objectCategory=user)(objectClass=user)(msExchHomeServerName=/o=Contoso/ou=Exchange\20Administrative\20Group\20\28FYDIBOHF23SPDLT\29/cn=Configuration/cn=Servers/cn=MBX2*))"
$strAttributes = "msExchHomeServerName, homeMTA, homeMDB"
$users = get-ADUser -LDAPFilter "$filter" -ResultSetSize $null -properties $strAttributes
foreach ($user in $users)
{
$mbx = $null;
$mbx = get-mailbox -Identity $user.DistinguishedName;
write-host "working on user" $user.name
write-host "working on mailbox" $mbx.name
set-mailbox $mbx -Database $mbx.database -confirm:$false -force -verbose
}

Make sure you modify the LDAP filter $filter and the MBX2 per your configuration.

The conclusion from this case was very interesting to me, the scenario we had here was a “typical” cross site activation of an Exchange 2010 DAG, but due to the nature of the failure, we were forced to re-home the mailboxes as if we were using Database Portability! (excluding the actual database change of course). See the links below for more about Database Potability.

Hope you find this information useful,
Ilantz

Datacenter Switchovers

Move a Mailbox Database Using Database Portability

Mind blowing features coming with Windows Server 2012 Hyper-v

Hey Again,
It seems like next year will keep me busy with visualization projects based on Hyper-V 🙂
Many promising features are coming up with the next Windows Server version and I’m already excited !

Just take a look at tables below for a comparison between 2008 R2 and 2012 RC Hyper-V:

Processor and Memory Support

Processor/Memory Feature Windows Server 2008 R2 Windows Server  2012 Release Candidate
Logical processors on hardware 64 320
Physical memory 1 TB 4 TB
Virtual processors per host 512 2,048
Virtual processors per virtual machine 4 64
Memory per virtual machine 64 GB 1 TB
Active virtual machines 384 1,024
Maximum Cluster Nodes 16 64
Maximum Cluster Virtual machines 1,000 4,000

Network

Network Feature Windows Server 2008 R2 Windows Server  2012 Release Candidate
NIC Teaming Yes, through partners Yes, Windows NIC teaming in box
VLAN Tagging Yes Yes
MAC Spoofing Protection Yes, with R2 SP1 Yes
ARP Spoofing Protection Yes, with R2 SP1 Yes
SR-IOV Networking No Yes
Network QoS No Yes
Network Metering No Yes
Network Monitor Modes No Yes
Ipsec Task Offload No Yes
VM Trunk Mode No Yes

Storage

Storage Feature Windows Server 2008 R2 Windows Server  2012 Release Candidate
Live storage migration No, quick storage migration through System

Center Virtual Machine Manager

Yes, with no limits (as many as the hardware will allow)
Virtual machines on file storage No Yes, Server Message Block 3.0 (SMB3)
Guest Fibre Channel No Yes
Virtual disk format VHD up to 2 TB VHD up to 2 TB VHDX up to 64 TB
Virtual machine guest clustering Yes, through iSCSI Yes, through iSCSI, Fibre Channel, or Fibre

Channel over Ethernet (FCoE)

Native 4 KB disk support No Yes
Live virtual hard disk merge No, offline Yes
Live new parent No Yes
Secure offloaded data transfer No Yes

Manageability

Manageability Feature Windows Server 2008 R2 Windows Server  2012 Release Candidate
Hyper-V PowerShell No Yes
Network PowerShell No Yes
Storage PowerShell No Yes
REST APIs No Yes
SCONFIG Yes Yes
Enable/Disable shell No, server core at operating system setup Yes
VMConnect support for RemoteFX No Yes

Additional links for your reading:

Hyper-V Comparison Guide

Competitive Advantages of Windows Server 2012 RC Hyper-V

Windows Server 2012 Release Candidate Server Virtualization

Windows Server 2012 Release Candidate

Adjust allowed attachment size for EWS

Hey again,

If you you have any MAC users working against your Exchange 2007 or Exchange 2010 servers, you’ve probably already solved this issue, so this is just for future reference and general knowledge.

Following the Microsoft reference on the subject: Set Message Size Limits for Exchange Web Services the below example is for Exchange 2007

  1. Configure the application to receive requests 50 MB:
    1. Open CMD
    2. %windir%system32inetsrvappcmd set config “Default Web SiteEWS” -section:requestFiltering -requestLimits.maxAllowedContentLength:69905067
  2. Edit web.config to allow 50 MB requests:
    1. Backup %ProgramFiles%MicrosoftExchange ServerV14ClientAccessexchwebewsweb.config
    2. Edit the web.config file , search for maxRequestLength
    3. Change the value from its default 13280 to 51200
  3. IISReset to make sure configurations take place

Notes:

maxAllowedContentLenght value is entered as Bytes, calculate 50MB Base64 encoded size: =((1024*50)*1024)*4/3
maxRequestLenght value is entered as Kilo Bytes, calculate 50MB =1024*50

Hope this post  helped you

ilantz

50 GB of Exchange database logs are filling up my server

Hey again,

Today I wanted to share with you another field report regarding a troubleshooting case I’ve had with Exchange 2010.

A while back in 2009 I’ve re-posted a blog post from the Exchange Team Blog- Troubleshooting Exchange 2003 and 2007 Store Log/Database growth issues – it included a link to Mike Lagase‘s blog and massive troubleshooting guide on this matter.

This week I’ve been called to help with an Exchange server 2010 that was creating tremendous amounts of log files for a specific database, with regards to the blog post I’ve mentioned, ExMon – Exchange Server User Monitor came to the rescue, real fast.

Fired it up with an interval of 15 minutes, and located the user that is responsible for the issue, note the screen shot, sorting by “Log Bytes” the top user created 800 MB in 15 minutes !!

ExMon Screen Shot - Log Bytes Winner

 

 

From this point forward it was easy to solve this issue, disabling both MAPI And Active Sync feature for the user, and detected the cause.

Case closed 🙂

Update – Apr-2013

The Exchange Team has a new post with lots of additions from the original post from 2009 – Troubleshooting Rapid Growth in Databases and Transaction Log Files in Exchange Server 2007 and 2010

Cluster Creation Wizard fails with 0x80090322 error

Hi again,

Today I’ve got a call from a client regarding a new Hyper-V 2008 R2 SP1 cluster, the create cluster wizard kept failing during the forming cluster step with a timeout..

Well, troubleshooting..

  1. Running Cluster LOG /gen
  2. Doing some reading…
  3. Locating the step and the error:

    [NODE] Node 1: New join with node2: stage: ‘Authenticate Initial Connection’ status HrError(0x80090322) reason: ‘[SV] Authentication failed’
    DBG [CHANNEL 172.16.1.2:~3343~] Close().
    WARN cxl::ConnectWorker::operator (): HrError(0x80090322)’ because of ‘[SV] Authentication or Authorization Failed’

  4. Looking up 0x80090322 with ERR.EXE means SEC_E_WRONG_PRINCIPAL
  5. Lazy me , doing google for 0x80090322 cluster authentication failed
  6. Read “2 node cluster windows 2008 R2 cluster won’t form”
  7. Delete two users that were created to manage the servers.. node1 and node2 🙂
  8. Run create cluster wizard again – SUCCESS !
  9. Retrospective understand the error… SEC_E_WRONG_PRINCIPAL seems like the wizard can’t tell the difference between the user accounts and the computer accounts. (“By Design”)

Nice ! I was shocked from the solution.. but hey.. it worked instantly.