Missing Microsoft-Server-ActiveSync and OMA virtual directories in IIS and Active Directory with Exchange 2003

Hi Again,

I’ve encountered a situation today with an Exchange 2003 to 2010 migration, The Exchange 2003 Back-End server was missing the virtual directories in IIS, but this issue had an interesting twist … both vdirs were missing also in the Active Directory !

Missing or corrupt virtual directories with Exchange are common and can be easily solved with KB 883380 – How to reset the default virtual directories that are required to provide Outlook Web Access, Exchange ActiveSync, and Outlook Mobile Access services in Exchange Server 2003

Exchange Server setup creates each virtual directory in the AD forest configuration partition under services, microsoft Exchange, administrative groups, administration group name, server name, protocols, http, virtual server name (usually 1).

In this case, both ActiveSync and OMA virtual directories were missing from Active Directory and as a result were also missing from the System Manager MMC, so performing a Repair Setup or the Reset virtual directories method will render useless.. both will not write anything new to the AD.. we had to re-create the two virtual directories both in AD and the IIS, so using System Manager, we try to create a new virtual directory, but the ActiveSync and OMA is greyed out !

Microsoft-Server-ActiveSync and  OMA Missing From Active Directory

Creating New Virtual Directory With System Manager Grayed out

With some searching I’ve reached a solution that worked perfectly (dated back to 2007 from the Microsoft Exchange newsgroup), this will enable the options within the System Manager and allow us to re-create the virtual directories and restore order 🙂

1. Using ADSIEDIT locate the Exchange 2003 server container – services, microsoft Exchange, administrative groups, administration group name and right click the server name to open it’s properties.
2. Locate the Heuristics attribute and note the current value (just in-case…) our value in this case was 805310468.
4. Change the value to 270012416 , click apply and ok.
5. Refresh the Exchange System Manager or close and re-open it.
6. Now, locate the server name within the tree, expend it, expend protocols, expend HTTP, expend the virtual server name and right click to create a new Virtual Directory. You should now be able to recreate the Microsoft-Server-ActiveSync and OMA virtual directory.

Note – This will also reset your RPC over HTTP and other “server specific” settings that you configured on the server using the System Manager GUI. so make sure to note all configurations under the server properties page and re-enable any changes after setting the value.

Credits – http://microsoft.newsgroups.archived.at/public.exchange.setup/200702/07021815421.html

I hope this helps anyone struggling with this,

Ilantz

TCP/IP KeepAlive, Session Timeout, RPC Timeout, Exchange, Outlook and you

Update June 21th, 2016 following feedback and a (true golden) blog post by the Exchange Team – Checklist for troubleshooting Outlook connectivity in Exchange 2013 and 2016 (on-premises) I’ve updated the recommended values for the timeout settings, and shortened the article overall for better reading. Do read the post in general, and in topic – check the CAS & Load Balancer configuration paragraphs.


Hi Again,

This post will spotlight networking considerations that are mostly overlooked. I’ve gathered a few of these issues that might brought you here searching for an answer:

  • Outlook is retrieving data from the Microsoft Exchange Server
  • The connection to Microsoft Exchange is unavailable. Outlook must be online or connected to complete this action
  • Sent items are stuck in Outbox or delayed
  • Outlook freezes or stuck when sending a message
  • Event ID 3033 regarding Exchange Server ActiveSync complaining about the most recent heartbeat intervals used by clients
  • Other strange / weird issues “but PING works! / telnet to the port works great!” – my personal favorite

The mentioned issues or symptoms could take place in any network environment, thus more common in complex network setups where multiple devices are protecting / route network traffic. Some typical configurations examples could be one of the following:

  • Outlook Anywhere or RPC over HTTP is being used, servers are protected or published by ISA / TMG / UAG / F5 / Juniper or any other reverse proxy / publishing solutions
  • Exchange servers are located behind a firewall, router or other network device
  • Clients / Remote clients are located behind a firewall, router or other network device (just to be clear on that…)
  • Exchange servers are being load-balanced with an external physical / virtual appliance

If you’ve read this post up until here and got disappointed because the above does not fit your issue, I’d like to suggest reviewing other RPC troubleshooting topics that might help Troubleshooting Outlook RPC dialog boxes – revisited or Outlook RPC Dialog Box Troubleshooting

Exchange Server traditionally (2000 to 2010) used MAPI over RPC to communicate “natively”, RPC is known to be “sensitive” and that’s why Exchange Server 2013 and beyond allows only Outlook Anywhere (RPC over HTTP) connections from clients which in my opinion is a great change that will simplify future deployments.

Client<>Server connections in general remains active while data “flows” , mails are sent/received etc. but when the connection is Idle, we might have a situation that it will be terminated. Here comes the term KeepAlive – a “dummy” packet that makes sure the connection remain active while no data is flowing and idle.

Here’s my “how-to” suggestion:

  • Configure the RPC timeout on Exchange servers to make sure that components which use RPC will trigger a keep alive signal within the time frame you would expect
    reg add "HKLM\Software\Policies\Microsoft\Windows NT\RPC" -v "MinimumConnectionTimeout" -t REG_DWORD -d 120
  • Consider modifying the server TCP/IP KeepAlive to reduce the chance of “IDLE” connections being terminated – (Default is Two hours – The recommended value is 30 minutes , and no less then 15 minutes) – this controls the OS TCP behavior with idle connections, could greatly improve responsiveness and scalability – http://support.microsoft.com/kb/314053/EN-US
  • Make sure that you are aware of any router, firewall or any other network device that is placed between your clients and your servers. Once you do – note their session timeout, session TTL or session ageing setting for the relevant protocol and port! (this could be tricky, so do not treat this lightly)

The trick for success here is that timeout settings should be configured without overlapping one another while following the client access “path” – for example – Client > FW > Load Balancer > Server:

  • FW timeout TCP/IP timeout – 40 minutes
  • Load Balancer – TCP/IP timeout – 35 minutes
  • Server – TCP/IP timeout – 30 minutes

If additional network devices are placed between the server and your clients, make sure that session timeout settings continue to be configured accordingly.
With today’s security measures, network security has become much more complex. A typical corporate network will implement many different network appliances or software based solutions to secure data, restrict access, prevent attacks and unwanted traffic.
Bottom line – don’t think you are done with network considerations just because “ping works” or an email comes with a statement like “your port is now open”.

I hope this post will benefit others as this issue was and will probably remain common with Exchange and other client / server services.

Don’t get timed out 🙂
Ilantz

Additional useful links and sources of data:

50 GB of Exchange database logs are filling up my server

Hey again,

Today I wanted to share with you another field report regarding a troubleshooting case I’ve had with Exchange 2010.

A while back in 2009 I’ve re-posted a blog post from the Exchange Team Blog- Troubleshooting Exchange 2003 and 2007 Store Log/Database growth issues – it included a link to Mike Lagase‘s blog and massive troubleshooting guide on this matter.

This week I’ve been called to help with an Exchange server 2010 that was creating tremendous amounts of log files for a specific database, with regards to the blog post I’ve mentioned, ExMon – Exchange Server User Monitor came to the rescue, real fast.

Fired it up with an interval of 15 minutes, and located the user that is responsible for the issue, note the screen shot, sorting by “Log Bytes” the top user created 800 MB in 15 minutes !!

ExMon Screen Shot - Log Bytes Winner

 

 

From this point forward it was easy to solve this issue, disabling both MAPI And Active Sync feature for the user, and detected the cause.

Case closed 🙂

Update – Apr-2013

The Exchange Team has a new post with lots of additions from the original post from 2009 – Troubleshooting Rapid Growth in Databases and Transaction Log Files in Exchange Server 2007 and 2010

Solving Sync Issues Error 80004005-501-4B9-560 in Exchange 2010 RTM and SP1

Update

The current “Best Practice” is to upgrade your Exchange Server to Service Pack 2 and apply Update Rollup 3 for Exchange Server 2010 Service Pack 2 (KB2685289), as this issue has been permanently solved.

See Synchronization of an organizational forms library fails when you use Outlook in Cache mode in an Exchange Server 2010 for additional information.


Hello Everyone,

Since the first migrations of Exchange 2003 to Exchange 2010 we’ve seen a really annoying error within Outlook 2003, 2007 and Outlook 2010 when trying to De-commission  legacy servers, specifically when moving all public folders replicas including the EFORMS REGISTRY system folder and it’s children folders. once the organizational forms ( respectively you might see a different folder name within your organizatino ) is replicated only to the Exchange 2010 – a log / error message will be created in the Sync Issues upon an Outlook Send/Receive operation:

11:56:54 Synchronizing Forms
11:56:54 Downloading from server ‘public folder server
11:56:54 Error synchronizing folder
11:56:54 [80004005-501-4B9-560]
11:56:54 The client operation failed.
11:56:54 Microsoft Exchange Information Store

Notice: Use this method at your own risk ! This method is for organizations that do not use Forms !

Many posts and different resolutions were recommended, my original “fix” for this issue was to not replicate the organizational forms folder to the new Exchange 2010 public folder when starting to De-commission the Exchange 2003 server, practically “leaving it behind”.

I recently handled a situation where the Exchange 2003 server was already removed, and the forms folder was already replicated to Exchange 2010, and the error was already in place. I could not use Exchange 2003 System Manager to remove the replica, Exchange Management Shell or EXFolders. You cannot really leave an empty replica list within the tools.

MFCMapi to the rescue 🙂

  1. Open MFCMapi, click the session menu, select the logon and display store table option.
  2. Double click public folders, expend the public root tree, expend NON_IPM_SUBTREE, expand EFORMS REGISTRY.
  3. Locate and select the organizational forms folder.
  4. Scroll the property list to find the PR_REPLICA_LIST entry – double click it and clear the value inside – clear means delete the values inside the property. Setting PR_REPLICA_LIST to NULL actually leaves us with an empty replica list – which “solves” this issue.
  5. Note that when you click to apply the change of the PR_REPLICA_LIST the property list will immediately shrink, this is normal 🙂
  6. Exit Outlook, wait and see that indeed the Sync Issues folder does not include a new log with the 80004005-501-4B9-560 error.

Use this method at your own risk ! and again – this method is for organizations that do not use Forms !

Some references for you usage:

Troubleshooting: Error synchronizing folder Synchronizing Forms 80004005-501-4B9-560

Outlook synchronization error [80004005-501-4B9-560] with a Microsoft Exchange Server 2010 mailbox

“80004005-501-4B9-560” synchronization error logs are generated in the Sync Issues folder in Outlook in a Business Productivity Online Suite Dedicated environment 

Configure Static “fake” server names for RPC over HTTPS

Hi !

This came up with a request to “fake” exchange server names, which are actually old DE-commissioned servers. users are using RPC over HTTP , and the exchange profile they use had those server names as the actual mailbox server.

This issue might show up when performing cross forest migration or removing servers, while manipulating name resolving using DNS CNAME records, etc..

With Exchange 2010 and Exchange 2007 Outlook Anywhere settings are applied automatically when you enable the outlook anywhere feature on a CAS server, proxy names in the registry ( HKEY_LOCAL_MACHINESOFTWAREMicrosoftRpcRpcProxy) are automatically entered, that is all back-end  servers that were enabled for RPC over HTTP (2003) and all mailbox servers 2007 / 2010.

So by default RpcProxy will only answer for existing  mailbox servers, we want to add our own “old” , “fake” exchange server names.

Here’s how to manipulate the RpcProxy entry in the Exchange server make it stick.

Use at your own risk!

Under each CAS you will enable for Outlook Anywhere follow these steps:

  1. Configure “PeriodicPollingMinutes” to 0 , this will stop automatic settings overwrite – removing the static entries you will add later.Locate the value  PeriodicPollingMinutes, under HKEY_LOCAL_MACHINESYSTEMCurrentControlSetservicesMSExchangeServiceHostRpcHttpConfigurator
  2.  

  3. Configure the “fake” names, for example “email2.fake.com” will be the fake mailbox server we will add.
    Append “;email2.fake.com:6001-6002;email2.fake.com:6004” to the value of “ValidPorts_AutoConfig_Exchange”
    The String Value is under : HKEY_LOCAL_MACHINESOFTWAREMicrosoftRpcRpcProxyFor example:

    EX2010:6001-6002;EX2010:6004;EX2010.test.lab:6001-6002;
    EX2010.test.lab:6004;ex2k3:6001-6002;ex2k3:6004; ex2k3.test.lab:6001-6002;ex2k3.test.lab:6004;
    email2.fake.com:6001-6002;email2.fake.com:6004
  4.  

  5. Restart the services: MSExchangeServiceHost and MSExchangeProtectedServiceHost
  6.  

  7. IISReset

Done !

Now configure outlook to use the email2.fake.com server and configure Outlook Anywhere to verify it works.

How to publish Exchange 2003 and Exchange 2010 with ISA 2006

Hi,

First Step-By-Step !

This guide will show you how to configure ISA 2006 for coexistence of Exchange 2003 with Exchange 2010 remote connectivity services, including:

  • Outlook Web Access & Outlook WebApp
  • Microsoft ActiveSync
  • RPCoverHTTP – Outlook Anywhere
  • Publishing Exchange 2010 FARM – two client access servers

This guide assumes that:

  • ISA 2006 is configured to publish OWA 2003 and all additional services
  • SSL is configured for the Exchange 2003 server
  • Windows Integrated Authentication is enabled on the ActiveSync Vdir in the Exchange 2003 Back-End server ( http://support.microsoft.com/?kbid=937031 )
  • RPC-over-HTTP was working for for 2003 mailboxes, and the 2003 back-end is configured as an RPC-over-HTTP
  • The current configuration works 😉
  • This guide will not cover scenarios when exchange is directly exposed to the internet. which I personally do not recommend in generally….

Okay here we go:

  1. Configure redirection for Exchange 2003 OWA:
    Exchange 2010 will redirect a user that holds a mailbox in exchange 2003, this will be possible when the following cmdlet will be run on the Exchange 2010 Client Access server:
    Get-OwaVirtualDirectory -server cas01-2010 | Set-OwaVirtualDirectory -Exchange2003Url https://owa.ext.com/exchange
  2. Publish Exchange 2010 client access web farm with ISA 2006, OWA first:

New OWA 2010 Publishing Rule Outlook Web Access Publishing

– Notice ISA 2006 does not provide a wizard (or the new form) for OWA 2010 – for that you need TMG

– Now we need to create the Web Farm and select it as the target for the publishing rule

– Configure the web listener and authentication delegation option

– The web listener should be already configured for Form Authentication and a valid SSL certificate

– The publishing rule for the Web Farm is now complete.

– Two additional configurations are now required:

    1. Edit the new “exchange2010” Rule:
      Remove the legacy virtual directory’s – /Exchange, /Exchweb and /Public they will continue to be published to your original 2003 rule.
      Add /ecp/* as this is the new “options” applications for users, and a powerful administration web console with Exchange 2010.
    2. Edit the original OWA 2003 publishing rule and remove Microsoft-Server-ActiveSync path, we will next create ActiveSync publishing rule for Exchange 2010.

Now we have three last steps to finish our Exchange 2010 publishing:

  1. Create a new Exchange Web Client Access rule – and select ActiveSync – Repeat most of part 1 except we select ActiveSync, publish the webfarm, enter the same info, and select the same listener.
  2. Now as same for ActiveSync, we need to move the RPCoverHTTP (Outlook Anywhere) from the 2003 publishing rule to 2010 publishing rule. Delete the existing rule. Next you we will create a new publishing rule for Outlook Anywhere based on Exchange 2010.
  3. Create a new Exchange Web Client Access rule – and select Outlook Anywhere – Repeat most of part 1 except we select Outlook Anywhere, publish the webfarm, enter the same info, and select the same listener.

That’s it 🙂

if you kept up with all the requirements, all should be fine and you are now able to migrate your 2003 users to 2010 with ease, while both systems are allowed for external connectivity.

Enjoy!

More relevant links on the subject:

Upgrading Outlook Web App to Exchange 2010

Transitioning Client Access to Exchange Server 2010

RTL Plain Text emails with Outlook 2007 – fixed

Finally, a long term solution to a problem that have been annoying quite a while…

Outlook 2007 + Plain Text replies , that uses Right to Left languages , in my case Hebrew, were received reversed in order .. that is the words in the sentences were displayed literally reversed.

A quite annoying word issue actually… A workaround for this was making sure that users were sending out Rich Text (RTF) email’s. then the replies were displayed correctly.

Long story short. to solve this, request and install the following patch http://support.microsoft.com/kb/973401 –
Description of the Word 2007 hotfix package (Word-x-none.msp, Wordconv-x-none.msp): August 25, 2009

Additional configuration to align the text to the right could be done by following daniel’s petri post regarding this :

http://www.petri.co.il/correcting-email-display-direction-in-outlook.htm

This solved my issue on the spot.

Better later then never 🙂

Configure Session TTL / Timeout in Fortinet

Hey there Mobile admins..

Recently, I’ve did some troubleshooting with Fortinet and ActiveSync timeout, also known as Event ID 3030 Source: Server ActiveSync with the following being output to the Application Log on an Exchange Server 2003 and 2007.

Event Type: Warning
Event Source: Server ActiveSync
Event Category: None
Event ID: 3033
Description:
The average of the most recent [200] heartbeat intervals used by clients is less than or equal to [9]. Make sure that your firewall configuration is set to work correctly with Exchange ActiveSync and direct push technology. Specifically, make sure that your firewall is configured so that requests to Exchange ActiveSync do not expire before they have the opportunity to be processed.

Read more on the Direct Push in Technet : Understanding Direct Push , typically you will need to adjust your session TTL to no less then 12 minutes.

Fortinet  lists the official help on the subject in http://kb.fortinet.com/kb/microsites/microsite.do?cmd=displayKC&externalId=FD31862 – FD31862 – Customizing Session TTL in FortiOS 4.0 , FortiOS 4 also allows this in Per rule ! so for all those with FortiOS 3 , use the mentioned KB from Fortinet try the FortiOS CLI Reference..

Usually i set this time out to no less the 15 minutes or 900 seconds.. you’r call 🙂

-updated the link to Fortinet KB

Troubleshooting Exchange 2003 and 2007 Store Log/Database growth issues

Issues like these always come up with various customers, I’d thought it might be good to share with you all.

As posted also in the official Microsoft Exchange Team blog:

Mike Lagase has just posted a very detailed troubleshooting guide for these problems on his blog – Troubleshooting Exchange 2007 Store Log/Database growth issues

This is one of the most comprehensive collection of information on how to troubleshot those issues. Read it !