Category: Exchange 2007

Exchange upgrade fails due to missing language files

Just to help anyone out there that might be facing this issue. I’ve helped troubleshoot an Exchange 2010 RTM upgrade to Exchange 2010 SP3 which kept failing due to missing language files…

Event ID 1603 was also thrown as per to the KB 2784788 – “1635” or “1603” error code when you install update rollups or service packs for Exchange Server 2007 or Exchange Server 2010

The MSILOG indeed showed that the setup was looking for the RTM language files in the original location where the setup files were, but they are long gone… with the RTM DVD no where to be-found (RTM trial files + the oldest Language Pack bundle are in a non compatible version) this situation was doomed to failure.

So, I’ve turned to manually remove any references to the Client / Server language packs on the server, this included removing a whole bunch of registry keys:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v14\Language Packs\ <-- the whole KEY
HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Installer\Products\  <-- Whatever "Microsoft Exchange ** Language Pack" I found
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\  <-- Whatever "Microsoft Exchange ** Language Pack" I found
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData\S-1-5-18\Products  <-- Whatever "Microsoft Exchange ** Language Pack" I found

Following this brutal way, I’ve stumbled upon a way to Applying Small Updates by Reinstalling the Product this actually achieves what the installer wants:

msiexec /i Server<or>ClientLanguagePack.msi REINSTALLMODE=vomus

And it works ! Now, I guess that with a script this would have been much quicker then the registry method, but at least now I’m (and you are) aware of this workaround , and here’s the script for your usage:

** edit the $setuplocation variable for your directory of the servicepack.

$setupLocation = "c:\sp3"
$allDirs = dir $setupLocation -Directory
foreach ($dir in $allDirs)
{
if (Test-Path ($dir.FullName + "\clientlanguagepack.msi")) {Write-Host "Installing" $dir.name ; Start-Process -FilePath msiexec -ArgumentList /i, ($dir.FullName + "\clientlanguagepack.msi"), "REINSTALLMODE=vomus" -Wait }
if (Test-Path ($dir.FullName + "\serverlanguagepack.msi")) {Write-Host "Installing" $dir.name ; Start-Process -FilePath msiexec -ArgumentList /i, ($dir.FullName + "\serverlanguagepack.msi"), "REINSTALLMODE=vomus" -Wait }
}

ilantz

Additional references:

Upgrading Service pack – keep asking for language pack

http://stackoverflow.com/a/7916340 – credit for the REINSTALLMODE=vomus trick

How to restore the missing Windows Installer cache files and resolve problems that occur during a SQL Server update – kb 969052

Microsoft Exchange Web Services Managed API 1.1 Download

Hi All,

Following a workaround mentioned on http://support.microsoft.com/kb/2512023 – “GetUserOofSettings”, “SetUserOofSettings” and “GetUserAvailability” operations do not support Exchange Impersonation on the Exchange Server 2010 SP1 schema

Seems like the EWS Managed API 1.1 download link does not work anymore, and it’s virtually impossible to get a hold of the files on the WEB, so I’ve uploaded the redistributable package here for anyone looking around for it.

Anyway – grab it here: Microsoft Exchange Web Services Managed API 1.1

Hope this helps.

ilantz

Exchange Web Service Managed API 1.1, Download – Microsoft ( Non working download links) :
www.microsoft.com/en-us/download/details.aspx?id=13480
http://www.microsoft.com/downloads/en/details.aspx?FamilyID=c3342fb3-fbcc-4127-becf-872c746840e1

TCP/IP KeepAlive, Session Timeout, RPC Timeout, Exchange, Outlook and you

Update June 21th, 2016 – following feedback and a (true golden) blog post by the Exchange Team – Checklist for troubleshooting Outlook connectivity in Exchange 2013 and 2016 (on-premises) I’ve updated the recommended values for the timeout settings, and shortened the article overall for better reading. Do read the post in general, and in topic – check the CAS & Load Balancer configuration paragraphs.

Hi Again,

This post will spotlight networking considerations that are mostly overlooked. I’ve gathered a few of these issues that might brought you here searching for an answer:

  • Outlook is retrieving data from the Microsoft Exchange Server
  • The connection to Microsoft Exchange is unavailable. Outlook must be online or connected to complete this action
  • Sent items are stuck in Outbox or delayed
  • Outlook freezes or stuck when sending a message
  • Event ID 3033 regarding Exchange Server ActiveSync complaining about the most recent heartbeat intervals used by clients
  • Other strange / weird issues “but PING works! / telnet to the port works great!” – my personal favorite

The mentioned issues or symptoms could take place in any network environment, thus more common in complex network setups where multiple devices are protecting / route network traffic. Some typical configurations examples could be one of the following:

  • Outlook Anywhere or RPC over HTTP is being used, servers are protected or published by ISA / TMG / UAG / F5 / Juniper or any other reverse proxy / publishing solutions
  • Exchange servers are located behind a firewall, router or other network device
  • Clients / Remote clients are located behind a firewall, router or other network device (just to be clear on that…)
  • Exchange servers are being load-balanced with an external physical / virtual appliance

If you’ve read this post up until here and got disappointed because the above does not fit your issue, I’d like to suggest reviewing other RPC troubleshooting topics that might help Troubleshooting Outlook RPC dialog boxes – revisited or Outlook RPC Dialog Box Troubleshooting

Exchange Server traditionally (2000 to 2010) used MAPI over RPC to communicate “natively”, RPC is known to be “sensitive” and that’s why Exchange Server 2013 and beyond allows only Outlook Anywhere (RPC over HTTP) connections from clients which in my opinion is a great change that will simplify future deployments.

Client<>Server connections in general remains active while data “flows” , mails are sent/received etc. but when the connection is Idle, we might have a situation that it will be terminated. Here comes the term KeepAlive – a “dummy” packet that makes sure the connection remain active while no data is flowing and idle.

Here’s my “how-to” suggestion:

  • Configure the RPC timeout on Exchange servers to make sure that components which use RPC will trigger a keep alive signal within the time frame you would expect
    reg add "HKLM\Software\Policies\Microsoft\Windows NT\RPC" -v "MinimumConnectionTimeout" -t REG_DWORD -d 120
  • Consider modifying the server TCP/IP KeepAlive to reduce the chance of “IDLE” connections being terminated – (Default is Two hours – The recommended value is 30 minutes , and no less then 15 minutes) – this controls the OS TCP behavior with idle connections, could greatly improve responsiveness and scalability – http://support.microsoft.com/kb/314053/EN-US
  • Make sure that you are aware of any router, firewall or any other network device that is placed between your clients and your servers. Once you do – note their session timeout, session TTL or session ageing setting for the relevant protocol and port! (this could be tricky, so do not treat this lightly)

The trick for success here is that timeout settings should be configured without overlapping one another while following the client access “path” – for example – Client > FW > Load Balancer > Server:

  • FW timeout TCP/IP timeout – 40 minutes
  • Load Balancer – TCP/IP timeout – 35 minutes
  • Server – TCP/IP timeout – 30 minutes

If additional network devices are placed between the server and your clients, make sure that session timeout settings continue to be configured accordingly.
With today’s security measures, network security has become much more complex. A typical corporate network will implement many different network appliances or software based solutions to secure data, restrict access, prevent attacks and unwanted traffic.
Bottom line – don’t think you are done with network considerations just because “ping works” or an email comes with a statement like “your port is now open”.

I hope this post will benefit others as this issue was and will probably remain common with Exchange and other client / server services.

Don’t get timed out 🙂
Ilantz

Additional useful links and sources of data:

Adjust allowed attachment size for EWS

Hey again,

If you you have any MAC users working against your Exchange 2007 or Exchange 2010 servers, you’ve probably already solved this issue, so this is just for future reference and general knowledge.

Following the Microsoft reference on the subject: Set Message Size Limits for Exchange Web Services the below example is for Exchange 2007

  1. Configure the application to receive requests 50 MB:
    1. Open CMD
    2. %windir%system32inetsrvappcmd set config “Default Web SiteEWS” -section:requestFiltering -requestLimits.maxAllowedContentLength:69905067
  2. Edit web.config to allow 50 MB requests:
    1. Backup %ProgramFiles%MicrosoftExchange ServerV14ClientAccessexchwebewsweb.config
    2. Edit the web.config file , search for maxRequestLength
    3. Change the value from its default 13280 to 51200
  3. IISReset to make sure configurations take place

Notes:

maxAllowedContentLenght value is entered as Bytes, calculate 50MB Base64 encoded size: =((1024*50)*1024)*4/3
maxRequestLenght value is entered as Kilo Bytes, calculate 50MB =1024*50

Hope this post  helped you

ilantz

50 GB of Exchange database logs are filling up my server

Hey again,

Today I wanted to share with you another field report regarding a troubleshooting case I’ve had with Exchange 2010.

A while back in 2009 I’ve re-posted a blog post from the Exchange Team Blog- Troubleshooting Exchange 2003 and 2007 Store Log/Database growth issues – it included a link to Mike Lagase‘s blog and massive troubleshooting guide on this matter.

This week I’ve been called to help with an Exchange server 2010 that was creating tremendous amounts of log files for a specific database, with regards to the blog post I’ve mentioned, ExMon – Exchange Server User Monitor came to the rescue, real fast.

Fired it up with an interval of 15 minutes, and located the user that is responsible for the issue, note the screen shot, sorting by “Log Bytes” the top user created 800 MB in 15 minutes !!

ExMon Screen Shot - Log Bytes Winner

 

 

From this point forward it was easy to solve this issue, disabling both MAPI And Active Sync feature for the user, and detected the cause.

Case closed 🙂

Update – Apr-2013

The Exchange Team has a new post with lots of additions from the original post from 2009 – Troubleshooting Rapid Growth in Databases and Transaction Log Files in Exchange Server 2007 and 2010

How to manually purge Exchange server logs – clean and easy

Update 9/Jun/2015 – Thanks to Josh Davis for the feedback, I’ve added a note about making sure to include both drives (if EDB and LOG files are separated).

Update 21/Oct/2013 – This article suggests that you cannot sustain downtime or interruption for your users while battling with deleting log files or restoring your working backup solution. If you can sustain a downtime (should be around minutes or so) the easiest method will be to enable Circular Logging on your database / storage group – see more here – http://technet.microsoft.com/en-us/library/bb331958%28v=exchg.141%29.aspx#UTL

Update 01/May/2013 – The exchange team has written a script which helps troubleshoot and identity issues with Backups etc.. The script use the DiskShadow utility as well ! check it out @ http://blogs.technet.com/b/exchange/archive/2013/04/29/troubleshoot-your-exchange-2010-database-backup-functionality-with-vsstester-script.aspx

Hi Again !

I often get calls and questions regarding backups and Exchange Server, since ever this issue is not always working as required or as you would expect, but that’s off-topic 🙂

One of the most common stories is that without a working Exchange Server backup when  you perform massive mailbox moves, transaction logs will get piled and fill up the volume or disk that they reside in. and then panic starts, “hey my databases were dismounted…” then of course the administrator realizes that the space on the log drive or volume has indeed ran out and now he needs to figure out what to delete.. and here’s where this post comes in…

So how can you delete or purge Exchange server logs without any risk ? well, in simple – you cannot, because the whole idea of restoring an Exchange or for this matter any transactional database requires you to have a first – “full” backup of the database itself and all transaction logs that were generated since the the date of the database creation date, or the last “successful” “full backup”.

Now here’s a nice method to “fake” a “full backup” or an on-demand transaction logs purge when you see you will be soon out of space, using the Exchange VSS writers and the diskshadow utility (available with Server 2008 or 2008 R2) . This procedure also “proves” that a VSS backup for your Exchange Server will work fine.

note: This method was tested on an Exchange server with Locally Attached Disks, not storage attached LUNs.

Use this method on on your risk. You should preform a “Full Backup” right after this process is done.

This example will show you how to purge the logs for a database that is located on Drive D, the log files of the databases are also located in Drive D. we will “fake backup” drive D and this will trigger the logs to be purged.

Note: If you have separated your log files and database file in different drives, or you want to include additional databases in the “backup” you must include the additional drives in the process, so in the example below, you will “Add volume e:” after “Add volume drive d:” and so on…

  1. Open Command prompt
  2. Launch Diskshadow
    1. Add volume d:
    2. (optional, add one line for each additional drive to include) Add volume X:
    3. Begin Backup
    4. Create
    5. End Backup
  3. At this step you should notice the following events in the application log indicating that the backup was indeed successful and logs will now be deleted.

Here’s some screenshots from the process:

Diskshadow commands for the example

The Diskshadow example screenshot.

ESE Event ID 2005

ESE – Event ID 2005 – Starting a Full Shadow Copy Backup

MSExchangeIS Event ID 9811

MSexchangeIS – Exchange VSS Writer preparation.

ESE Event ID 224 - Logs being Purged

ESE Event ID 224 – Logs are now purged 🙂

MSExchangeIS Event ID 9780 - Backup complete

MSExchangeIS Event ID 9780 – Backup is now complete.

side note: although this example was tested against Exchange 2010, it should work just as fine with Exchange 2013 & 2007.

Hope this helps you !

ilantz

Authentication pop ups and annoyances with Exchange 2007 / 2010 and Outlook Anywhere

Hi again,

This issue has came up too much, so I wanted to blog something short about this.

Prerequisites:

  • Update (Added June 29th 2013) – If using Exchange 2013, check out Exchange 2013 Outlook Anywhere Considerations for some additional specific Exchange 2013 issues.
  • Exchange 2007 or 2010
  • Outlook 2003 / 2007 / 2010
  • Windows XP / 7 / etc..
  • Outlook Anywhere ( RPC over HTTP ) enabled – with Basic Authentication or NTLM Authentication
  • Autodiscover – working correctly 😉

So, you’ve got it all configured, you enabled Outlook Anywhere, configured ISA 2006 / TMG / UAG to publish the Outlook Anywhere (or not), you published Autodiscover records an all is working great !

BUT ! you have this annoying user credentials pop ups, and users are going nuts ! and so do you !@ ( enough sarcasm ) it may work for a while, and then you are prompted again for user and password, or even worse – it might not work at all…

Here’s what can go wrong in bullets because we have a few different issues that might cause troubles..

  • Outlook Anywhere is configured to use NTLM authentication:
    • Solution 1 – Configure MSSTD or the Certificate Principle Name correctly (see below)
    • Solution 2 – Configure your clients local security policy, in specific – LmCompatiblilityLevel to 2 or 3
    • Solution 3 – If you try to pull NTLM with ISA / TMG / UAG, either configure “Kerberos Constrained Delegation” – check links below for the white-paper from Microsoft” or change the publishing rule to apply to “All Users” and in the Authentication Delegation tab choose the option “No delegation, but client may authenticate directly”
  • SSL Certificates issues
    • Outlook Anywhere was enabled for – mail.company.com (ExternalHostName), but you have a wildcard certificate or the certificate subject name does not match mail.company.com
    • Solution – Configure MSSTD or the Certificate Principle Name correctly (see below)
  • Outlook Anywhere continuously keep being configured automatically !%

So what’s that MSSTD or Certificate Principle Name ? well it’s a method Outlook can verify that the server you are connecting to indeed holds the correct SSL certificate subject name before sending credentials to.. well yeah it ain’t that secure.

Microsoft Exchange Proxy Settings

This setting is actually being configured automatically since Exchange 2007 and continue to be with Exchange 2010.

So here’s what you can do with it – all examples follow the Set Outlook Provider cmdlet syntax:CertPrincipalName

    • You have a wildcard certificate – Run this command:

Set-OutlookProvider EXPR -CertPrincipalName msstd:*.company.com

    • You have a differnet subject name on your SSL certificate then the ExternalHostName you configured for Outlook anywhere on your CAS server

Set-OutlookProvider EXPR -CertPrincipalName msstd:correctsubject.company.com

    • You don’t want that “only connect to proxy servers that have this principle name in their certificate” check box marked at all ! 🙂

Set-OutlookProvider EXPR -CertPrincipalName none

New feature with Exchange 2010 – The Set-OutlookProvider cmdlet now allows Outlook 2010 clients to connect exclusively through RPC over HTTP (Outlook Anywhere) before trying RPC over TCP connections when connecting over the Internet. !

This means you can control the check box “On fast network, connect using HTTP first, then connect using TCP/IP”, here’s the two options:

    • Always connect using HTTP (mark “on fast networks”) :

Set-OutlookProvider EXPR -OutlookProviderFlags:ServerExclusiveConnect

    • User TCP/IP first then HTTP (default):

Set-OutlookProvider EXPR -OutlookProviderFlags:None

This should cover it, no more pop ups and hopefully Outlook Anywhere and you will be friends again !

ilantz

Credits (or links) :

When, if and how do you modify Outlook Providers?

Set-OutlookProvider

Publishing Outlook Anywhere Using NTLM Authentication With Forefront TMG or Forefront UAG

Exchange 2013 Outlook Anywhere Considerations

Enable or disable POP3 and IMAP4 by group membership in Exchange 2007

Hi everyone !

My good friend podlisk has finally got his blog up and running, check out the script for this highly required task for Exchange 2007 / 2010 🙂

http://podlisk.wordpress.com/2011/01/13/enable-or-disable-pop3-and-imap4-by-group-membership-in-exchange-2007

Enjoy !

Configure Static “fake” server names for RPC over HTTPS

Hi !

This came up with a request to “fake” exchange server names, which are actually old DE-commissioned servers. users are using RPC over HTTP , and the exchange profile they use had those server names as the actual mailbox server.

This issue might show up when performing cross forest migration or removing servers, while manipulating name resolving using DNS CNAME records, etc..

With Exchange 2010 and Exchange 2007 Outlook Anywhere settings are applied automatically when you enable the outlook anywhere feature on a CAS server, proxy names in the registry ( HKEY_LOCAL_MACHINESOFTWAREMicrosoftRpcRpcProxy) are automatically entered, that is all back-end  servers that were enabled for RPC over HTTP (2003) and all mailbox servers 2007 / 2010.

So by default RpcProxy will only answer for existing  mailbox servers, we want to add our own “old” , “fake” exchange server names.

Here’s how to manipulate the RpcProxy entry in the Exchange server make it stick.

Use at your own risk!

Under each CAS you will enable for Outlook Anywhere follow these steps:

  1. Configure “PeriodicPollingMinutes” to 0 , this will stop automatic settings overwrite – removing the static entries you will add later.Locate the value  PeriodicPollingMinutes, under HKEY_LOCAL_MACHINESYSTEMCurrentControlSetservicesMSExchangeServiceHostRpcHttpConfigurator

    2.  

  2. Configure the “fake” names, for example “email2.fake.com” will be the fake mailbox server we will add.
    Append “;email2.fake.com:6001-6002;email2.fake.com:6004” to the value of “ValidPorts_AutoConfig_Exchange”
    The String Value is under : HKEY_LOCAL_MACHINESOFTWAREMicrosoftRpcRpcProxyFor example:

    EX2010:6001-6002;EX2010:6004;EX2010.test.lab:6001-6002;
    EX2010.test.lab:6004;ex2k3:6001-6002;ex2k3:6004; ex2k3.test.lab:6001-6002;ex2k3.test.lab:6004;
    email2.fake.com:6001-6002;email2.fake.com:6004

    3.  

  3. Restart the services: MSExchangeServiceHost and MSExchangeProtectedServiceHost

    4.  

  4. IISReset

Done !

Now configure outlook to use the email2.fake.com server and configure Outlook Anywhere to verify it works.