Exchange upgrade fails due to missing language files

Just to help anyone out there that might be facing this issue. I’ve helped troubleshoot an Exchange 2010 RTM upgrade to Exchange 2010 SP3 which kept failing due to missing language files…

Event ID 1603 was also thrown as per to the KB 2784788 – “1635” or “1603” error code when you install update rollups or service packs for Exchange Server 2007 or Exchange Server 2010

The MSILOG indeed showed that the setup was looking for the RTM language files in the original location where the setup files were, but they are long gone… with the RTM DVD no where to be-found (RTM trial files + the oldest Language Pack bundle are in a non compatible version) this situation was doomed to failure.

So, I’ve turned to manually remove any references to the Client / Server language packs on the server, this included removing a whole bunch of registry keys:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v14\Language Packs\ <-- the whole KEY
HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Installer\Products\  <-- Whatever "Microsoft Exchange ** Language Pack" I found
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\  <-- Whatever "Microsoft Exchange ** Language Pack" I found
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData\S-1-5-18\Products  <-- Whatever "Microsoft Exchange ** Language Pack" I found

Following this brutal way, I’ve stumbled upon a way to Applying Small Updates by Reinstalling the Product this actually achieves what the installer wants:

msiexec /i Server<or>ClientLanguagePack.msi REINSTALLMODE=vomus

And it works ! Now, I guess that with a script this would have been much quicker then the registry method, but at least now I’m (and you are) aware of this workaround , and here’s the script for your usage:

** edit the $setuplocation variable for your directory of the servicepack.

$setupLocation = "c:\sp3"
$allDirs = dir $setupLocation -Directory
foreach ($dir in $allDirs)
{
if (Test-Path ($dir.FullName + "\clientlanguagepack.msi")) {Write-Host "Installing" $dir.name ; Start-Process -FilePath msiexec -ArgumentList /i, ($dir.FullName + "\clientlanguagepack.msi"), "REINSTALLMODE=vomus" -Wait }
if (Test-Path ($dir.FullName + "\serverlanguagepack.msi")) {Write-Host "Installing" $dir.name ; Start-Process -FilePath msiexec -ArgumentList /i, ($dir.FullName + "\serverlanguagepack.msi"), "REINSTALLMODE=vomus" -Wait }
}

ilantz


Additional references:

Upgrading Service pack – keep asking for language pack

http://stackoverflow.com/a/7916340 – credit for the REINSTALLMODE=vomus trick

How to restore the missing Windows Installer cache files and resolve problems that occur during a SQL Server update – kb 969052

Don’t forget to modify your Windows Server Power Options

Following a troubleshooting session I’ve had lately, I wanted to share with you an important recommended settings that most folks (myself included) often overlook.

With more and more virtual servers and less and less physical servers being deployed, capabilities like SpeedStep of a CPU were forgotten. Take for example the following “modest” specifications of Intel Xeon E5-2690 v2, with 10 cores @ 3.0 GHz this is a “fare” spec for a high load / CPU intensive profile server.

BUT ! if you forget to select the “High Performance” power option in Windows Server for example, you could end up with:

e5-2690v2 at half speed

Notice that the speed of the CPU is less the half the speed it can run at. now to make things better, just make sure to select the “preferred” settings for your busy server:

e5-2690v2 at full speed

Just a heads up for all you folks out there, the default “Balanced” option caused a performance issue with an Exchange 2013 server that was running on this physical hardware and once the option was changed – all was back to normal 🙂

ilantz

Exchange 2013 CU5 upgrade fails with error – The object already exists

Just ran into this one,

An existing Exchange 2013 CU2 installation was requested to be updated to CU5, nothing special there…

Once trying to run the CU5 setup.exe /PrepareAD , the setup failed with an error:
[02/07/2014 11:57:49.0192] [1] [ERROR] Active Directory operation failed on dc.domain.com. The object 'CN=Default,CN=OWA Mailbox Policies,CN=Exchange,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domain,DC=com' already exists.
[02/07/2014 11:57:49.0192] [1] [ERROR] The object exists.

Digging in the ExchangeSetup.log file, I’ve tried to identify the cause.

[02/07/2014 11:57:49.0192] [1] [ERROR] The following error was generated when "$error.Clear();
$policyDefault = Get-OwaMailboxPolicy -DomainController $RoleDomainController | where {$_.Identity -eq "Default"};

if($policyDefault -eq $null)
{
New-OwaMailboxPolicy -Name "Default" -DomainController $RoleDomainController
}
" was run:
"Microsoft.Exchange.Data.Directory.ADObjectAlreadyExistsException:
Active Directory operation failed on dc.domain.com. The object 'CN=Default,CN=OWA Mailbox Policies,CN=Exchange,CN=Microsoft
Exchange,CN=Services,CN=Configuration,DC=domain,DC=com' already exists. ---> System.DirectoryServices.Protocols.DirectoryOperationException:
The object exists.

So I’ve tried to reproduce the test manually using the same command in the setup:
Get-OwaMailboxPolicy -DomainController $RoleDomainController | where {$_.Identity -eq "Default"}
And the result was indeed $null .. which made no sense here… because it does exists, as the error states – the object ‘CN=Default,CN=OWA Mailbox Policies,CN=Exchange,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domain,DC=com’ already exists.

Then I’ve noticed that the CN was “default” with lower “d” … although the Where-Object and -eq should be case insensitive, the check failed…

get-owamailboxpolicy before the change - default

So, I’ve modified the value to be “Default” with capital “D”:

Set-OwaMailboxPolicy -Identity default -Name Default

get-owamailboxpolicy after the change

that did the trick 🙂 and the setup.exe /PrepareAD was successful.

ilantz

This message could not be sent – Error 0x80070005 – Office 365 | Report non-inherited Send-As permissions script

After a few incidents from Office 365 deployments, I’d like to share this issue to help anyone facing it.

If you or anyone of your users tried to send an email and use the “From” option to send as another recipient you might face NDR’s (non delivery reports) which will include these errors:

  • Delivery has failed to these recipients or groups
  • This message could not be sent. Try sending the message again later, or contact your network administrator.  Error is [0x80070005-00000000-00000000]

Using Exchange Server Error Code Look-up (Download Err.exe), 0x80070005 resolves back to MAPI_E_NO_ACCESS or E_ACCESSDENIED which bring us to the actual cause of the issue.

SendAs / Send-as permissions are not retained in migrations to Office 365 just because it is based on an ACL set in Active Directory and ACLs are not synced to Office 365.

To add a SendAs permission use the Add-RecipientPermission cmdlet with Exchange Online Remote PowerShell or use the Exchange Admin Control Panel and add the Send As permission from the “Mailbox Delegation” menu.

Add-RecipientPermission "Help Desk" -AccessRights SendAs -Trustee "Ayla Kol"

See the full reference about the command here – http://technet.microsoft.com/en-us/library/ff935839(v=exchg.150).aspx

As a result of this issue, I’ve created a small script to report which recipients (of any type) have non inherited SendAs permissions ACL’s.  You can later use the report to re-create the permission in 365.

Download the script here: http://gallery.technet.microsoft.com/Report-non-inherited-Send-86ab658b

 

Manually adding a secondary SMTP proxy address for hybrid Exchange Online and Office 365

Update – 05-02-2015 – Thanks for the feedback about this post, some more work has been done, please download the new version…

Update – 07-30-2014 – Thanks for the feedback about this post, I’ve republished the code. it is now wrapped as a script and also logs results to a log file. download the new version…

I’ve been busy with more Office 365 and Hybrid Exchange Online deployments and came up with a script I hope will help some of you out there.

While deploying an Hybrid Exchange Online setup, one of the steps the Hybrid Configuration Wizard is doing is modifying the email address policy and adding “alias@tenant.mail.onmicrosoft.com” to the relevant EAP policies. This is great although there’s a good chance you have some mailboxes that are set with EmailAddressPolicyEnabled:$false

I’ve written a function script that will help you add the additional secondary SMTP proxy address to those mailboxes easily 🙂

Here’s an example on how to use the script:

.\Add-OnMicrosoftSMTP.ps1 -Tenant:ilantz

The script will require your “Tenant” name, for example – if your Office 365 tenant is ilantz.onmicrosoft.com, enter ilantz as the tenant name. Once entered it will find all mailboxes with the property EmailAddressPolicyEnabled:$false and have no routing SMTP address like *@tenant.mail.onmicrosoft.com (following the default Exchange Hybrid Configuration Wizard settings).
Then the script will add the required SMTP proxy address following the PrimarySmtpAdress prefix, if that SMTP proxy address is already taken, the function will add a random 5 digit number to the prefix – John.Doe12345@tenant.mail.onmicrosoft.com.

The script will catch and display any exceptions that may occur during the process. and will automatically log the results to a log file.

Get the script here – http://gallery.technet.microsoft.com/Office-365-Add-Exchange-14c7f0c3
Revision History
——————————————————————————–

1.0 – Initial release

1.1 – Updated and rewritten as a script instead of a function which caused confusion

1.2 – Added Logging of successful addresses being added and failures

1.3 – Updated with server-side filtering to get all relevant users for better efficiency,
an updated logging mechanism and using now the PrimarySmtpAddress prefix value for the routing address.

Enjoy !

ilantz

Exchange 2013 Outlook Anywhere Considerations

Hi,

With Exchange 2013 deployments already in place, I’ve wanted to share with you all some “new” behaviors, tips and more to help you prevent headaches and issues 🙂

With regards to two previously posts – Prevent Outlook Anywhere (aka RPC over HTTP) from being automatically configured in Exchange 2007 with autodiscover and also Authentication pop ups and annoyances with Exchange 2007 / 2010 and Outlook Anywhere – this post is some sort of a follow-up.

With Exchange 2013, Outlook Anywhere (aka RPC over HTTP/s) is the default method for Outlook clients connections – that is no more direct RPC connections to the servers for Outlook clients. Exchange 2013 will essentially require you to utilize Autodiscover and Outlook Anywhere to actually get your Outlook client connected. This is the main reason for writing this post. This information will come useful if you are getting ready or already started to deploy Exchange 2013, I’ll try to keep it simple and write this down as a list of things to consider so this will be rather easy to all.

  1. If you followed my post about how to prevent Outlook Anywhere from being configured and removed the EXPR outlook provider, start with restoring it. Run the following powershell command to restore it:
    New-OutlookProvider -Name:EXPR
  2. If you’re using any additional methods to configure Outlook Clients or Outlook Anywhere like, static XML files, Registry settings or Group Policy settings make sure to revise or even remove them. See also http://support.microsoft.com/kb/2212902
  3. Pay attention to publishing guides for Exchange 2013 – see Publishing Exchange Server 2013 using TMG and also Exchange 2013 Client Access Server Configuration
  4. When enabling Outlook Anywhere on Exchange 2013 notice the following:
    1. Retain the current External authentication method (Basic,NTLM) your internal authentication method should always be NTLM.
      Get-OutlookAnywhere –Server (hostname) | Set-OutlookAnywhere -InternalHostname "mail.domain.com" -InternalClientAuthenticationMethod Ntlm -InternalClientsRequireSsl $true -ExternalHostname "mail.domain.com" -ExternalClientAuthenticationMethod Basic -ExternalClientsRequireSsl $true -IISAuthenticationMethods NTLM,Basic -ssloffloading:$false
    2. Enable NTLM on the IIS /rpc directory of your Exchange 2007/2010 servers
      Get-OutlookAnywhere | ?{ $_.AdminDisplayVersion -notlike "Version 15.*"} | Set-OutlookAnywhere -IISAuthenticationMethods NTLM,Basic
  5. Plan the CertPrincipalName value you will use, that is the certificate Subject Name that your clients will use to populate the msstd:server.domain.com value – both internally and externally (reminding you to see the note above). My personal best practice is to use the same Subject Name on the certificate you will use on your External TMG/UAG/Juniper/F5 reverse proxy and your internal server or servers.
    Once you are aware of this value you can configure your Outlook Provider accordingly (you can refer to this post for more information on the subject).
  6. If you installed a wildcard certificate on your Exchange 2013 server – you must perform the following:
    1. Update your EXPR setting – Set-OutlookProvider EXPR -CertPrincipalName msstd:*.company.com
    2. Update your EXCH setting (yes!) – Set-OutlookProvider EXCH -CertPrincipalName msstd:*.company.com
  7. Don’t freak out when you see Exchange 2013 “new” server name – it’s value is actually the Mailbox GUID value, and will be unique for all users. This means that – you must use the Autodiscover wizard to configure outlooks from now on, Email, password and click next.
    If you have full mailbox access to a different mailbox – that’s great- just type it’s email address and enter whatever you want for password. (will work only inside the LAN…)
  8. Don’t forget to update your Outlook clients – or else they will not connect to Exchange 2013 – see Exchange 2013 System Requirements for the exact information.

That’s it for now, while deployments continue I will update this topic with new “gotchas”.

Hope this helps anyone out there.
Ilantz

The action cannot be completed error using Outlook – Exchange 2010 or Office 365

Hi,

Quick note from the field, if you are moving to Exchange Online / Office 365 you should double check your current office group-policy settings and registry for Outlook.

You should make sure that you did not enable the Closest GC setting, or configured a specific global catalog server with the DS Server registry entries under HKEY_CURRENT_USER\Software\Microsoft\Exchange\Exchange Provider

Both registry values, errors and methods for resolution are located at:

http://support.microsoft.com/kb/2507626 – Error in Outlook: “The action cannot be completed. The Bookmark is not valid”

http://support.microsoft.com/kb/319206 – How to configure Outlook to a specific global catalog server or to the closest global catalog server

And if we are on the subject, it’s also a good practice to make sure the following when moving to Office 365:

  • You do not have Autodiscover related registry settings also – http://support.microsoft.com/kb/2212902 – Unexpected Autodiscover behavior when you have registry settings under the \Autodiscover key
  • Make sure that the “Encrypt data between Microsoft Office Outlook and Microsoft Exchange Server” option under account settings of the Outlook Profile is indeed selected. Office 365 is restricting clients to encrypt MAPI traffic – see the following KB for additional information (originally written for Exchange 2010 RTM) – http://support.microsoft.com/kb/2006508

ilantz

Exchange 2013 Migration Batch Stalled Due To Content Indexing CiAgeOfLastNotification

Hi,

I’ve just encountered this issue during a LAB for migrating Exchange 2010 to Exchange 2013, migration batches were getting stuck in Syncing, in addition I noticed two annoying warning messages in the application log of the server with Event ID 1009 and Event ID 1013 with source MSExchangeFastSearch

I’ve looked in the migration report using:
Get-MigrationUserStatistics -IncludeReport -Identity ilantz@lab.com | fl
......
4/28/2013 7:44:46 AM [EX2013] The job is currently stalled due to 'Content Indexing' lagging behind on resource 'CiAgeOfLastNotification(Mailbox Database
.....

So, indeed the Content Indexing which was failing and keeping the migration back… nothing special here, Exchange 2010 had this issue as well …
Quick search showed a very odd solution to this…

Quoting http://support.microsoft.com/kb/2807668 – Content Index status of all or most of the mailbox databases in the environment shows “Failed”

This issue may occur if the search platform tries to check its membership in a security group that is named “ContentSubmitters.” This group is not created by the search platform or by Exchange Server 2013 and is therefore not usually present. Although the check usually fails silently, without any consequences, an exception sometimes occurs. This causes the search component to fail.

Wow … well 🙂 .. hope this will be fixed with CU2.. go with Method 1 in the KB, worked like a charm here.

ilantz

TCP/IP KeepAlive, Session Timeout, RPC Timeout, Exchange, Outlook and you

Update June 21th, 2016 following feedback and a (true golden) blog post by the Exchange Team – Checklist for troubleshooting Outlook connectivity in Exchange 2013 and 2016 (on-premises) I’ve updated the recommended values for the timeout settings, and shortened the article overall for better reading. Do read the post in general, and in topic – check the CAS & Load Balancer configuration paragraphs.


Hi Again,

This post will spotlight networking considerations that are mostly overlooked. I’ve gathered a few of these issues that might brought you here searching for an answer:

  • Outlook is retrieving data from the Microsoft Exchange Server
  • The connection to Microsoft Exchange is unavailable. Outlook must be online or connected to complete this action
  • Sent items are stuck in Outbox or delayed
  • Outlook freezes or stuck when sending a message
  • Event ID 3033 regarding Exchange Server ActiveSync complaining about the most recent heartbeat intervals used by clients
  • Other strange / weird issues “but PING works! / telnet to the port works great!” – my personal favorite

The mentioned issues or symptoms could take place in any network environment, thus more common in complex network setups where multiple devices are protecting / route network traffic. Some typical configurations examples could be one of the following:

  • Outlook Anywhere or RPC over HTTP is being used, servers are protected or published by ISA / TMG / UAG / F5 / Juniper or any other reverse proxy / publishing solutions
  • Exchange servers are located behind a firewall, router or other network device
  • Clients / Remote clients are located behind a firewall, router or other network device (just to be clear on that…)
  • Exchange servers are being load-balanced with an external physical / virtual appliance

If you’ve read this post up until here and got disappointed because the above does not fit your issue, I’d like to suggest reviewing other RPC troubleshooting topics that might help Troubleshooting Outlook RPC dialog boxes – revisited or Outlook RPC Dialog Box Troubleshooting

Exchange Server traditionally (2000 to 2010) used MAPI over RPC to communicate “natively”, RPC is known to be “sensitive” and that’s why Exchange Server 2013 and beyond allows only Outlook Anywhere (RPC over HTTP) connections from clients which in my opinion is a great change that will simplify future deployments.

Client<>Server connections in general remains active while data “flows” , mails are sent/received etc. but when the connection is Idle, we might have a situation that it will be terminated. Here comes the term KeepAlive – a “dummy” packet that makes sure the connection remain active while no data is flowing and idle.

Here’s my “how-to” suggestion:

  • Configure the RPC timeout on Exchange servers to make sure that components which use RPC will trigger a keep alive signal within the time frame you would expect
    reg add "HKLM\Software\Policies\Microsoft\Windows NT\RPC" -v "MinimumConnectionTimeout" -t REG_DWORD -d 120
  • Consider modifying the server TCP/IP KeepAlive to reduce the chance of “IDLE” connections being terminated – (Default is Two hours – The recommended value is 30 minutes , and no less then 15 minutes) – this controls the OS TCP behavior with idle connections, could greatly improve responsiveness and scalability – http://support.microsoft.com/kb/314053/EN-US
  • Make sure that you are aware of any router, firewall or any other network device that is placed between your clients and your servers. Once you do – note their session timeout, session TTL or session ageing setting for the relevant protocol and port! (this could be tricky, so do not treat this lightly)

The trick for success here is that timeout settings should be configured without overlapping one another while following the client access “path” – for example – Client > FW > Load Balancer > Server:

  • FW timeout TCP/IP timeout – 40 minutes
  • Load Balancer – TCP/IP timeout – 35 minutes
  • Server – TCP/IP timeout – 30 minutes

If additional network devices are placed between the server and your clients, make sure that session timeout settings continue to be configured accordingly.
With today’s security measures, network security has become much more complex. A typical corporate network will implement many different network appliances or software based solutions to secure data, restrict access, prevent attacks and unwanted traffic.
Bottom line – don’t think you are done with network considerations just because “ping works” or an email comes with a statement like “your port is now open”.

I hope this post will benefit others as this issue was and will probably remain common with Exchange and other client / server services.

Don’t get timed out 🙂
Ilantz

Additional useful links and sources of data: