Exchange upgrade fails due to missing language files

Just to help anyone out there that might be facing this issue. I’ve helped troubleshoot an Exchange 2010 RTM upgrade to Exchange 2010 SP3 which kept failing due to missing language files…

Event ID 1603 was also thrown as per to the KB 2784788 – “1635” or “1603” error code when you install update rollups or service packs for Exchange Server 2007 or Exchange Server 2010

The MSILOG indeed showed that the setup was looking for the RTM language files in the original location where the setup files were, but they are long gone… with the RTM DVD no where to be-found (RTM trial files + the oldest Language Pack bundle are in a non compatible version) this situation was doomed to failure.

So, I’ve turned to manually remove any references to the Client / Server language packs on the server, this included removing a whole bunch of registry keys:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v14\Language Packs\ <-- the whole KEY
HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Installer\Products\  <-- Whatever "Microsoft Exchange ** Language Pack" I found
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\  <-- Whatever "Microsoft Exchange ** Language Pack" I found
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData\S-1-5-18\Products  <-- Whatever "Microsoft Exchange ** Language Pack" I found

Following this brutal way, I’ve stumbled upon a way to Applying Small Updates by Reinstalling the Product this actually achieves what the installer wants:

msiexec /i Server<or>ClientLanguagePack.msi REINSTALLMODE=vomus

And it works ! Now, I guess that with a script this would have been much quicker then the registry method, but at least now I’m (and you are) aware of this workaround , and here’s the script for your usage:

** edit the $setuplocation variable for your directory of the servicepack.

$setupLocation = "c:\sp3"
$allDirs = dir $setupLocation -Directory
foreach ($dir in $allDirs)
{
if (Test-Path ($dir.FullName + "\clientlanguagepack.msi")) {Write-Host "Installing" $dir.name ; Start-Process -FilePath msiexec -ArgumentList /i, ($dir.FullName + "\clientlanguagepack.msi"), "REINSTALLMODE=vomus" -Wait }
if (Test-Path ($dir.FullName + "\serverlanguagepack.msi")) {Write-Host "Installing" $dir.name ; Start-Process -FilePath msiexec -ArgumentList /i, ($dir.FullName + "\serverlanguagepack.msi"), "REINSTALLMODE=vomus" -Wait }
}

ilantz


Additional references:

Upgrading Service pack – keep asking for language pack

http://stackoverflow.com/a/7916340 – credit for the REINSTALLMODE=vomus trick

How to restore the missing Windows Installer cache files and resolve problems that occur during a SQL Server update – kb 969052

This message could not be sent – Error 0x80070005 – Office 365 | Report non-inherited Send-As permissions script

After a few incidents from Office 365 deployments, I’d like to share this issue to help anyone facing it.

If you or anyone of your users tried to send an email and use the “From” option to send as another recipient you might face NDR’s (non delivery reports) which will include these errors:

  • Delivery has failed to these recipients or groups
  • This message could not be sent. Try sending the message again later, or contact your network administrator.  Error is [0x80070005-00000000-00000000]

Using Exchange Server Error Code Look-up (Download Err.exe), 0x80070005 resolves back to MAPI_E_NO_ACCESS or E_ACCESSDENIED which bring us to the actual cause of the issue.

SendAs / Send-as permissions are not retained in migrations to Office 365 just because it is based on an ACL set in Active Directory and ACLs are not synced to Office 365.

To add a SendAs permission use the Add-RecipientPermission cmdlet with Exchange Online Remote PowerShell or use the Exchange Admin Control Panel and add the Send As permission from the “Mailbox Delegation” menu.

Add-RecipientPermission "Help Desk" -AccessRights SendAs -Trustee "Ayla Kol"

See the full reference about the command here – http://technet.microsoft.com/en-us/library/ff935839(v=exchg.150).aspx

As a result of this issue, I’ve created a small script to report which recipients (of any type) have non inherited SendAs permissions ACL’s.  You can later use the report to re-create the permission in 365.

Download the script here: http://gallery.technet.microsoft.com/Report-non-inherited-Send-86ab658b

 

Exchange Hybrid Configuration failed with error Subtask Configure execution failed

Hi Again,

While setting up the Hybrid Configuration Wizard on an Exchange 2010 server for Office 365, I’ve encountered this error:

[2/4/2014 13:36:8] INFO:Running command: Get-FederationInformation -DomainName 'contoso.mail.onmicrosoft.com' -BypassAdditionalDomainValidation 'True'
[2/4/2014 13:36:8] INFO:Cmdlet: Get-FederationInformation --Start Time: 2/4/2014 3:36:08 PM.
[2/4/2014 13:36:16] INFO:Cmdlet: Get-FederationInformation --End Time: 2/4/2014 3:36:16 PM.
[2/4/2014 13:36:16] INFO:Cmdlet: Get-FederationInformation --Processing Time: 7690.8.
[2/4/2014 13:36:16] INFO:Disconnected from On-Premises session
[2/4/2014 13:36:17] INFO:Disconnected from Tenant session
[2/4/2014 13:36:17] ERROR:Updating hybrid configuration failed with error 'Subtask Configure execution failed: Creating Organization Relationships.


Execution of the Get-FederationInformation cmdlet had thrown an exception. This may indicate invalid parameters in your Hybrid Configuration settings.


Operation is not valid due to the current state of the object.
at System.Management.Automation.PowerShell.CoreInvoke[TOutput](IEnumerable input, PSDataCollection`1 output, PSInvocationSettings settings)
at System.Management.Automation.PowerShell.Invoke(IEnumerable input, PSInvocationSettings settings)
at System.Management.Automation.PowerShell.Invoke()
at Microsoft.Exchange.Management.Hybrid.RemotePowershellSession.RunCommand(String cmdlet, Dictionary`2 parameters, Boolean ignoreNotFoundErrors)
'.


Additional troubleshooting information is available in the Update-HybridConfiguration log file located at C:\Program Files\Microsoft\Exchange Server\V14\Logging\Update-HybridConfiguration\HybridConfiguration_2_4_2014_13_35_39_635271177398297855.log.

Looking at the application log in the Exchange server showed an Event ID 403 with source MSExchange Common:
The Certificate named xxxxxxx in the Federation Trust 'Microsoft Federation Gateway' is expired. Please review the Federation Trust properties and the certificates installed in the certificate store of the server.

The Certificate named xxxxxxx in the Federation Trust 'Microsoft Federation Gateway' is expired. Please review the Federation Trust properties and the certificates installed in the certificate store of the server.

After checking of course, the Federation certificate was just created… and is indeed valid…..

All that was required was a quick “restart” to the application pools on the server, I usually just restart the MSExchangeServiceHost and MSExchangeProtectedServiceHost services. after that the wizard completed successfully 🙂

Hope this helped anyone,

ilantz

Manually adding a secondary SMTP proxy address for hybrid Exchange Online and Office 365

Update – 05-02-2015 – Thanks for the feedback about this post, some more work has been done, please download the new version…

Update – 07-30-2014 – Thanks for the feedback about this post, I’ve republished the code. it is now wrapped as a script and also logs results to a log file. download the new version…

I’ve been busy with more Office 365 and Hybrid Exchange Online deployments and came up with a script I hope will help some of you out there.

While deploying an Hybrid Exchange Online setup, one of the steps the Hybrid Configuration Wizard is doing is modifying the email address policy and adding “alias@tenant.mail.onmicrosoft.com” to the relevant EAP policies. This is great although there’s a good chance you have some mailboxes that are set with EmailAddressPolicyEnabled:$false

I’ve written a function script that will help you add the additional secondary SMTP proxy address to those mailboxes easily 🙂

Here’s an example on how to use the script:

.\Add-OnMicrosoftSMTP.ps1 -Tenant:ilantz

The script will require your “Tenant” name, for example – if your Office 365 tenant is ilantz.onmicrosoft.com, enter ilantz as the tenant name. Once entered it will find all mailboxes with the property EmailAddressPolicyEnabled:$false and have no routing SMTP address like *@tenant.mail.onmicrosoft.com (following the default Exchange Hybrid Configuration Wizard settings).
Then the script will add the required SMTP proxy address following the PrimarySmtpAdress prefix, if that SMTP proxy address is already taken, the function will add a random 5 digit number to the prefix – John.Doe12345@tenant.mail.onmicrosoft.com.

The script will catch and display any exceptions that may occur during the process. and will automatically log the results to a log file.

Get the script here – http://gallery.technet.microsoft.com/Office-365-Add-Exchange-14c7f0c3
Revision History
——————————————————————————–

1.0 – Initial release

1.1 – Updated and rewritten as a script instead of a function which caused confusion

1.2 – Added Logging of successful addresses being added and failures

1.3 – Updated with server-side filtering to get all relevant users for better efficiency,
an updated logging mechanism and using now the PrimarySmtpAddress prefix value for the routing address.

Enjoy !

ilantz

The Outlook Web App address is out of date – Office 365 Hybrid

Quick note from the field..

I’ve encountered an issue with an Exchange 2010 and Office 365 Hybrid configuration, users that were moved to Office 365 and tried to reach the original On-Premise OWA URL were receiving an error – The Outlook Web App address https://owa.domain.com/owa is out of date.

The Outlook Web App Address Is Out Of Date

What should have happen is that the OWA will offer the users to use the URL configured on the TargetOwaUrl parameter on the Organization Relationship to the Office 365 routing domain. After some digging I’ve realized that this hybrid setup was performed using the manual steps that were documented for Exchange 2010 SP1, so the Hybrid Configuration Wizard did not do it’s magic….

Anyhow, after comparing this setup with a working hybrid configuration including the OWA redirection, I’ve noticed that the TargetOwaUrl value did not had xxx/owa/xxxx in it’s URL.

So instead of http://outlook.com/owa/domain.mail.onmicrosoft.com – I’ve had http://outlook.com/domain.mail.onmicrosoft.com

So after running Set-OrganizationRelationship -TargetOwaURL “http://outlook.com/owa/domain.mail.onmicrosoft.com” the redirection worked as expected.

Hope this helps out anyone,

ilantz

See also: Simplify the OWA URL for Office 365 Hybrid

Missing Microsoft-Server-ActiveSync and OMA virtual directories in IIS and Active Directory with Exchange 2003

Hi Again,

I’ve encountered a situation today with an Exchange 2003 to 2010 migration, The Exchange 2003 Back-End server was missing the virtual directories in IIS, but this issue had an interesting twist … both vdirs were missing also in the Active Directory !

Missing or corrupt virtual directories with Exchange are common and can be easily solved with KB 883380 – How to reset the default virtual directories that are required to provide Outlook Web Access, Exchange ActiveSync, and Outlook Mobile Access services in Exchange Server 2003

Exchange Server setup creates each virtual directory in the AD forest configuration partition under services, microsoft Exchange, administrative groups, administration group name, server name, protocols, http, virtual server name (usually 1).

In this case, both ActiveSync and OMA virtual directories were missing from Active Directory and as a result were also missing from the System Manager MMC, so performing a Repair Setup or the Reset virtual directories method will render useless.. both will not write anything new to the AD.. we had to re-create the two virtual directories both in AD and the IIS, so using System Manager, we try to create a new virtual directory, but the ActiveSync and OMA is greyed out !

Microsoft-Server-ActiveSync and  OMA Missing From Active Directory

Creating New Virtual Directory With System Manager Grayed out

With some searching I’ve reached a solution that worked perfectly (dated back to 2007 from the Microsoft Exchange newsgroup), this will enable the options within the System Manager and allow us to re-create the virtual directories and restore order 🙂

1. Using ADSIEDIT locate the Exchange 2003 server container – services, microsoft Exchange, administrative groups, administration group name and right click the server name to open it’s properties.
2. Locate the Heuristics attribute and note the current value (just in-case…) our value in this case was 805310468.
4. Change the value to 270012416 , click apply and ok.
5. Refresh the Exchange System Manager or close and re-open it.
6. Now, locate the server name within the tree, expend it, expend protocols, expend HTTP, expend the virtual server name and right click to create a new Virtual Directory. You should now be able to recreate the Microsoft-Server-ActiveSync and OMA virtual directory.

Note – This will also reset your RPC over HTTP and other “server specific” settings that you configured on the server using the System Manager GUI. so make sure to note all configurations under the server properties page and re-enable any changes after setting the value.

Credits – http://microsoft.newsgroups.archived.at/public.exchange.setup/200702/07021815421.html

I hope this helps anyone struggling with this,

Ilantz

The action cannot be completed error using Outlook – Exchange 2010 or Office 365

Hi,

Quick note from the field, if you are moving to Exchange Online / Office 365 you should double check your current office group-policy settings and registry for Outlook.

You should make sure that you did not enable the Closest GC setting, or configured a specific global catalog server with the DS Server registry entries under HKEY_CURRENT_USER\Software\Microsoft\Exchange\Exchange Provider

Both registry values, errors and methods for resolution are located at:

http://support.microsoft.com/kb/2507626 – Error in Outlook: “The action cannot be completed. The Bookmark is not valid”

http://support.microsoft.com/kb/319206 – How to configure Outlook to a specific global catalog server or to the closest global catalog server

And if we are on the subject, it’s also a good practice to make sure the following when moving to Office 365:

  • You do not have Autodiscover related registry settings also – http://support.microsoft.com/kb/2212902 – Unexpected Autodiscover behavior when you have registry settings under the \Autodiscover key
  • Make sure that the “Encrypt data between Microsoft Office Outlook and Microsoft Exchange Server” option under account settings of the Outlook Profile is indeed selected. Office 365 is restricting clients to encrypt MAPI traffic – see the following KB for additional information (originally written for Exchange 2010 RTM) – http://support.microsoft.com/kb/2006508

ilantz

Microsoft Exchange Web Services Managed API 1.1 Download

Hi All,

Following a workaround mentioned on http://support.microsoft.com/kb/2512023 – “GetUserOofSettings”, “SetUserOofSettings” and “GetUserAvailability” operations do not support Exchange Impersonation on the Exchange Server 2010 SP1 schema

Seems like the EWS Managed API 1.1 download link does not work anymore, and it’s virtually impossible to get a hold of the files on the WEB, so I’ve uploaded the redistributable package here for anyone looking around for it.

Anyway – grab it here: Microsoft Exchange Web Services Managed API 1.1

Hope this helps.

ilantz

Exchange Web Service Managed API 1.1, Download – Microsoft ( Non working download links) :
www.microsoft.com/en-us/download/details.aspx?id=13480
http://www.microsoft.com/downloads/en/details.aspx?FamilyID=c3342fb3-fbcc-4127-becf-872c746840e1

TCP/IP KeepAlive, Session Timeout, RPC Timeout, Exchange, Outlook and you

Update June 21th, 2016 following feedback and a (true golden) blog post by the Exchange Team – Checklist for troubleshooting Outlook connectivity in Exchange 2013 and 2016 (on-premises) I’ve updated the recommended values for the timeout settings, and shortened the article overall for better reading. Do read the post in general, and in topic – check the CAS & Load Balancer configuration paragraphs.


Hi Again,

This post will spotlight networking considerations that are mostly overlooked. I’ve gathered a few of these issues that might brought you here searching for an answer:

  • Outlook is retrieving data from the Microsoft Exchange Server
  • The connection to Microsoft Exchange is unavailable. Outlook must be online or connected to complete this action
  • Sent items are stuck in Outbox or delayed
  • Outlook freezes or stuck when sending a message
  • Event ID 3033 regarding Exchange Server ActiveSync complaining about the most recent heartbeat intervals used by clients
  • Other strange / weird issues “but PING works! / telnet to the port works great!” – my personal favorite

The mentioned issues or symptoms could take place in any network environment, thus more common in complex network setups where multiple devices are protecting / route network traffic. Some typical configurations examples could be one of the following:

  • Outlook Anywhere or RPC over HTTP is being used, servers are protected or published by ISA / TMG / UAG / F5 / Juniper or any other reverse proxy / publishing solutions
  • Exchange servers are located behind a firewall, router or other network device
  • Clients / Remote clients are located behind a firewall, router or other network device (just to be clear on that…)
  • Exchange servers are being load-balanced with an external physical / virtual appliance

If you’ve read this post up until here and got disappointed because the above does not fit your issue, I’d like to suggest reviewing other RPC troubleshooting topics that might help Troubleshooting Outlook RPC dialog boxes – revisited or Outlook RPC Dialog Box Troubleshooting

Exchange Server traditionally (2000 to 2010) used MAPI over RPC to communicate “natively”, RPC is known to be “sensitive” and that’s why Exchange Server 2013 and beyond allows only Outlook Anywhere (RPC over HTTP) connections from clients which in my opinion is a great change that will simplify future deployments.

Client<>Server connections in general remains active while data “flows” , mails are sent/received etc. but when the connection is Idle, we might have a situation that it will be terminated. Here comes the term KeepAlive – a “dummy” packet that makes sure the connection remain active while no data is flowing and idle.

Here’s my “how-to” suggestion:

  • Configure the RPC timeout on Exchange servers to make sure that components which use RPC will trigger a keep alive signal within the time frame you would expect
    reg add "HKLM\Software\Policies\Microsoft\Windows NT\RPC" -v "MinimumConnectionTimeout" -t REG_DWORD -d 120
  • Consider modifying the server TCP/IP KeepAlive to reduce the chance of “IDLE” connections being terminated – (Default is Two hours – The recommended value is 30 minutes , and no less then 15 minutes) – this controls the OS TCP behavior with idle connections, could greatly improve responsiveness and scalability – http://support.microsoft.com/kb/314053/EN-US
  • Make sure that you are aware of any router, firewall or any other network device that is placed between your clients and your servers. Once you do – note their session timeout, session TTL or session ageing setting for the relevant protocol and port! (this could be tricky, so do not treat this lightly)

The trick for success here is that timeout settings should be configured without overlapping one another while following the client access “path” – for example – Client > FW > Load Balancer > Server:

  • FW timeout TCP/IP timeout – 40 minutes
  • Load Balancer – TCP/IP timeout – 35 minutes
  • Server – TCP/IP timeout – 30 minutes

If additional network devices are placed between the server and your clients, make sure that session timeout settings continue to be configured accordingly.
With today’s security measures, network security has become much more complex. A typical corporate network will implement many different network appliances or software based solutions to secure data, restrict access, prevent attacks and unwanted traffic.
Bottom line – don’t think you are done with network considerations just because “ping works” or an email comes with a statement like “your port is now open”.

I hope this post will benefit others as this issue was and will probably remain common with Exchange and other client / server services.

Don’t get timed out 🙂
Ilantz

Additional useful links and sources of data: