Archive | MSExchange RSS for this section

Possible bug in Exchange 2013 CU5 Setup.exe /PrepareAD

Recently ran into an issue when attempting to run Setup.exe /PrepareAD for Exchange 2013 Cu5.  As with all new Exchange updates, the expectation is that there will be schema updates included.

Upon running the Setup.exe /IAcceptExchangeServerLicenseTerms /PrepareAD command I get this error.

Performing Microsoft Exchange Server Prerequisite Check

Prerequisite Analysis COMPLETED

Configuring Microsoft Exchange Server

Organization Preparation FAILED
The following error was generated when "$error.Clear();
initialize-ExchangeUniversalGroups -DomainController $RoleDomainController -ActiveDirectorySplitPermissions $Rol
eActiveDirectorySplitPermissions

" was run: "Microsoft.Exchange.Data.Directory.SuitabilityDirectoryException: An Active Directory error 0x51 occurred whe
n trying to check the suitability of server 'DC01.Example.com'. Error: 'Active directory response: The LDAP server
is unavailable.' ---> System.DirectoryServices.Protocols.LdapException: The LDAP server is unavailable.
at System.DirectoryServices.Protocols.LdapConnection.Connect()
at System.DirectoryServices.Protocols.LdapConnection.BindHelper(NetworkCredential newCredential, Boolean needSetCrede
ntial)
at Microsoft.Exchange.Data.Directory.PooledLdapConnection.BindWithLogging()
at Microsoft.Exchange.Data.Directory.PooledLdapConnection.TryBindWithRetry(Int32 maxRetries, ADErrorRecord& errorReco
rd)
--- End of inner exception stack trace ---
at Microsoft.Exchange.Data.Directory.TopologyDiscovery.SuitabilityVerifier.CheckIsServerSuitable(String fqdn, Boolean
isGlobalCatalog, NetworkCredential credentials, String& writableNC)
at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType, String partit
ionFqdn, ADObjectId domain, String serverName, Int32 port, NetworkCredential credential)
at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType, String partit
ionFqdn, NetworkCredential networkCredential, String serverName, Int32 port)
at Microsoft.Exchange.Data.Directory.ADDataSession.GetConnection(String preferredServer, Boolean isWriteOperation, St
ring optionalBaseDN, ADObjectId& rootId, ADScope scope)
at Microsoft.Exchange.Data.Directory.ADDataSession.GetReadConnection(String preferredServer, String optionalBaseDN, A
DObjectId& rootId, ADRawEntry scopeDeteriminingObject, DualSearchMode dualSearchMode)
at Microsoft.Exchange.Data.Directory.ADGenericReader.GetNextResultCollection(Type controlType, DirectoryControl& resp
onseControl)
at Microsoft.Exchange.Data.Directory.ADPagedReader`1.GetNextResultCollection()
at Microsoft.Exchange.Data.Directory.ADGenericPagedReader`1.GetNextPage()
at Microsoft.Exchange.Data.Directory.ADGenericPagedReader`1.<GetEnumerator>d__0.MoveNext()
at Microsoft.Exchange.Data.Directory.Recipient.ADRecipientObjectSession.<FindByAccountName>d__2`1.MoveNext()
at Microsoft.Exchange.Data.Directory.Recipient.ADRecipientObjectSession.FindByAccountName[T](String domainName, Strin
g accountName)
at Microsoft.Exchange.Management.Tasks.InitializeExchangeUniversalGroups.InternalProcessRecord()
at Microsoft.Exchange.Configuration.Tasks.Task.<ProcessRecord>b__b()
at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePip
elineIfFailed)".


The Exchange Server setup operation didn't complete. More details can be found in ExchangeSetup.log located in the
<SystemDrive>:\ExchangeSetupLogs folder.

I looked at a network capture while attempting to run it again and saw something odd. Right at the time of the failure in the ExchangeSetup.log there’s an attempt to contact DC01.example.com a non-GC Domain Controller on port 3268.  Which obviously fails.

The problem appears to occur during the execution of the command:
initialize-ExchangeUniversalGroups -DomainController $RoleDomainController -ActiveDirectorySplitPermissions $RoleActiveDirectorySplitPermissions

Looking in the log file it identifies a Preferred GC (DC02) and Preferred DC (DC01 which is the Schema Master). They are used successfully it seems earlier in earlier portions of the update.  But when initialize-ExchangeUniversalGroups runs, for some reason it doesn’t use the identified GC, doesn’t even try before attempting a connection to the non-GC.

In trying to fix this, I moved the Schema Master role from DC01 to DC02 (the GC).  The next time /PrepareAD worked fine.  Presumably making the Schema Master a GC would have also worked.  Strange problem though as I’ve done the rest of the schema updates involved in the previous 2013 CUs without any problem.

 

Exchange 2013 – OWA and ECP logins fail with 500 error

exchange-2013-500-error

After troubleshooting another issue, and having one of the 2013 servers crash a few times while running diagnostics, OWA and ECP logons started showing an error.

500 Unexpected Error :( An error occurred and your request couldn’t be completed. Please try again.

Reseting IIS, restarting the servers, clearing cookies etc had no effect.

Event 4 appears in the Application log at the time of the login.

Current user: 'Example.com/Test User'
Request for URL 'https://server01.example.com:444/ecp/default.aspx(https://server01/ecp/)' failed with the following error:
System.NullReferenceException: Object reference not set to an instance of an object.
 at Microsoft.Exchange.Clients.Common.Canary15.Init(Byte[] userContextIdBinary, Byte[] timeStampBinary, String logonUniqueKey, Byte[] hashBinary, String logData)
 at Microsoft.Exchange.Clients.Common.Canary15..ctor(String logonUniqueKey)
 at Microsoft.Exchange.Clients.Common.Canary15Cookie.TryCreateFromHttpCookie(HttpCookie cookie, String logonUniqueKey, Canary15Profile profile)
 at Microsoft.Exchange.Clients.Common.Canary15Cookie.TryCreateFromHttpContext(HttpContext httpContext, String logOnUniqueKey, Canary15Profile profile)
 at Microsoft.Exchange.Management.ControlPanel.CanaryExtensions.CheckCanary15(HttpContext context, Boolean shouldRenew, String canaryName)
 at Microsoft.Exchange.Management.ControlPanel.CanaryExtensions.CheckCanary(HttpContext context)
 at Microsoft.Exchange.Management.ControlPanel.RbacModule.Application_PostAuthenticateRequest(Object sender, EventArgs e)
 at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
 at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
 at Microsoft.Exchange.Clients.Common.Canary15.Init(Byte[] userContextIdBinary, Byte[] timeStampBinary, String logonUniqueKey, Byte[] hashBinary, String logData)
 at Microsoft.Exchange.Clients.Common.Canary15..ctor(String logonUniqueKey)
 at Microsoft.Exchange.Clients.Common.Canary15Cookie.TryCreateFromHttpCookie(HttpCookie cookie, String logonUniqueKey, Canary15Profile profile)
 at Microsoft.Exchange.Clients.Common.Canary15Cookie.TryCreateFromHttpContext(HttpContext httpContext, String logOnUniqueKey, Canary15Profile profile)
 at Microsoft.Exchange.Management.ControlPanel.CanaryExtensions.CheckCanary15(HttpContext context, Boolean shouldRenew, String canaryName)
 at Microsoft.Exchange.Management.ControlPanel.CanaryExtensions.CheckCanary(HttpContext context)
 at Microsoft.Exchange.Management.ControlPanel.RbacModule.Application_PostAuthenticateRequest(Object sender, EventArgs e)
 at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
 at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

The error appears to be related to corrupt attributes in Active Directory, specifically under CN=Client Access,CN=<org name>,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=<domain>. The attribute msExchCanaryData0, or msExchCanaryData1, 2, 3, etc.  can contain bad data.

500-error-adsiedit

As always, be safe, have an AD backup you can rely on.  Then proceed to to clear the value of all the msExchCanaryData# attributes (shown above in ADSIEdit).  Then the App Pool(s) for MSExchangeECPAppPool and MSExchangeOWAAppPool need to be recycled by going into IIS Manager and right-clicking each pool, then choosing “Recycle…” At this point all was sorted out for me.

500-error-apppools

 

Found info regarding the issue on TechNet:  http://social.technet.microsoft.com/Forums/exchange/en-US/777b51ee-330d-43cc-a56e-4614d44aed7b/unable-to-access-owa-or-ecp-something-went-wrong-or-500-unexpected-error?forum=exchangesvrclients

Outlook prompts for password? Maybe it’s your LMCompatabilityLevel.

outlook2013-password-prompt

I’ve spent time off and on over months trying to track down why certain Office 2013 apps (Lync and Outlook) would unexpectedly prompt for passwords, at startup, or sometimes in the middle of normal operation.  I’d looked at all sorts of things, internal and external authentication types, kernel mode authentication in IIS, whether there was a saved password, was the client loading a shared mailbox?  None of these were the cause.

I finally stumbled upon an 2003 era document  http://support.microsoft.com/kb/820281 which indicates that there can be a prompt if the LMCompatabilityLevel on the workstation is set below 2.  Low and behold…

lmcompatabilitylevel

What is that doing there?!?!

The workstations had this set in local group policy, so simply deleting the key restored the default of using LM level 3.  As a more global, and permanent fix, the setting could be applied via Domain Group Policy.

Lync had been attempting to grab HTTP resources via a web proxy, and failing to pass credentials to it.  Likewise Outlook was passing LANMAN instead of NTLM or NTLMv2 and failing. I’d looked at the traffic in Wireshark and used Outlook logging, but it wasn’t apparent in either case that this was the problem.

Exchange 2013 install fails when Scripting Agent is enabled in organization

Found a bug in the Exchange 2013 install, similar to 2010.  It appears that when the Scripting Agent is enabled in your Exchange org (via Enable-CmdletExtensionAgent “Scripting Agent”) the Exchange 2013 installer will fail with the error below.

ScriptingAgentConfigxml_error

Text:

Error:
The following error was generated when “$error.Clear();
Set-WERRegistryMarkers;
” was run: “Provisioning layer initialization failed: ‘”Scripting Agent initialization failed: “File is not found: ‘E:\Program Files\Microsoft\Exchange Server\V15\Bin\CmdletExtensionAgents\ScriptingAgentConfig.xml’.””‘”.

There are a few ways to work around the issue.

  1. Disable the Scripting Agent extension
    Disable-CmdletExtensionAgent Scripting Agent

    OR

  2. Copy the ScriptingAgentConfig.xml file to the “(intended install drive letter):\Program Files\Microsoft\Exchange Server\V15\Bin\CmdletExtensionAgents” folder on the new server and go through with the install as normal.

I’ve tried copying the current  ScriptingAgentConfig.xml file to the (install media)\serverroles\common\cmdletextensionagents folder, but it doesn’t seem to get moved over during the install.  That would be ideal in my situation as I don’t edit the config often, if at all.

Exchange 2013 Content Index failure causes stalled Mailbox Migration

Been trying to migrate some mailboxes to from 2010 to 2013 for testing, and some small/empty ones move fine, but larger mailboxes fail to move, showing long “Stalled duration” in the Migration Batch details.

Queued duration: 00:00:08 
In-progress duration: 00:11:51 
Synced duration: 
Stalled duration: 03:16:33

The issue is caused by a Content Indexing failure on the target database, caused by what is apparently a missing element of the AD Prep portion of setup.

The problem is documented at http://support.microsoft.com/kb/2807668

I’m running CU1 (released April 2013) and the KB article was posted in January. so it’s not clear why this still wasn’t addressed.

Of the 2 fixes listed in the KB, I chose Method 1 because it felt like a more reliable, permanent fix.  I didn’t want to run into this again when adding a new mailbox server.

The KB directs you to add a ContentSubmitters group to AD, set permissions on it, and restart Exchange search services.  It took a few more steps in my case to actually solve the issue.  I also had to stop the services, remove the current content index and start services again.

The steps all together were:

  1. Add a ContentSubmitters group to AD.
    I did this manually, creating it as a Security enabled Universal group.
  2. Assign Network Service and Administrators groups the “Full Access” permission (I think they mean “Full Control” permission).
    Ended up using the PowerShell commands
    Add-ADPermission -Id ContentSubmitters -User “Network Service” -Access Rights GenericAll
    Add-ADPermission -Id ContentSubmitters -User “Administrators” -Access Rights GenericAll
  3. Stop the Microsoft Exchange Search and Microsoft Exchange Search Host Controller services
  4. Delete the Content Index data for the affected database(s)
    This is the folder with a GUID name inside the folder where the mailbox database lives.
  5. Start the  Microsoft Exchange Search and Microsoft Exchange Search Host Controller services.

After a few minutes, the status of the content index went from FailedAndSuspended to Crawling, at which point the mailbox move resumed.

Exchange 2010 Journaling: Stuck messages in Submission queue

Update: This appears to be a documented bug in  KB2681250
However, the KB notes that it should be patched as of SP2 RU3 and it still appears to be an issue in SP3.  Opening support case now.

Further update: Looks like this is a distinct bug from the reported on in the above KB.  Scheduled to be patched in SP3 RU3.  Specifically, my issue seems to deal mostly with meeting invite replies (Accepted/Declined, etc.)

There’s an issue that appears to pop up periodically with Exchange Journaling which annecdotally, is due to having a BES server attached to your Exchange environment or  if a user has a broken signature in the Outlook client.  It appears some or all of these messages are Calendaring responses with a subject like “Accepted: Meeting Name” or “Declined: Meeting Name”

Messages get stuck in “Retry” in the Submission queue, which shows as an Event 9213 in the Application log:

Log Name: Application
Source: MSExchangeTransport
Date: <date>
Event ID: 9213
Task Category: Categorizer
Level: Error
Keywords: Classic
User: N/A
Computer: <servername>
Description:
A non-expirable message with the Internal Message ID 12345678 could not be categorized. This message may be a journal report or other system message. The message will remain in the queue until administrative action is taken to resolve the error. Other messages may also have encountered this error. To further diagnose the error, use the Queue Viewer or the Exchange Mail Flow Troubleshooter.

Looking at the message info it shows:

Last Error: 400 4.4.7 The server responded with: 550 5.6.0 M2MCVT.StorageError; storage error in content conversion. The failure was replaced by a retry response because the message was marked for retry if rejected.

A note about the line “The server responded with: 550 5.6.0…”  This appears to be the local Exchange server, not the intended destination server.  In context, since the message is in the “Submission” queue and not in one of the destination queues, that makes sense.

Without a way to specifically stop the messages, they can at least be manually removed from the queue using:

Get-Message -Server <servername> -Filter {FromAddress -eq "<>" -and Status -eq 'Retry'} | Remove-Message

Note, the second part of the filter “Status -eq “Retry”.  Normal Journaling messages also fly by with blank from addresses and this avoids removing any of those.