Possible bug in Exchange 2013 CU5 Setup.exe /PrepareAD
Recently ran into an issue when attempting to run Setup.exe /PrepareAD for Exchange 2013 Cu5. As with all new Exchange updates, the expectation is that there will be schema updates included.
Upon running the Setup.exe /IAcceptExchangeServerLicenseTerms /PrepareAD command I get this error.
Performing Microsoft Exchange Server Prerequisite Check Prerequisite Analysis COMPLETED Configuring Microsoft Exchange Server Organization Preparation FAILED The following error was generated when "$error.Clear(); initialize-ExchangeUniversalGroups -DomainController $RoleDomainController -ActiveDirectorySplitPermissions $Rol eActiveDirectorySplitPermissions " was run: "Microsoft.Exchange.Data.Directory.SuitabilityDirectoryException: An Active Directory error 0x51 occurred whe n trying to check the suitability of server 'DC01.Example.com'. Error: 'Active directory response: The LDAP server is unavailable.' ---> System.DirectoryServices.Protocols.LdapException: The LDAP server is unavailable. at System.DirectoryServices.Protocols.LdapConnection.Connect() at System.DirectoryServices.Protocols.LdapConnection.BindHelper(NetworkCredential newCredential, Boolean needSetCrede ntial) at Microsoft.Exchange.Data.Directory.PooledLdapConnection.BindWithLogging() at Microsoft.Exchange.Data.Directory.PooledLdapConnection.TryBindWithRetry(Int32 maxRetries, ADErrorRecord& errorReco rd) --- End of inner exception stack trace --- at Microsoft.Exchange.Data.Directory.TopologyDiscovery.SuitabilityVerifier.CheckIsServerSuitable(String fqdn, Boolean isGlobalCatalog, NetworkCredential credentials, String& writableNC) at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType, String partit ionFqdn, ADObjectId domain, String serverName, Int32 port, NetworkCredential credential) at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType, String partit ionFqdn, NetworkCredential networkCredential, String serverName, Int32 port) at Microsoft.Exchange.Data.Directory.ADDataSession.GetConnection(String preferredServer, Boolean isWriteOperation, St ring optionalBaseDN, ADObjectId& rootId, ADScope scope) at Microsoft.Exchange.Data.Directory.ADDataSession.GetReadConnection(String preferredServer, String optionalBaseDN, A DObjectId& rootId, ADRawEntry scopeDeteriminingObject, DualSearchMode dualSearchMode) at Microsoft.Exchange.Data.Directory.ADGenericReader.GetNextResultCollection(Type controlType, DirectoryControl& resp onseControl) at Microsoft.Exchange.Data.Directory.ADPagedReader`1.GetNextResultCollection() at Microsoft.Exchange.Data.Directory.ADGenericPagedReader`1.GetNextPage() at Microsoft.Exchange.Data.Directory.ADGenericPagedReader`1.<GetEnumerator>d__0.MoveNext() at Microsoft.Exchange.Data.Directory.Recipient.ADRecipientObjectSession.<FindByAccountName>d__2`1.MoveNext() at Microsoft.Exchange.Data.Directory.Recipient.ADRecipientObjectSession.FindByAccountName[T](String domainName, Strin g accountName) at Microsoft.Exchange.Management.Tasks.InitializeExchangeUniversalGroups.InternalProcessRecord() at Microsoft.Exchange.Configuration.Tasks.Task.<ProcessRecord>b__b() at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePip elineIfFailed)". The Exchange Server setup operation didn't complete. More details can be found in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.
I looked at a network capture while attempting to run it again and saw something odd. Right at the time of the failure in the ExchangeSetup.log there’s an attempt to contact DC01.example.com a non-GC Domain Controller on port 3268. Which obviously fails.
The problem appears to occur during the execution of the command:
initialize-ExchangeUniversalGroups -DomainController $RoleDomainController -ActiveDirectorySplitPermissions $RoleActiveDirectorySplitPermissions
Looking in the log file it identifies a Preferred GC (DC02) and Preferred DC (DC01 which is the Schema Master). They are used successfully it seems earlier in earlier portions of the update. But when initialize-ExchangeUniversalGroups runs, for some reason it doesn’t use the identified GC, doesn’t even try before attempting a connection to the non-GC.
In trying to fix this, I moved the Schema Master role from DC01 to DC02 (the GC). The next time /PrepareAD worked fine. Presumably making the Schema Master a GC would have also worked. Strange problem though as I’ve done the rest of the schema updates involved in the previous 2013 CUs without any problem.
Exchange 2013 – OWA and ECP logins fail with 500 error
After troubleshooting another issue, and having one of the 2013 servers crash a few times while running diagnostics, OWA and ECP logons started showing an error.
500 Unexpected Error :( An error occurred and your request couldn’t be completed. Please try again.
Reseting IIS, restarting the servers, clearing cookies etc had no effect.
Event 4 appears in the Application log at the time of the login.
Current user: 'Example.com/Test User' Request for URL 'https://server01.example.com:444/ecp/default.aspx(https://server01/ecp/)' failed with the following error: System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.Exchange.Clients.Common.Canary15.Init(Byte[] userContextIdBinary, Byte[] timeStampBinary, String logonUniqueKey, Byte[] hashBinary, String logData) at Microsoft.Exchange.Clients.Common.Canary15..ctor(String logonUniqueKey) at Microsoft.Exchange.Clients.Common.Canary15Cookie.TryCreateFromHttpCookie(HttpCookie cookie, String logonUniqueKey, Canary15Profile profile) at Microsoft.Exchange.Clients.Common.Canary15Cookie.TryCreateFromHttpContext(HttpContext httpContext, String logOnUniqueKey, Canary15Profile profile) at Microsoft.Exchange.Management.ControlPanel.CanaryExtensions.CheckCanary15(HttpContext context, Boolean shouldRenew, String canaryName) at Microsoft.Exchange.Management.ControlPanel.CanaryExtensions.CheckCanary(HttpContext context) at Microsoft.Exchange.Management.ControlPanel.RbacModule.Application_PostAuthenticateRequest(Object sender, EventArgs e) at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) at Microsoft.Exchange.Clients.Common.Canary15.Init(Byte[] userContextIdBinary, Byte[] timeStampBinary, String logonUniqueKey, Byte[] hashBinary, String logData) at Microsoft.Exchange.Clients.Common.Canary15..ctor(String logonUniqueKey) at Microsoft.Exchange.Clients.Common.Canary15Cookie.TryCreateFromHttpCookie(HttpCookie cookie, String logonUniqueKey, Canary15Profile profile) at Microsoft.Exchange.Clients.Common.Canary15Cookie.TryCreateFromHttpContext(HttpContext httpContext, String logOnUniqueKey, Canary15Profile profile) at Microsoft.Exchange.Management.ControlPanel.CanaryExtensions.CheckCanary15(HttpContext context, Boolean shouldRenew, String canaryName) at Microsoft.Exchange.Management.ControlPanel.CanaryExtensions.CheckCanary(HttpContext context) at Microsoft.Exchange.Management.ControlPanel.RbacModule.Application_PostAuthenticateRequest(Object sender, EventArgs e) at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
The error appears to be related to corrupt attributes in Active Directory, specifically under CN=Client Access,CN=<org name>,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=<domain>. The attribute msExchCanaryData0, or msExchCanaryData1, 2, 3, etc. can contain bad data.
As always, be safe, have an AD backup you can rely on. Then proceed to to clear the value of all the msExchCanaryData# attributes (shown above in ADSIEdit). Then the App Pool(s) for MSExchangeECPAppPool and MSExchangeOWAAppPool need to be recycled by going into IIS Manager and right-clicking each pool, then choosing “Recycle…” At this point all was sorted out for me.
Found info regarding the issue on TechNet: http://social.technet.microsoft.com/Forums/exchange/en-US/777b51ee-330d-43cc-a56e-4614d44aed7b/unable-to-access-owa-or-ecp-something-went-wrong-or-500-unexpected-error?forum=exchangesvrclients
Outlook prompts for password? Maybe it’s your LMCompatabilityLevel.
I’ve spent time off and on over months trying to track down why certain Office 2013 apps (Lync and Outlook) would unexpectedly prompt for passwords, at startup, or sometimes in the middle of normal operation. I’d looked at all sorts of things, internal and external authentication types, kernel mode authentication in IIS, whether there was a saved password, was the client loading a shared mailbox? None of these were the cause.
I finally stumbled upon an 2003 era document http://support.microsoft.com/kb/820281 which indicates that there can be a prompt if the LMCompatabilityLevel on the workstation is set below 2. Low and behold…
What is that doing there?!?!
The workstations had this set in local group policy, so simply deleting the key restored the default of using LM level 3. As a more global, and permanent fix, the setting could be applied via Domain Group Policy.
Lync had been attempting to grab HTTP resources via a web proxy, and failing to pass credentials to it. Likewise Outlook was passing LANMAN instead of NTLM or NTLMv2 and failing. I’d looked at the traffic in Wireshark and used Outlook logging, but it wasn’t apparent in either case that this was the problem.
Exchange 2013 install fails when Scripting Agent is enabled in organization
Found a bug in the Exchange 2013 install, similar to 2010. It appears that when the Scripting Agent is enabled in your Exchange org (via Enable-CmdletExtensionAgent “Scripting Agent”) the Exchange 2013 installer will fail with the error below.
Text:
Error:
The following error was generated when “$error.Clear();
Set-WERRegistryMarkers;
” was run: “Provisioning layer initialization failed: ‘”Scripting Agent initialization failed: “File is not found: ‘E:\Program Files\Microsoft\Exchange Server\V15\Bin\CmdletExtensionAgents\ScriptingAgentConfig.xml’.””‘”.
There are a few ways to work around the issue.
- Disable the Scripting Agent extension
Disable-CmdletExtensionAgent “Scripting Agent”OR
- Copy the ScriptingAgentConfig.xml file to the “(intended install drive letter):\Program Files\Microsoft\Exchange Server\V15\Bin\CmdletExtensionAgents” folder on the new server and go through with the install as normal.
I’ve tried copying the current ScriptingAgentConfig.xml file to the (install media)\serverroles\common\cmdletextensionagents folder, but it doesn’t seem to get moved over during the install. That would be ideal in my situation as I don’t edit the config often, if at all.
Exchange 2013 Content Index failure causes stalled Mailbox Migration
Been trying to migrate some mailboxes to from 2010 to 2013 for testing, and some small/empty ones move fine, but larger mailboxes fail to move, showing long “Stalled duration” in the Migration Batch details.
Queued duration: 00:00:08 In-progress duration: 00:11:51 Synced duration: Stalled duration: 03:16:33
The issue is caused by a Content Indexing failure on the target database, caused by what is apparently a missing element of the AD Prep portion of setup.
The problem is documented at http://support.microsoft.com/kb/2807668
I’m running CU1 (released April 2013) and the KB article was posted in January. so it’s not clear why this still wasn’t addressed.
Of the 2 fixes listed in the KB, I chose Method 1 because it felt like a more reliable, permanent fix. I didn’t want to run into this again when adding a new mailbox server.
The KB directs you to add a ContentSubmitters group to AD, set permissions on it, and restart Exchange search services. It took a few more steps in my case to actually solve the issue. I also had to stop the services, remove the current content index and start services again.
The steps all together were:
- Add a ContentSubmitters group to AD.
I did this manually, creating it as a Security enabled Universal group. - Assign Network Service and Administrators groups the “Full Access” permission (I think they mean “Full Control” permission).
Ended up using the PowerShell commands
Add-ADPermission -Id ContentSubmitters -User “Network Service” -Access Rights GenericAll
Add-ADPermission -Id ContentSubmitters -User “Administrators” -Access Rights GenericAll - Stop the Microsoft Exchange Search and Microsoft Exchange Search Host Controller services
- Delete the Content Index data for the affected database(s)
This is the folder with a GUID name inside the folder where the mailbox database lives. - Start the Microsoft Exchange Search and Microsoft Exchange Search Host Controller services.
After a few minutes, the status of the content index went from FailedAndSuspended to Crawling, at which point the mailbox move resumed.
Exchange 2010 Journaling: Stuck messages in Submission queue
Update: This appears to be a documented bug in KB2681250
However, the KB notes that it should be patched as of SP2 RU3 and it still appears to be an issue in SP3. Opening support case now.
Further update: Looks like this is a distinct bug from the reported on in the above KB. Scheduled to be patched in SP3 RU3. Specifically, my issue seems to deal mostly with meeting invite replies (Accepted/Declined, etc.)
There’s an issue that appears to pop up periodically with Exchange Journaling which annecdotally, is due to having a BES server attached to your Exchange environment or if a user has a broken signature in the Outlook client. It appears some or all of these messages are Calendaring responses with a subject like “Accepted: Meeting Name” or “Declined: Meeting Name”
Messages get stuck in “Retry” in the Submission queue, which shows as an Event 9213 in the Application log:
Log Name: Application
Source: MSExchangeTransport
Date: <date>
Event ID: 9213
Task Category: Categorizer
Level: Error
Keywords: Classic
User: N/A
Computer: <servername>
Description:
A non-expirable message with the Internal Message ID 12345678 could not be categorized. This message may be a journal report or other system message. The message will remain in the queue until administrative action is taken to resolve the error. Other messages may also have encountered this error. To further diagnose the error, use the Queue Viewer or the Exchange Mail Flow Troubleshooter.
Looking at the message info it shows:
Last Error: 400 4.4.7 The server responded with: 550 5.6.0 M2MCVT.StorageError; storage error in content conversion. The failure was replaced by a retry response because the message was marked for retry if rejected.
A note about the line “The server responded with: 550 5.6.0…” This appears to be the local Exchange server, not the intended destination server. In context, since the message is in the “Submission” queue and not in one of the destination queues, that makes sense.
Without a way to specifically stop the messages, they can at least be manually removed from the queue using:
Get-Message -Server <servername> -Filter {FromAddress -eq "<>" -and Status -eq 'Retry'} | Remove-Message
Note, the second part of the filter “Status -eq “Retry”. Normal Journaling messages also fly by with blank from addresses and this avoids removing any of those.
Providing Exchange 2010 Activesync HA with multiple sites
Update: a bug in Exchange 2010 SP3 breaks the mechanism described below. See http://msitpros.com/?p=1818 for a good writeup. Apparently slated for a fix in RU2. As a workaround, I changed the activesync.company.com CNAME to point only to the activesync-site1.company.com record and the errors are resolved.
Assuming you already have a solid handle on DR for your mailboxes, the next most important thing is client connectivity. Many DR solutions for AS (and OWA too) involve repointing DNS to the DR site in case of an disaster. Changes to our external DNS (it’s hosted) take well over an hour to propagate. And that problem can be avoided.
By using extra DNS entries and the built-in proxying and redirection functions in the Activesync service, AS clients can seamlessly failover between datacenters. It requires adding a one extra level to the namespace meaning more Subject Alternative Names on the SSL cert. Beyond that, I’ve perhaps over-complicated the namespace a bit more here, but I don’t like changing A records, so I have some CNAMEs that can be shifted in a pinch without modifying the As.
The idea is that regardless of what name is used to access the Activesync service, the CAS server will verify that it is in the same site as the desired mailbox and if not, based on the version of the client:
- EAS 14.0+ clients will get a 451 redirect to the externalURL of the EAS Virtual Directory on a CAS in the same site as the current mailbox location.
- EAS 12.1 or earlier clients will be proxied to the CAS in the correct site if the /proxy directory is set for Windows Authentication.
Note: In my case, both datacenters are connected internally by a fast link, so any proxied traffic is not an issue.
The setup is as follows:
Names highlighted in orange need to be listed as SANs or as the common name of the SSL cert.
These two DNS A records are specific to the two sites and used as the domain name of the external Activesync URLs for the two locations. (these names are used by the client if redirected)
A activesync-site1.company.com 192.168.1.100
A activesync-site2.company.com 192.168.2.100
These overloaded A records are used for the round robin.
A activesync-allsites.company.com 192.168.1.100
A activesync-allsites.company.com 192.168.2.100
The last record is just for my convenience. I want to point it directly at site2 if site1 was down, otherwise it’s always pointed at activesync-allsites.company.com.
CNAME activesync.company.com -> activesync-allsites.company.com
When not using Autodiscover to configure the AS device, it makes it easy to give the user the server name activesync.company.com and letting the back-end functions take care of getting the device connected to the right place.
ril3y 