Recently ran into an issue when attempting to run Setup.exe /PrepareAD for Exchange 2013 Cu5. As with all new Exchange updates, the expectation is that there will be schema updates included.
Upon running the Setup.exe /IAcceptExchangeServerLicenseTerms /PrepareAD command I get this error.
Performing Microsoft Exchange Server Prerequisite Check Prerequisite Analysis COMPLETED Configuring Microsoft Exchange Server Organization Preparation FAILED The following error was generated when "$error.Clear(); initialize-ExchangeUniversalGroups -DomainController $RoleDomainController -ActiveDirectorySplitPermissions $Rol eActiveDirectorySplitPermissions " was run: "Microsoft.Exchange.Data.Directory.SuitabilityDirectoryException: An Active Directory error 0x51 occurred whe n trying to check the suitability of server 'DC01.Example.com'. Error: 'Active directory response: The LDAP server is unavailable.' ---> System.DirectoryServices.Protocols.LdapException: The LDAP server is unavailable. at System.DirectoryServices.Protocols.LdapConnection.Connect() at System.DirectoryServices.Protocols.LdapConnection.BindHelper(NetworkCredential newCredential, Boolean needSetCrede ntial) at Microsoft.Exchange.Data.Directory.PooledLdapConnection.BindWithLogging() at Microsoft.Exchange.Data.Directory.PooledLdapConnection.TryBindWithRetry(Int32 maxRetries, ADErrorRecord& errorReco rd) --- End of inner exception stack trace --- at Microsoft.Exchange.Data.Directory.TopologyDiscovery.SuitabilityVerifier.CheckIsServerSuitable(String fqdn, Boolean isGlobalCatalog, NetworkCredential credentials, String& writableNC) at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType, String partit ionFqdn, ADObjectId domain, String serverName, Int32 port, NetworkCredential credential) at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType, String partit ionFqdn, NetworkCredential networkCredential, String serverName, Int32 port) at Microsoft.Exchange.Data.Directory.ADDataSession.GetConnection(String preferredServer, Boolean isWriteOperation, St ring optionalBaseDN, ADObjectId& rootId, ADScope scope) at Microsoft.Exchange.Data.Directory.ADDataSession.GetReadConnection(String preferredServer, String optionalBaseDN, A DObjectId& rootId, ADRawEntry scopeDeteriminingObject, DualSearchMode dualSearchMode) at Microsoft.Exchange.Data.Directory.ADGenericReader.GetNextResultCollection(Type controlType, DirectoryControl& resp onseControl) at Microsoft.Exchange.Data.Directory.ADPagedReader`1.GetNextResultCollection() at Microsoft.Exchange.Data.Directory.ADGenericPagedReader`1.GetNextPage() at Microsoft.Exchange.Data.Directory.ADGenericPagedReader`1.<GetEnumerator>d__0.MoveNext() at Microsoft.Exchange.Data.Directory.Recipient.ADRecipientObjectSession.<FindByAccountName>d__2`1.MoveNext() at Microsoft.Exchange.Data.Directory.Recipient.ADRecipientObjectSession.FindByAccountName[T](String domainName, Strin g accountName) at Microsoft.Exchange.Management.Tasks.InitializeExchangeUniversalGroups.InternalProcessRecord() at Microsoft.Exchange.Configuration.Tasks.Task.<ProcessRecord>b__b() at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePip elineIfFailed)". The Exchange Server setup operation didn't complete. More details can be found in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.
I looked at a network capture while attempting to run it again and saw something odd. Right at the time of the failure in the ExchangeSetup.log there’s an attempt to contact DC01.example.com a non-GC Domain Controller on port 3268. Which obviously fails.
The problem appears to occur during the execution of the command:
initialize-ExchangeUniversalGroups -DomainController $RoleDomainController -ActiveDirectorySplitPermissions $RoleActiveDirectorySplitPermissions
Looking in the log file it identifies a Preferred GC (DC02) and Preferred DC (DC01 which is the Schema Master). They are used successfully it seems earlier in earlier portions of the update. But when initialize-ExchangeUniversalGroups runs, for some reason it doesn’t use the identified GC, doesn’t even try before attempting a connection to the non-GC.
In trying to fix this, I moved the Schema Master role from DC01 to DC02 (the GC). The next time /PrepareAD worked fine. Presumably making the Schema Master a GC would have also worked. Strange problem though as I’ve done the rest of the schema updates involved in the previous 2013 CUs without any problem.
After troubleshooting another issue, and having one of the 2013 servers crash a few times while running diagnostics, OWA and ECP logons started showing an error.
500 Unexpected Error :( An error occurred and your request couldn’t be completed. Please try again.
Reseting IIS, restarting the servers, clearing cookies etc had no effect.
Event 4 appears in the Application log at the time of the login.
Current user: 'Example.com/Test User' Request for URL 'https://server01.example.com:444/ecp/default.aspx(https://server01/ecp/)' failed with the following error: System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.Exchange.Clients.Common.Canary15.Init(Byte userContextIdBinary, Byte timeStampBinary, String logonUniqueKey, Byte hashBinary, String logData) at Microsoft.Exchange.Clients.Common.Canary15..ctor(String logonUniqueKey) at Microsoft.Exchange.Clients.Common.Canary15Cookie.TryCreateFromHttpCookie(HttpCookie cookie, String logonUniqueKey, Canary15Profile profile) at Microsoft.Exchange.Clients.Common.Canary15Cookie.TryCreateFromHttpContext(HttpContext httpContext, String logOnUniqueKey, Canary15Profile profile) at Microsoft.Exchange.Management.ControlPanel.CanaryExtensions.CheckCanary15(HttpContext context, Boolean shouldRenew, String canaryName) at Microsoft.Exchange.Management.ControlPanel.CanaryExtensions.CheckCanary(HttpContext context) at Microsoft.Exchange.Management.ControlPanel.RbacModule.Application_PostAuthenticateRequest(Object sender, EventArgs e) at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) at Microsoft.Exchange.Clients.Common.Canary15.Init(Byte userContextIdBinary, Byte timeStampBinary, String logonUniqueKey, Byte hashBinary, String logData) at Microsoft.Exchange.Clients.Common.Canary15..ctor(String logonUniqueKey) at Microsoft.Exchange.Clients.Common.Canary15Cookie.TryCreateFromHttpCookie(HttpCookie cookie, String logonUniqueKey, Canary15Profile profile) at Microsoft.Exchange.Clients.Common.Canary15Cookie.TryCreateFromHttpContext(HttpContext httpContext, String logOnUniqueKey, Canary15Profile profile) at Microsoft.Exchange.Management.ControlPanel.CanaryExtensions.CheckCanary15(HttpContext context, Boolean shouldRenew, String canaryName) at Microsoft.Exchange.Management.ControlPanel.CanaryExtensions.CheckCanary(HttpContext context) at Microsoft.Exchange.Management.ControlPanel.RbacModule.Application_PostAuthenticateRequest(Object sender, EventArgs e) at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
The error appears to be related to corrupt attributes in Active Directory, specifically under CN=Client Access,CN=<org name>,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=<domain>. The attribute msExchCanaryData0, or msExchCanaryData1, 2, 3, etc. can contain bad data.
As always, be safe, have an AD backup you can rely on. Then proceed to to clear the value of all the msExchCanaryData# attributes (shown above in ADSIEdit). Then the App Pool(s) for MSExchangeECPAppPool and MSExchangeOWAAppPool need to be recycled by going into IIS Manager and right-clicking each pool, then choosing “Recycle…” At this point all was sorted out for me.
Found info regarding the issue on TechNet: http://social.technet.microsoft.com/Forums/exchange/en-US/777b51ee-330d-43cc-a56e-4614d44aed7b/unable-to-access-owa-or-ecp-something-went-wrong-or-500-unexpected-error?forum=exchangesvrclients
I’ve spent time off and on over months trying to track down why certain Office 2013 apps (Lync and Outlook) would unexpectedly prompt for passwords, at startup, or sometimes in the middle of normal operation. I’d looked at all sorts of things, internal and external authentication types, kernel mode authentication in IIS, whether there was a saved password, was the client loading a shared mailbox? None of these were the cause.
I finally stumbled upon an 2003 era document http://support.microsoft.com/kb/820281 which indicates that there can be a prompt if the LMCompatabilityLevel on the workstation is set below 2. Low and behold…
The workstations had this set in local group policy, so simply deleting the key restored the default of using LM level 3. As a more global, and permanent fix, the setting could be applied via Domain Group Policy.
Lync had been attempting to grab HTTP resources via a web proxy, and failing to pass credentials to it. Likewise Outlook was passing LANMAN instead of NTLM or NTLMv2 and failing. I’d looked at the traffic in Wireshark and used Outlook logging, but it wasn’t apparent in either case that this was the problem.
Found a bug in the Exchange 2013 install, similar to 2010. It appears that when the Scripting Agent is enabled in your Exchange org (via Enable-CmdletExtensionAgent “Scripting Agent”) the Exchange 2013 installer will fail with the error below.
The following error was generated when “$error.Clear();
” was run: “Provisioning layer initialization failed: ‘”Scripting Agent initialization failed: “File is not found: ‘E:\Program Files\Microsoft\Exchange Server\V15\Bin\CmdletExtensionAgents\ScriptingAgentConfig.xml’.””‘”.
There are a few ways to work around the issue.
- Disable the Scripting Agent extension
Disable-CmdletExtensionAgent “Scripting Agent”
- Copy the ScriptingAgentConfig.xml file to the “(intended install drive letter):\Program Files\Microsoft\Exchange Server\V15\Bin\CmdletExtensionAgents” folder on the new server and go through with the install as normal.
I’ve tried copying the current ScriptingAgentConfig.xml file to the (install media)\serverroles\common\cmdletextensionagents folder, but it doesn’t seem to get moved over during the install. That would be ideal in my situation as I don’t edit the config often, if at all.
Been trying to migrate some mailboxes to from 2010 to 2013 for testing, and some small/empty ones move fine, but larger mailboxes fail to move, showing long “Stalled duration” in the Migration Batch details.
Queued duration: 00:00:08 In-progress duration: 00:11:51 Synced duration: Stalled duration: 03:16:33
The issue is caused by a Content Indexing failure on the target database, caused by what is apparently a missing element of the AD Prep portion of setup.
The problem is documented at http://support.microsoft.com/kb/2807668
I’m running CU1 (released April 2013) and the KB article was posted in January. so it’s not clear why this still wasn’t addressed.
Of the 2 fixes listed in the KB, I chose Method 1 because it felt like a more reliable, permanent fix. I didn’t want to run into this again when adding a new mailbox server.
The KB directs you to add a ContentSubmitters group to AD, set permissions on it, and restart Exchange search services. It took a few more steps in my case to actually solve the issue. I also had to stop the services, remove the current content index and start services again.
The steps all together were:
- Add a ContentSubmitters group to AD.
I did this manually, creating it as a Security enabled Universal group.
- Assign Network Service and Administrators groups the “Full Access” permission (I think they mean “Full Control” permission).
Ended up using the PowerShell commands
Add-ADPermission -Id ContentSubmitters -User “Network Service” -Access Rights GenericAll
Add-ADPermission -Id ContentSubmitters -User “Administrators” -Access Rights GenericAll
- Stop the Microsoft Exchange Search and Microsoft Exchange Search Host Controller services
- Delete the Content Index data for the affected database(s)
This is the folder with a GUID name inside the folder where the mailbox database lives.
- Start the Microsoft Exchange Search and Microsoft Exchange Search Host Controller services.
After a few minutes, the status of the content index went from FailedAndSuspended to Crawling, at which point the mailbox move resumed.