Wednesday, October 09, 2013

Exchange 2010 Multi-Mailbox search fails over 100mb

Hi All

We are now coming to the end of our Exchange 2010 upgrade project and everything has gone pretty well so far.

From time to time our administrators are asked to complete searches in Exchange.  This could be for a number of reasons e.g Data Protection request or a disciplinary query.

In Exchange 2007 we used to complete multi mailbox searches using a third party tool which would use MAPI to search every users' mailbox for items which matched our search terms.  This process was extremely slow (90 hours to cover 2000 mailboxes) and not particular reliable.

In Exchange 2010 Microsoft released a feature called Multi-Mailbox search.  This allowed administrators to search all of the mailboxes using the search index which is built into Exchange.  This means the searches in Exchange 2010 are much much quicker.  It also means that this search will look at attachments which is excellent, although does create far more results to review.

MSExchange Link for more information on Multi-Mailbox search

We did hit one issue whilst testing this search after all of the mailboxes had been migrated.  The search would fail against a large number of mailboxes.  It would retry 6 times and eventually fail and move onto the next folder.  This was bad news as the rest of the folder might contain extremely important information.

  • Log Name:      Application
    Source:        MSExchange MailboxSearchDate:          10/2/2013 2:05:26 PMEvent ID:      21000Task Category: (21)Level:         ErrorKeywords:      ClassicUser:          N/AComputer:      Server01.domain.comDescription:Batch copy for MailboxSearch 'Test 2' on mailbox 'Joe Blogs' was retried '5' times but failed on all instances. The last exception encountered was: 'Microsoft.Exchange.Data.Storage.PartialCompletionException: Move/Copy messages failed. ---> Microsoft.Mapi.MapiExceptionPartialCompletion: MapiExceptionPartialCompletion: Unable to copy message(s). (hr=0x40680, ec=0)Diagnostic context:    Lid: 21750     Lid: 18122   StoreEc: 0x4DA        Lid: 25510     Lid: 25270   StoreEc: 0x4DA        Lid: 19830     Lid: 25290   StoreEc: 0x4DA        Lid: 55847   EMSMDBPOOL.EcPoolSessionDoRpc called [length=45]    Lid: 43559   EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0x0][length=118][latency=0]    Lid: 23226   --- ROP Parse Start ---    Lid: 31418   --- ROP Parse Done ---    Lid: 16465     Lid: 24657   StoreEc: 0x4DA       at Microsoft.Mapi.MapiExceptionHelper.ThrowIfErrorOrWarning(String message, Int32 hresult, Boolean allowWarnings, SafeExInterfaceHandle iUnknown, Exception innerException)   at Microsoft.Mapi.MapiUnk.ThrowIfErrorOrWarning(String message, Int32 hr)   at Microsoft.Mapi.MapiFolder.CopyMessages(CopyMessagesFlags flags, MapiFolder destFolder, Byte[][] entryIds)   at Microsoft.Exchange.Data.Storage.CoreFolder.<>c__DisplayClass24.<InternalMoveOrCopyItems>b__1e()   at Microsoft.Exchange.Data.Storage.CoreFolder.ExecuteMapiGroupOperationMethod(String operationAttempted, StoreObjectId[] sourceObjectIds, MapiGroupOperation mapiGroupOperationCall)   --- End of inner exception stack trace ---   at Microsoft.Exchange.Data.Storage.CoreFolder.ExecuteMapiGroupOperationMethod(String operationAttempted, StoreObjectId[] sourceObjectIds, MapiGroupOperation mapiGroupOperationCall)   at Microsoft.Exchange.Data.Storage.CoreFolder.ExecuteMapiGroupOperationInternal(StoreObjectId[] sourceObjectIds, MapiGroupOperationInternalMethod method)'.
    Source:        MSExchange MailboxSearch
    Date:          10/2/2013 2:05:26 PM
    Event ID:      21003
    Task Category: (21)
    Level:         Warning
    Keywords:      Classic
    User:          N/A
    Computer:      Server01.domain.com
    Description:
    Batch copy for MailboxSearch 'Test 2' on mailbox 'Joe Bloggs' failed on folder 'DPATEST' failed even after retries, and it will be skipped. The exception encountered was: 'Microsoft.Exchange.Data.Storage.PartialCompletionException: Move/Copy messages failed. ---> Microsoft.Mapi.MapiExceptionPartialCompletion: MapiExceptionPartialCompletion: Unable to copy message(s). (hr=0x40680, ec=0)
    Diagnostic context:
        Lid: 21750  
        Lid: 18122   StoreEc: 0x4DA     
        Lid: 25510  
        Lid: 25270   StoreEc: 0x4DA     
        Lid: 19830  
        Lid: 25290   StoreEc: 0x4DA     
        Lid: 55847   EMSMDBPOOL.EcPoolSessionDoRpc called [length=45]
        Lid: 43559   EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0x0][length=118][latency=0]
        Lid: 23226   --- ROP Parse Start ---
        Lid: 31418   --- ROP Parse Done ---
        Lid: 16465  
        Lid: 24657   StoreEc: 0x4DA     
       at Microsoft.Mapi.MapiExceptionHelper.ThrowIfErrorOrWarning(String message, Int32 hresult, Boolean allowWarnings, SafeExInterfaceHandle iUnknown, Exception innerException)
       at Microsoft.Mapi.MapiUnk.ThrowIfErrorOrWarning(String message, Int32 hr)
       at Microsoft.Mapi.MapiFolder.CopyMessages(CopyMessagesFlags flags, MapiFolder destFolder, Byte[][] entryIds)
       at Microsoft.Exchange.Data.Storage.CoreFolder.<>c__DisplayClass24.<InternalMoveOrCopyItems>b__1e()
       at Microsoft.Exchange.Data.Storage.CoreFolder.ExecuteMapiGroupOperationMethod(String operationAttempted, StoreObjectId[] sourceObjectIds, MapiGroupOperation mapiGroupOperationCall)
       --- End of inner exception stack trace ---
       at Microsoft.Exchange.Data.Storage.CoreFolder.ExecuteMapiGroupOperationMethod(String operationAttempted, StoreObjectId[] sourceObjectIds, MapiGroupOperation mapiGroupOperationCall)
       at Microsoft.Exchange.Data.Storage.CoreFolder.ExecuteMapiGroupOperationInternal(StoreObjectId[] sourceObjectIds, MapiGroupOperationInternalMethod method)'.
    Source:        MSExchange Mid-Tier Storage
    Date:          10/2/2013 2:05:33 PM
    Event ID:      3003
    Task Category: Discovery
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      Server01.domain.com
    Description:
    Search 'SearchStatus\2e0e5de4-30a3-46db-8bff-0dd831a341c7' has following error:
    An error occurred when searching Joe Bloggs. The message is 'Move/Copy messages failed.'.


    Log Name:      Application
    Log Name:      Application

Initially we couldn't work out why this was occurring, my colleague (John Manley), looked into this further and worked out that the Discovery Search mailbox did not have items over 100mb.  This was a big find and once we had a common reason for the search failing it made the next steps easier.

Our company has a much smaller global transport limit configured in Exchange, but this is typically covering messages which were sent and received before this limit was imposed.

We tested the search with the Transport Limit off, but the same problem occurred.  We moved the Discovery Search mailbox to its own DB but the problem still occurred.

Eventually we found that although the MaxSendSize and MaxReceiveSize for every user in the Exchange Organisation was unlimited, the Discovery Search mailbox was set to 100mb.

We set this mailbox to be unlimited.  This change takes a number of hours to propagate through Exchange, but once replicated the searches begain to return results greater than 100mb......yay!

Hopefully this will save someone a few painstaking hours of trawling log files etc.

Sam

Thursday, October 03, 2013

Solarwinds - Error: "Network connection failed. HResult: No

During our Exchange 2010 upgrade project we noticed the monitoring on one server stopped working.

We use a product called Solarwinds, this uses a combination of Powershell, WMI and SNMP to monitor services on hardware and software from around the business.


We noticed that Solarwinds would return the following error message when polling this particular server

Network connection failed. HResult: No data to return. Error: Unable to connect to the specified computer, or the computer is offline.

After a quick search online we found the following Solarwinds KB

It recommends restarting the WMI service on the server. We completed this but the issue remained.

One of my learned colleagues kept searching and found that the Remote Regsitry service was set to Manual. He set it to Automatic and started the server.

Immediately the polling began to work!

So if you have this problem, check the Remote Registry service!

Sam

Cheers to John Manley for finding this!