Exchange 2010 Multi-Mailbox search fails over 100mb

Hi All

We are now coming to the end of our Exchange 2010 upgrade project and everything has gone pretty well so far.

From time to time our administrators are asked to complete searches in Exchange.  This could be for a number of reasons e.g Data Protection request or a disciplinary query.

In Exchange 2007 we used to complete multi mailbox searches using a third party tool which would use MAPI to search every users' mailbox for items which matched our search terms.  This process was extremely slow (90 hours to cover 2000 mailboxes) and not particular reliable.

In Exchange 2010 Microsoft released a feature called Multi-Mailbox search.  This allowed administrators to search all of the mailboxes using the search index which is built into Exchange.  This means the searches in Exchange 2010 are much much quicker.  It also means that this search will look at attachments which is excellent, although does create far more results to review.

MSExchange Link for more information on Multi-Mailbox search

We did hit one issue whilst testing this search after all of the mailboxes had been migrated.  The search would fail against a large number of mailboxes.  It would retry 6 times and eventually fail and move onto the next folder.  This was bad news as the rest of the folder might contain extremely important information.

• Log Name:      Application
  Source:        MSExchange MailboxSearchDate:          10/2/2013 2:05:26 PMEvent ID:      21000Task Category: (21)Level:         ErrorKeywords:      ClassicUser:          N/AComputer:      Server01.domain.comDescription:Batch copy for MailboxSearch 'Test 2' on mailbox 'Joe Blogs' was retried '5' times but failed on all instances. The last exception encountered was: 'Microsoft.Exchange.Data.Storage.PartialCompletionException: Move/Copy messages failed. ---> Microsoft.Mapi.MapiExceptionPartialCompletion: MapiExceptionPartialCompletion: Unable to copy message(s). (hr=0x40680, ec=0)Diagnostic context:    Lid: 21750     Lid: 18122   StoreEc: 0x4DA        Lid: 25510     Lid: 25270   StoreEc: 0x4DA        Lid: 19830     Lid: 25290   StoreEc: 0x4DA        Lid: 55847   EMSMDBPOOL.EcPoolSessionDoRpc called [length=45]    Lid: 43559   EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0x0][length=118][latency=0]    Lid: 23226   --- ROP Parse Start ---    Lid: 31418   --- ROP Parse Done ---    Lid: 16465     Lid: 24657   StoreEc: 0x4DA       at Microsoft.Mapi.MapiExceptionHelper.ThrowIfErrorOrWarning(String message, Int32 hresult, Boolean allowWarnings, SafeExInterfaceHandle iUnknown, Exception innerException)   at Microsoft.Mapi.MapiUnk.ThrowIfErrorOrWarning(String message, Int32 hr)   at Microsoft.Mapi.MapiFolder.CopyMessages(CopyMessagesFlags flags, MapiFolder destFolder, Byte[][] entryIds)   at Microsoft.Exchange.Data.Storage.CoreFolder.<>c__DisplayClass24.<InternalMoveOrCopyItems>b__1e()   at Microsoft.Exchange.Data.Storage.CoreFolder.ExecuteMapiGroupOperationMethod(String operationAttempted, StoreObjectId[] sourceObjectIds, MapiGroupOperation mapiGroupOperationCall)   --- End of inner exception stack trace ---   at Microsoft.Exchange.Data.Storage.CoreFolder.ExecuteMapiGroupOperationMethod(String operationAttempted, StoreObjectId[] sourceObjectIds, MapiGroupOperation mapiGroupOperationCall)   at Microsoft.Exchange.Data.Storage.CoreFolder.ExecuteMapiGroupOperationInternal(StoreObjectId[] sourceObjectIds, MapiGroupOperationInternalMethod method)'.
  Source:        MSExchange MailboxSearch
  Date:          10/2/2013 2:05:26 PM
  Event ID:      21003
  Task Category: (21)
  Level:         Warning
  Keywords:      Classic
  User:          N/A
  Computer:      Server01.domain.com
  Description:
  Batch copy for MailboxSearch 'Test 2' on mailbox 'Joe Bloggs' failed on folder 'DPATEST' failed even after retries, and it will be skipped. The exception encountered was: 'Microsoft.Exchange.Data.Storage.PartialCompletionException: Move/Copy messages failed. ---> Microsoft.Mapi.MapiExceptionPartialCompletion: MapiExceptionPartialCompletion: Unable to copy message(s). (hr=0x40680, ec=0)
  Diagnostic context:
      Lid: 21750  
      Lid: 18122   StoreEc: 0x4DA     
      Lid: 25510  
      Lid: 25270   StoreEc: 0x4DA     
      Lid: 19830  
      Lid: 25290   StoreEc: 0x4DA     
      Lid: 55847   EMSMDBPOOL.EcPoolSessionDoRpc called [length=45]
      Lid: 43559   EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0x0][length=118][latency=0]
      Lid: 23226   --- ROP Parse Start ---
      Lid: 31418   --- ROP Parse Done ---
      Lid: 16465  
      Lid: 24657   StoreEc: 0x4DA     
     at Microsoft.Mapi.MapiExceptionHelper.ThrowIfErrorOrWarning(String message, Int32 hresult, Boolean allowWarnings, SafeExInterfaceHandle iUnknown, Exception innerException)
     at Microsoft.Mapi.MapiUnk.ThrowIfErrorOrWarning(String message, Int32 hr)
     at Microsoft.Mapi.MapiFolder.CopyMessages(CopyMessagesFlags flags, MapiFolder destFolder, Byte[][] entryIds)
     at Microsoft.Exchange.Data.Storage.CoreFolder.<>c__DisplayClass24.<InternalMoveOrCopyItems>b__1e()
     at Microsoft.Exchange.Data.Storage.CoreFolder.ExecuteMapiGroupOperationMethod(String operationAttempted, StoreObjectId[] sourceObjectIds, MapiGroupOperation mapiGroupOperationCall)
     --- End of inner exception stack trace ---
     at Microsoft.Exchange.Data.Storage.CoreFolder.ExecuteMapiGroupOperationMethod(String operationAttempted, StoreObjectId[] sourceObjectIds, MapiGroupOperation mapiGroupOperationCall)
     at Microsoft.Exchange.Data.Storage.CoreFolder.ExecuteMapiGroupOperationInternal(StoreObjectId[] sourceObjectIds, MapiGroupOperationInternalMethod method)'.
  Source:        MSExchange Mid-Tier Storage
  Date:          10/2/2013 2:05:33 PM
  Event ID:      3003
  Task Category: Discovery
  Level:         Error
  Keywords:      Classic
  User:          N/A
  Computer:      Server01.domain.com
  Description:
  Search 'SearchStatus\2e0e5de4-30a3-46db-8bff-0dd831a341c7' has following error:
  An error occurred when searching Joe Bloggs. The message is 'Move/Copy messages failed.'.
  
  Log Name:      Application
  Log Name:      Application

Initially we couldn't work out why this was occurring, my colleague (John Manley), looked into this further and worked out that the Discovery Search mailbox did not have items over 100mb.  This was a big find and once we had a common reason for the search failing it made the next steps easier.

Our company has a much smaller global transport limit configured in Exchange, but this is typically covering messages which were sent and received before this limit was imposed.

We tested the search with the Transport Limit off, but the same problem occurred.  We moved the Discovery Search mailbox to its own DB but the problem still occurred.

Eventually we found that although the MaxSendSize and MaxReceiveSize for every user in the Exchange Organisation was unlimited, the Discovery Search mailbox was set to 100mb.

We set this mailbox to be unlimited.  This change takes a number of hours to propagate through Exchange, but once replicated the searches begain to return results greater than 100mb......yay!

Hopefully this will save someone a few painstaking hours of trawling log files etc.

Sam

Google +