XADM: Jet Doesn't Detect Removed Page in B-tree Split Operation

- Microsoft Exchange Server, version 5.5

SYMPTOMS

The Microsoft Exchange information store database may experience one of the following symptoms during normal operation:

During an offline defrag (ESEUTIL /D) of the database, the process stops with JET error -1601(JET_errRecordNotFound) or -1603 (JET_errNoCurrentRecord).

During normal operation, you may get a -1069 error (JET_errVersionStoreOutOfMemory) in the application log.

If you run the consistency checker (ESEUTIL /G <db> /V /X piped to a text file), it reports ERROR: corrupted LV(number) (lid changed without root).

The information store service (Store.exe) may access violate during normal operation when it encounters corrupt data.

The information store process is utilizing 100 percent of the CPU time all consumed by one thread (there are occasions when up to three threads will be affected). You can view this by following these steps:

   1. Start the Windows NT Performance Monitor.
   2. From the Edit menu, click Add To Chart...
   3. In the Object drop-down list box, select Thread.
   4. In the Instance list box, select all values that contain "store".
   5. Click the Add button.

CAUSE

When the Jet database engine goes to insert a record into the binary tree (b-tree), it may find a record with the same primary key that is marked for deletion. Because the node is just marked for deletion and has not been physically deleted yet, the Jet database engine "undeletes" the node and changes its data to that of the new record. Because the Jet database engine is changing the data associated with the record, the Jet database engine may be changing the record's size. If the record becomes too big for the page, the Jet database engine has to perform a b-tree split. During the split, the Jet database engine must release the latch on the pages in the tree and then re-latch them when the split is completed. In rare circumstances where the information store cleanup thread comes along in the small delta where the pages were unlatched, the information store cleanup thread may remove the page marked for deletion. When the Jet database engine tries to re-latch the pages, it does not know the record has been removed. As a result, the information store either stops responding (uses 100 percent processor time in one or more threads) or corrupts the data.

RESOLUTION

Apply the fix described below. The Jet database engine has been fixed to detect whether the page has been removed and to take appropriate action if it has.

If you have already experienced the corruption symptoms of this problem, you can do the following to repair the corruption with NO loss of data and no need to run ISINTEG -fix:

1. Make sure the new Ese.dll file described below is in place (\winnt\system32).

If you have not experienced any of these symptoms but want to be proactive and prevent this corruption from occurring, make sure the Ese.dll file is in place (\winnt\system32) and restart the information store service.

STATUS

Microsoft has confirmed this to be a problem in Microsoft Exchange Server version 5.5. A supported fix is now available, but has not been fully regression-tested and should be applied only to systems experiencing this specific problem. Unless you are severely impacted by this specific problem, Microsoft recommends that you wait for the next Service Pack that contains this fix. Contact Microsoft Technical Support for more information.

MORE INFORMATION

High CPU utilization or a stoppage of the information store will occur if there are exactly two nodes on the page being split. This is because the Jet database engine will try to split one record on a page, which is not possible. Data corruption will occur if there are more than two nodes on the page being split. This is because Jet database engine will replace the node immediately after the one it intended to insert to.

Keywords          : XADM kbusage
Version           : 5.5
Platform          : WINDOWS
Issue type        : kbbug