The Architecture of Unread Marks in Lotus Notes

The Architecture of Unread Marks in Lotus Notes

 

Abstract
This document describes the architecture of unread marks in the Lotus Notes client. 

 

Content
In every database, Lotus Notes keeps track of each document that every user has read. This information is stored and maintained by several structures within both the .NSF and the Notes client itself. The following is a description of this architecture.
Note: Additional unread functionality was introduced in Notes/Domino 6.0.3 and 6.5.

 

The Unread ID Table

Each database stores the list of unread documents in an Unread ID Table within the NSF file. To save space, this table stores a list of Note IDs instead of the longer UNIDs. Because of this, the table is specific to the replica that stores it.

This table is user-specific and is stored within the .NSF file in a special structure named with the hierarchical user name to which it applies. When a database is opened, this table is transferred locally to the Notes client and stored temporarily in the DESKTOP.DSK file (in R5 this is renamed DESKTOP5.DSK). All updates to the table are then performed locally during the session for which the database is opened. It is important to recognize that there is no date information regarding when a

particular document is marked read, aside from the time stamp of the ID table itself. Therefore, onflicts between replicas are not resolved in a similar way to replication conflicts. It should also be noted that because these tables are lists of Note IDs, any synchronization between tables must involve the conversion of Note IDs to UNIDs.

While the Notes user is accessing a database, all read operations are logged in a structure called the Unread Journal. This Journal is stored in the client's CACHE.DSK file and stores about 20,000 consecutive operations in a circular data structure (FIFO) that tracks all of the UNIDs of documents marked read or unread while accessing a database. This Journal is specific to the Notes client and is not involved with tracking any operations performed by an API program. More detail on Journal

Playback is outlined below.

In Release 5.0, this Unread Mark Architecture is not drastically altered. Most of the changes to this process were to internal code while leaving the main procedures intact. The most significant new feature is a database property that prevents unread marks from being maintained completely. Because of the network traffic that is necessary to track and update the Unread ID tables, this feature should provide increased performance for databases that implement it.

Also in R5, the server started maintaining an in-memory copy of a user's unread ID table. This allows multiple clients or API programs to access a user's unread table without resulting in a 'last one out wins' scenario. Instead, as each client session sends back its updated version of the unread table, the server merges the changes with the version in-memory.

 

How are Unread Marks Updated?

The Unread ID table is updated in a few different ways, most often as users open or edit documents. The Unread ID table is transferred locally to the DESKTOP.DSK or DESKTOP5.DSK file and updated there. When the user closes the database, the ID table is transferred back to the .NSF file. This is true regardless of whether the database is residing on a server. Users see the unread marks in views that utilize the property to display them. The Unread ID table is overlaid onto the view index, and the client is then responsible for displaying the entries properly. When a user chooses to update the ead/unread status of a document using the menu, the same behavior is followed.

Every database stores a separate table called the Modified Note ID Table. Exposed through the Notes API, this is a list of every note in the database, in order of the last modified time. When the user opens a database, the client grabs a list of modified documents since the last time the unread table was updated using the Modified Note ID Table. The client then applies the newly modified documents to the Unread ID table. Note that newly added documents, such as mail, are not inserted into an Unread ID Table until the time a user opens the database. In cases where the modified note ID table includes incorrect information, the unread ID table may end up reflecting the error.Journal 'Playback'The Unread Journal in the CACHE.DSK file of a client maintains a list of consecutive unread mark operations performed on every database accessed by the client. The desktop file maintains a set of positions within this structure related to the last operations performed on a particular database and its replicas. When a database is then opened, the client checks to determine if the last operation played back against that particular replica is the last operation performed against any replica of that database. If it is, the client does not make any changes to the unread table based on the journal. If it is not, the client then plays scans through every operation stored in the journal and applies the appropriate ones to the unread table loaded in memory. This can sometimes result in inconsistent unread tables across replicas of a database in the event of an abnormal database closure (e.g. when the server or client crashes). If the unread table cannot be saved to the file appropriately, it may result in the desktop information being out of sync with the unread table.

Because the journal is limited to 20,000 operations (or entries), the Notes client uses special methods to deal with large unread mark operations, such as 'Mark All Read'. When a user marks a number of documents read or unread at once, the journal stores each operation in an individual entry. In cases where the number of individual operations associated with a single event exceeds 1000, the journal stores an entry called 'Mark All Before This Time'. This entry stores the time of the operation, and during journal playback, this entry results in any document last modified before the specific time being

marked read or unread accordingly. This can cause inconsistencies when documents fit into the timeframe of the event, but have not replicated into the database. Users may expect to see those documents unaffected, but, in fact, they will be.

How are Unread Marks Maintained?

Here we will investigate four different situations involving Notes clients and Notes database replicas.

Single client - Single replicaThe Unread ID table is transferred to the local DESKTOP.DSK or DESKTOP5.DSK as a user opens the database. The Unread Journal in the CACHE.DSK is then "played back" against this ID table and the table is updated to reflect any changes that the Journal has logged. The modified Unread ID table is then transferred back to the .NSF file when the database is closed. In these situations, the net result is that the Unread Journal does little more than track the user's actions. The Journal does not really come into play until multiple replicas are introduced.

 

Multiple client - Single replica
This is similar to the above situation, except two DESKTOP.DSK or DESKTOP5.DSK files are being used. As long as each client is closing the database before it is opened by the other, the Unread Table should be maintained correctly. However, if the two clients are accessing the same replica simultaneously, problems may arise. If each client is maintaining a local copy of the Unread ID table, then the "last one out" principle is observed. Whichever client is the last to close the database will

overwrite the entire Unread ID table maintained by the other client.

In R4, this is a situation that can also be caused by other API programs running on the same machine, such as Organizer 97 GS. Each program maintains its own Unread ID table, and inconsistencies appear because the two programs rarely touch the same documents. This can be mitigated by using the NOTES.INI parameter, NSF_UNREAD_METHOD=1, which allows the API programs (Notes included) to modify only the document entries that it touches. Changes to the R5 architecture take care of this limitation and the parameter is no longer required.

 

Single client - Multiple replica
Again, the Unread ID table is transferred to the local DESKTOP.DSK or DESKTOP5.DSK file. In this case however, the Unread Journal comes into play. Once the Unread ID table is local, the Unread Journal is "played back" against the table, updating it with the operations performed in the other replica. The limits outlined above will affect the performance of this scenario. Alternatively, the user may select two replica icons from the workspace and manually exchange unread marks. This can be performed either by using the menu command Edit, Unread Marks, Exchange Unread Marks, or by using a SmartIcon that performs @Command([ExchangeUnreadMarks]). These actions would have to be performed while icons are unstacked on the workspace. Note that the while the @Command still exists in R5, the menu command does not.

 

Multiple clients - Multiple replicas
This situation most likely will result in inconsistent unread mark lists. It is also fairly common. Stepping through the processes that are outlined above, it is clear that the structures and methods in place for maintaining unread marks are not sufficient for this situation. If two clients are exclusively accessing two separate replicas, neither the Unread Mark ID table nor the Unread Journal will be able to synchronize unread marks between the two replicas. Replication on its own does not synchronize unread marks either. Because the Unread Mark ID tables are user-specific and use Note IDs, it is not feasible to transfer all of these tables during replication.

To accommodate this type of situation, the Notes client implements an .INI parameter, REPLICATOR_SYNC_UNREAD. This parameter should equal the number of half-hours between Unread ID table synchronization. It is a client implementation only, and will not synchronize unread marks between servers unless a client is initiating the replication event. Even in this case, the only

Unread ID tables that will be updated will be the ones specific to the user initiating replication. The intention of this parameter is to help synchronize the Unread ID tables between two replicas that are primarily used by different clients. An example of this would be an office replica that resides on a server and a remote replica that resides on a home machine or a laptop.

This parameter works by synchronizing the Unread ID tables of the two replicas after each replication event that comes after the minimum number of half-hours specified in the .INI file. For example, if the .INI file has REPLICATOR_SYNC_UNREAD=2, the replicator will synchronize the Unread ID tables once an hour. If replication occurs as a background process every fifteen minutes, synchronization will occur only every fourth replication event. If this parameter is set to -1, the replicator will synchronize the Unread ID tables after each replication event. For more information on this parameter, refer to the document titled, "Unread Marks Are Not Exchanged Replicating to a Server From Two Different Clients" (#158903 ).

It should be noted that this will have a negative affect on replication times. In addition, this feature may be made obsolete or implemented differently in future versions of Notes. Another caveat that should be included here is that there are no replication conflicts in this process. These ID tables may have conflicting information in them regarding a single document.

Because documents generally go from 'unread' to 'read,' these conflicts are always resolved to 'read.' This might not be what the user expected. The Unread Journal does not have this same problem because while it does not track the exact times of the

operations it tracks, it does know in what order they occurred