Logical Record Caching

Sue Pavlakis, IBM TPF Development, George Leier, IBM TPF Test Core Team, and Fay Casatuta, IBM TPF ID Core Team

Overview

Logical record cache support (APAR PJ27083) allows applications to define and use memory caches, and provides fast access to frequently used information through logical record caches.

Logical record caching is a set of system functions that allow an application to define and use memory caches. A cache is a hashed structure for holding information in a temporary storage area (system heap) where a copy of the data is saved to avoid refetching that data on every access. The data is retrieved from temporary storage instead of its permanent residence. The cache gives the application fast access to frequently used information without always needing to retrieve the information from an external storage device.

Logical record caches are named caches that are subsystem shared with entries that are subsystem unique and either processor unique or processor shared. Processor unique caches hold data that is only used and updated by one processor. The data pertains only to the processor where it resides and does not need to be kept synchronous with the image of any other processor. A processor unique cache does not require a coupling facility (CF). Processor shared logical record caches are discussed later in this article.

The information held in the logical record cache must also be resident on an external storage device. A cache is lost over a processor IPL and must be reestablished by the application after an IPL occurs. There is no mechanism for preserving the contents of a cache over an IPL.

Two areas of the TPF 4.1 system use logical record caching:

• File system support: To improve file system performance, processor unique logical record caches were created for the directory information and the i-node information. This support was provided with APAR PJ26713 on program update tape (PUT) 11. With PUT 13, these caches were converted to processor shared logical record caches to keep the updates synchronized between all processors in the complex. This support was provided with APAR PJ27083 and is discussed later in this article.

• Domain name system (DNS) support: DNS support uses two DNS client caches, saved gethostbyaddr() responses and saved gethostbyname() responses. With logical record caching, there is no longer a need to retrieve data from an external DNS server on every reference. Both types of caches are processor unique caches with entries that are timed out. This support was provided with APAR PJ27268 on PUT 13.

Today, with logical record cache support provided on PUT 13, you can use the application programming interfaces (APIs) that are provided to do the following:

• Create a logical record cache
• Add a logical record cache entry
• Update a logical record cache entry
• Read a logical record cache entry
• Delete a logical record cache entry
• Flush a logical record cache
• Delete a logical record cache.

An explanation of each API and information about how to use it in your application program is provided later in this article. Coding examples are also provided.

Processor Shared Logical Record Caches

Processor shared logical record caches hold data that can be used and updated by all processors in a complex so the data must be kept synchronous between all the processors. CF support is used to keep this data synchronous for all processors in the complex. Therefore, to make use of processor shared logical record caches, at least one CF must be available to the complex. A discussion of how the operating system uses CF support to synchronize the data follows.

When an application creates a processor shared logical record cache, operating system logical record cache support creates a local cache for the processor using system heap and also creates a cache structure on a CF. When the application creates the same processor shared logical record cache on another processor in the complex, operating system support again creates a local cache for this other processor and connects this other processor to the existing cache structure on the CF; that is, there is a local cache for each processor, but only one cache structure on the CF for a given processor shared logical record cache.

When an application adds an entry to the processor shared logical record cache, operating system logical record cache support adds the entry to the processor's local cache and registers interest in the cache entry on the CF. An indicator in the processor's local cache vector is assigned to the added logical record cache entry.

When an application updates an entry in the processor shared logical record cache, operating system logical record cache support updates the entry in the processor's local cache and uses the CF's cross-invalidate mechanism to inform other processors in the complex of the change to the data. Cross-invalidate processing involves setting the indicator in each processor's local cache vector for the logical record cache entry to indicate that the locally cached copy of the data is no longer valid.

When an application reads an entry in the processor shared logical record cache, operating system logical record cache support checks the indicator in the local cache vector for the logical record cache entry to determine whether the locally cached copy of the data is still valid. If the locally cached copy is still valid, the data is returned to the application from the local cache. If the locally cached copy is no longer valid, a CACHE_NOT_FOUND indication is returned to the application and the application must retrieve the data from permanent storage and add the entry to the logical record cache again.

It is important to note that the cache structure on the CF is a directory-only cache. No data is stored in a directory-only cache. The directory-only cache is used to maintain consistency of data in the local caches of processors.

File System Usage of Processor Shared Caches

As discussed previously in this article, file system performance was improved by using processor unique logical record caches for the directory information and the i-node information. However, the data in these caches can be updated by any processor in the complex. To keep the updates synchronized between all the processors in the complex, these caches were converted to processor shared logical record caches. This support was provided with APAR PJ27083 on PUT 13.

Logical Record Cache Attributes

A logical record cache is given a name when it is created. An entry in a logical record cache is uniquely identified by the database ID (DBI) in the entry control block (ECB), a primary key, and optionally, a secondary key. When a logical record cache is created, it is assigned a cache token. This cache token is used on subsequent function calls to identify the logical record cache.

Creating a Logical Record Cache

To create a logical record cache, an application uses the newCache function, passing the following as parameters:

• The name of the logical record cache being created. If the logical record cache is processor shared, all processors in the complex must use the same name.

• A pointer to a field where the cache token for the created logical record cache will be stored.

• The maximum lengths of the primary and secondary keys.

• The maximum length of the data area for an entry in the logical record cache.

• The minimum number of unique cache entries the logical record cache will hold before it begins to reuse the least-referenced cache entries.

• The number of seconds that a cache entry can reside in the local cache before it is cast out.

• The type of cache (processor shared or processor unique).

Adding an Entry to a Logical Record Cache

To add an entry to a logical record cache, an application uses the updateCacheEntry function, passing the following as parameters:

• The cache token for the logical record cache
• The primary key and its length
• The secondary key and its length
• The pointer to the area that contains the entry data to be saved in the logical record cache
• The length of the passed entry data
• The number of seconds the entry can reside in the cache before it is flushed
• An indicator to change the local cache only; that is, don't invalidate other processors.

Applications should use a serialization mechanism when adding an entry to a logical record cache.

Updating an Entry in a Logical Record Cache

To update an entry in a logical record cache, an application uses the updateCacheEntry function, passing the following as parameters:

• The cache token for the logical record cache
• The primary key and its length
• The secondary key and its length
• The pointer to the area that contains the entry data to be saved in the logical record cache
• The length of the passed entry data
• The number of seconds the entry can reside in the cache before it is flushed
• An indicator to invalidate other processors.

Applications should use a serialization mechanism when updating an entry in a logical record cache.

Reading an Entry from a Logical Record Cache

To read an entry from a logical record cache, an application uses the readCacheEntry function, passing the following as parameters:

• The cache token for the logical record cache
• The primary key and its length
• The secondary key and its length
• The pointer to the area that is to be overlayed with the contents of the specified cache entry
• The length of the area to hold the contents of the specified cache entry.

Deleting an Entry from a Logical Record Cache

To delete an entry from a logical record cache, an application uses the deleteCacheEntry function, passing the following as parameters:

• The cache token for the logical record cache
• The primary key and its length
• The secondary key and its length.

Deleting an entry from a processor shared logical record cache causes the entry to be invalidated on all other processors in the complex.

Deleting a Logical Record Cache

To delete a logical record cache, an application uses the deleteCache function, passing the following parameter:

• The cache token for the logical record cache.

Flushing a Logical Record Cache

To flush a logical record cache, an application uses the flushCache function, passing the following parameter:

• The cache token for the logical record cache.

Coding Examples

The following code excerpt shows how to use the newCache function to create a processor shared logical record cache that uses a secondary key for its entries in addition to a primary key.

char      cache_name[CACHE_MAX_NAME_SIZE];
long      primeKeyLgh=NAME_MAX;      /* max lgh of path name */
long      secondaryKeyLgh=sizeof(ino_t);    /* INODE nbr lgh */
long      cacheSize = * (long *) &ecbptr()->ebw004;
long      dataLgh=sizeof(struct TPF_directory_entry);
long      castOutTime = TPF_FS_CACHE_TIMEOUT ;
char      typeOfCache=Cache_ProcS;     /* proc shared cache  */

strcpy(cache_name,TPF_FS_DIR_CACHE_NAME);
if (newCache(&cache_name,
          &contrl_ptr->icontrol_dcacheToken,
          primeKeyLgh,
          secondaryKeyLgh,
          dataLgh,
          cacheSize,
          castOutTime,
          &typeOfCache,
          NULL) != CACHE_SUCCESS)
   {
   ....error processing
   }

The following code excerpt shows how to use the readCacheEntry function to attempt to read an entry from a cache. If the entry is not found in the cache, the data is then read from the database and the entry is added to the cache using the updateCacheEntry function. The NULL used as the next to last parameter on the updateCacheEntry function call indicates that the entry will not be flushed from the cache. The NULL used as the last parameter on the updateCacheEntry function call indicates that the entry is not invalidated on the other processors in the complex.

long   primaryKeyLgh = strlen( tran->itran_response.itres_name );
long   secondaryKeyLgh = sizeof(ino_t);
struct TPF_directory_entry tde;
long   bufferSize= sizeof(struct TPF_directory_entry);
if  (readCacheEntry(&contrl_ptr->icontrol_dcacheToken,
            &tran->itran_response.itres_name,
            &primaryKeyLgh,
            &tran->itran_response.
                 itres_parent_inode.inode_ino,
            &secondaryKeyLgh,
            &bufferSize,
            &tde) != CACHE_SUCCESS)
{
....read directory from the database, it was not found in cache
    
/* Add the directory to the directory cache.                   */

updateCacheEntry(&contrl_ptr->icontrol_dcacheToken,
            &tran->itran_response.itres_name,
            &primaryKeyLgh,
            &tran->itran_response.
             itres_parent_inode.inode_ino,
            &secondaryKeyLgh,
            &bufferSize,
            &tde,                        
            NULL,                        
            NULL );                      
}

The following code excerpt shows how to use the updateCacheEntry function to update an entry in the cache after the data has been updated on the database. The NULL used as the next to last parameter on the updateCacheEntry function call indicates that the entry will not be flushed from the cache. The last parameter on the updateCacheEntry function call indicates that the entry is to be invalidated on the other processors in the complex since this processor updated the entry.

struct TPF_directory_entry new_tde =                             
       { TPF_FS_DIRECTORY_CURRENT_VERSION };                
const long dataLength = sizeof (new_tde);
const long primaryKeyLgh = strlen( link->ilink_file_name );
const long secondaryKeyLgh = sizeof(ino_t);
const char invalidate = Cache_Invalidate;

new_tde.TPF_directory_entry_ino =                        
   link->ilink_file_inode_ordinal;                      
new_tde.TPF_directory_entry_igen =                       
   link->ilink_file_inode->inode_igen;

...update the database

   updateCacheEntry(&contrl_ptr->icontrol_dcacheToken,
             link->ilink_file_name,
             &primaryKeyLgh,
             &link->ilink_parent_inode->inode_ino,
             &secondaryKeyLgh,
             &dataLength,
             &new_tde,                           
             NULL,                              
             &invalidate );

Using Processor Shared Logical Record Caches

Using processor shared logical record caches requires CF resources for TPF record locking to be established. You are not required to use CF record locking (through the ZMCFT MIGRATE functional message), but must set up one or more CFs for record locking before you can enable CFs for the synchronization of processor shared logical record caches. The following functional messages are used to establish the CF resources necessary for synchronization of processor shared logical record caches:

• ZMCFT ADD cfname — This functional message must be entered on all processors to add the CF to each processor.

• ZCFLK ADD cfname SIZE xxxx — This functional message is entered from one processor to add the CF to the locking configuration for all processors.

You enable CFs to be used for synchronization of processor shared logical record caches by entering the ZCACH CF ENABLE functional message. This functional message is entered from one processor and takes effect on all other active processors in the complex. If an inactive processor is brought into the complex later, you must enter this functional message again at that time.

Once this is done, applications can create and use processor shared logical record caches that will be synchronized across multiple loosely coupled processors. Additionally, any existing processor shared logical record caches will begin to be synchronized. Processor shared logical record caches behave like processor unique caches if CFs are not enabled for processor shared logical record cache synchronization. Therefore, processor shared logical record caches could be implemented with the understanding that the caches would not be synchronized. CF resources could then be introduced at a later date to enable the caches to be synchronized.

So How Are My Caches Doing ?

Logical record caches are established and used by applications as stated previously. Whether you are talking about processor shared or processor unique caches, they can be monitored for tuning purposes by using TPF Data Collection reports and by entering functional messages. The ZCACH DISPLAY functional message can be used to display attributes and count fields for the processor's local copies of the logical record caches, and the data collection TPF Logical Record Cache Summary report can be used to check on how your logical record caches are doing.

Similarly, the ZCFCH DISPLAY functional message can be used to display attributes and counts of CF-resident directory-only caches. The data collection Coupling Facility Caching Summary report should also be used for tuning CF-resident directory-only caches. Information such as castouts per second can be very helpful in determining if caching attributes need to be altered. As is always the case, tuning is very installation-specific. If you have additional questions about cache tuning, call your IBM TPF support representative.

If you want to modify cache attributes, you can enter the ZCACH MODIFY functional message. This functional message will establish a set of permanent override values that will be used by the TPF system whenever caches of the names specified are created. For the new attributes to take effect for caches that already exist, these caches must be deleted either by the application or by entering the ZCACH DELETE functional message. When these logical record caches are re-created, the new attributes will be used.

Similarly, if you want to modify CF-resident directory-only cache structure attributes, you can enter the ZCFCH MODIFY functional message. This functional message will establish a set of permanent override values that will be used by the TPF system whenever CF-resident directory-only caches of the names specified are created. For new attributes to be used on existing CF-resident directory-only cache structures, the associated processor shared logical record caches must be deleted on all processors by the application or by the ZCACH DELETE functional message. Deleting all the processor shared logical record caches that use the CF-resident directory-only cache structure will force the directory-only cache structure to be created with the new attributes the next time it is created.

Creating override values through the functional messages described previously can be used to try out or phase in new logical record cache or CF directory-only cache attributes. When the new attributes are verified, the associated applications can be updated to use these new attributes and the overrides can be deleted through a functional message.