|
Introducing the MQSeries Slow Queue Sweeper
Alice Williams-Obleton, IBM TPF Development
Alright, you got us! The slow queue sweeper (SQS) was introduced in
our Third Quarter 2000 TPF newsletter, but the TPF MQSeries team is so
excited about this hot new technology that we wanted to take this opportunity
to share with you intimate details of the SQS. For those who are new or
unfamiliar to the SQS, it is an agent that is used to move
messages residing in memory for all TPF MQSeries local queue manager memory
queues to TPF collection support (TPFCS) files, thus freeing up system
work blocks (SWBs) to return them to the system pool. This article follows
a topdown approach to unfolding the mysteries of the SQS.
The Big Picture
The SQS is an internal tool provided for TPF MQSeries support. It is
designed to identify queues that are being serviced regularly, yet growing
in size. Queues are serviced when messages are removed from local memory
queues by applications, or messages from transmission queues are sent over
channels. The SQS is kicked off every 5 seconds through the MQSeries monitor
routine, which also triggers queue deletions. The figure that follows is
a high-level view of the SQS at work.
To Sweep or Not to Sweep
When messages start to build up on the queue, this is an indication
that messages are coming into the system faster than they are being processed.
The SQS determines whether the queue needs to be swept by using the following
equation:
L1(x) > ½ L1(y)
where y is the number of messages currently residing in memory
on the local memory queue (L1), and x is the service rate of L1,
or the number of messages obtained from the same queue either through channel
batch processing or an application that issues an MQGET request within
a 5-second interval. Transmission queues associated with an active or in-doubt
channel will not have messages associated with a batch swept regardless
of the queue's service rate. Both persistent and nonpersistent messages
are counted for the queue. Once the queue length is greater than twice
the service rate, the queue is targeted for a sweep. When the queue is
swept, messages are then moved from memory and stored in a binary large
object (BLOB), which is part of the TPFCS collection. A buffer of messages
remains in memory that is equivalent to the total number of messages retrieved
from the queue within a 5-second interval plus 5. The buffer of messages
avoids the immediate unsweeping of the queue when the next MQGET request
or channel batch assignment occurs.
Sweeping Messages
The structure of a swept queue is shown in the following figure.
Front and Rear Message Lists
The queue definition, which resides in system heap, contains pointers
to the front and rear message lists and also contains the persistent identifier
(PID) of the first TPFCS BLOB. The purpose of the lists is to maintain
the order of messages on the queue.
The front message list of a local memory queue is a doubly linked
list structure containing messages residing in system work blocks that
are put on the queue by an application. All messages are retrieved from
a local memory queue by an application or channel using this list. The
front message list will be the only list in use for MQPUT and MQGET requests
if the queue is not in a swept state.
The rear message list is also a linked list structure
containing messages added to the local memory queue after the queue is
in a swept state. If a queue is swept or in the process of a sweep, all
incoming messages added to the queue will be put on its rear message list.
MQGET requests and channel batch processing will still be done using the
front message list of the queue.
Sweep Process
The SQS starts its execution by traversing the queue to get tallies
of all messages residing on both the front and rear message lists. Both
counts include only committed messages that have not been retrieved by
an application or a channel. The SQS determines whether the queue needs
to be swept by using the previously shown sweep equation for each queue
separately. During one sweep cycle, it is possible to create two new BLOBs
for a targeted queue: one for the front message list and one for the rear
message list. A sweep of the front message list results in a BLOB that
is chained to the beginning of the chain of swept PIDs, and a sweep of
the rear message list results in a BLOB chained to the end of the chain
of swept PIDs.
To begin a sweep, the SQS must first obtain a lock on the local memory
queue. Once the lock is in place, the next step is to construct a list
of messages targeted for the sweep. The SQS then proceeds to move the targeted
messages into the BLOB. While this step is being performed, the SQS releases
its lock on the queue. This part becomes a little tricky because once the
lock is released, the queue becomes available for use by other MQSeries
processes. In other words, new messages can be added or existing messages
can now be retrieved by applications or channel batch processing while
the SQS is busy writing to the BLOB. To ensure the integrity of the system,
a variety of checks are performed once the SQS restores its lock on the
queue. During this verification process, each targeted message is checked
for the following conditions:
-
A message has been assigned to a batch, or the batch that the message has
been assigned to has changed while the messages are filed to the BLOB.
In this situation, the copy of the message in the filed BLOB will not contain
the correct batch number. At this point, completion of the sweep will erase
the correct batch number from the message. A subsequent unsweep of the
message will cause a channel error because the channel will no longer find
all of the messages for its batch.
-
A message has been marked as gotten (obtained), but not committed on the
queue. In this case, a sweep would cause the commit process to fail because
there is no mechanism available to unsweep a queue at this point, and commit
processing requires the message to be in memory and not in a BLOB.
-
A message has been removed from the queue. In this case, it does not make
logical sense to sweep a message that no longer exists.
If any of the previous conditions occur, the sweep is no longer considered
valid and will be aborted. If the verification process is successful, all
messages that were written to file are deleted from system memory, thus
freeing up system work blocks to be used by other processes.
Getting Messages from a Queue
Once the front message list is depleted and an application issues an
MQGET request or channel batch processing occurs, the first-in-chain BLOB
is retrieved from TPFCS and its messages are used to replenish the front
message list. Once the last-in-chain BLOB is returned to memory, all messages
located on the rear message list will be chained to the end of the front
message list. When the last-in-chain BLOB is unswept, all messages again
reside in the front message list.
Happy Sweeping!
Now that we have unveiled the secrets to the SQS, we encourage you to
take full advantage of all that it has to offer. For more information about
the SQS and TPF MQSeries support, check out the TPF 4.1 library information.
For questions, contact your TPF customer service representative.
Second Quarter 2001 - Table of Contents
|
