CCR2

A publication for the IBM Tivoli and zSeries community

How would you begin the process?
If you already have substantial automation in operation and your IPL is managed largely by AF/OPERATOR or some other automation product, you may only need to extend the service from system load to where JES is initialized, and from when JES shuts down to Z EOD. Where do you start? You need a copy of the system messages and commands issued before SYSLOG begins. Here are three AF/OPERATOR traps you can use to capture those messages and write them to a user log.

"TRAP ADD($$MSGS) WTO('*') ENA AOTRAP NOLOG ACT('EX USRLOG')"
"TRAP ADD($$CMDS) CMD('*') ENA AOTRAP NOLOG ACT('EX USRLOG')"
"TRAP ADD($$STOPLOG) WTO('IEA630I *') ENA LOG",
     "AOTRAP ACT('EX STOPLOG')"

The USRLOG program simply issues the REXX 'logmsg(aotext,'USER' or 'RKOGLOGM') and STOPLOG issues an AO FREE for the appropriate dataset and deletes the $$MSGS & $$CMDS traps. Save the log dataset into a permanent dataset before the next IPL for analysis and development.

The last trap, $$STOPLOG, assumes you are running JES2 and it is the message issued when the log becomes available. If you are running JES3, simply replace the trap with the JES3 active message.

"TRAP DEL($$MSGS) WTO"
"TRAP DEL($$CMDS) CMD"
"AO FREE(USERLG)"

Why do you need the early log messages?
The early syslog messages should be scanned carefully to determine what needs to be added to your IPL code to truly automate the rest of the process. You may find start commands issued from a SYS1.PARMLIB member not yet in the code. In particular, these would be the COMMNDxx or IEFSSNxx members or IEACMD(SYS)xx. You also may find JCL or imbedded code that ties several starts to CAS9, for instance. This must come out and be integrated into your start and stop processes. You may also find WTOR messages that require responses:

*4102 IOS120D I/O TIMED OUT FOR DEVICE 3036. REPLY 'WAIT' FOR I/O
COMPLETION OR 'CONT' TO CONTINUE WITH DEVICE OFFLINE

This WTOR requires a CONT response before the IPL can commence. Traps for these conditions must be added very early in the automation process.

Configuring AF/OPERATOR
The steps in configuring AF/OPERATOR are to assure the DASD message logs and user logs are defined and cataloged as disk datasets. You will see in the CICAT configuration that if you select DISKMLOG, file definitions are added to the JCL. If this was not done during configuration, you will need to define them yourself. The DD names are RKOGLM01/n for the message log (n is the number of datasets you've defined) and RKOGLU01 for the user logs. They are physical sequential datasets that should be defined as fixed block with a logical record length of 133. With current DASD technology, 27930 seems to work out as an efficient block size. At this juncture of your automation development, it is unlikely that you will require 'Retain View' logs RKOGLH01/02. If you happen to use this facility, the logical record length is 480 with the rest of the parameters the same as the log files.

In addition to defining the logs in the JCL (and removing the '//RKOGLOGM DD SYSOUT=&outclass' statement), you will need to specify DISKMLOG in the AF/OPERATOR parameter file. If check pointing of TOD traps is in use and RELOAD is either TOD or ALL, the parameter TODISYNC should be set to "Y." If it is set to "N," the check pointed TOD traps will begin to run their scripts as soon as AF/OPERATOR starts and it's almost certain you'll want the rest of the environment to be initialized. The AO TODSTART command will then need to be added to the system initialization process sometime after the tasks which the TOD activities are impacting are initialized. This is probably just after the system IPL is complete, or at least after the USER SYSTEM variables are added and any requisite files are allocated.

What happens in the middle?
Between IPL and shutdown, the day-to-day automation takes place. The level of automation support during "working hours" varies from site to site. Manual processes executed by MVS operations should be automated as far as practical. Alert management and response technology should—as far as possible—eliminate manual intervention. All of this depends on a number of variables, not the least of which is the available technology, the mission of the automation team, and management support. A thorough discussion of this technology is a fitting subject for extended discussion, but beyond the scope of this article, which is dedicated to the notion of a single —button— start and stop process.

System shutdown
On the back end of the process, it is critical to assure that system shutdown proceeds in an orderly manner. For instance, in a JES2 environment, you must have received the $HASP099 message before the $PJES command is issued. This message is only issued after all JES resources are down. So, as a pre-stop process for JES, issue a D A,L and parse the output to assure all tasks—except JES, AF/OPERATOR, LLA, VLF and any other SUB=MSTR task stopping after JES—are down. OMVS creates subtasks that run under JES that do not show up in a D A,L display. For these you should issue a D OMVS,ALL command and trap and parse the output to assure all OMVS components are down.

The bottom line of this code assures JES is brought down successfully without having to "abend" it. If the $HASP099 message does not appear, there is some task or subtask of JES still active in the system and it must be brought down before the $PJES2 command can be issued successfully. This command should not be issued until the '$HASP099 ALL AVAILABLE FUNCTIONS COMPLETE'message appears on the console. It can have an adverse affect on stopping the remaining resources and may force operations to ABEND JES. This is not an action recommended or necessary for proper system shutdown. The long and short of this discussion is that you need something like:

"WAIT WTO('$HASP099*') ENA SECS(600) ACT('OPER '' $PJES2''')"

to stop JES. An ordinary WTO trap would work fine, but there are actions to take subsequent to the JES shutdown message receipt.

When JES terminates, stop LLA and VLF and wait for them to come down. These are simple P or STOP commands that execute very quickly. There may be other stop commands you need to issue to get the system completely purged. In general, the system security task does not need to be stopped, but some shops run some clean up code after JES is down.

At this juncture the system is all but down. Only three things remain to be done:

• Stop AF/OPERATOR
• Issue the Z EOD command
• Vary the LPAR out of the SYSPLEX

If this is the last LPAR in the SYSPLEX, the V XCF must be issued manually or by an outboard automation source. If there are LPARs remaining and AF/OPERATOR is up and running, the vary can be handled by routing a command to the remaining LPAR. This will set a TOD trap for a few seconds in the future that issues the vary to the coupling facility. The command looks like:

"OPER 'RO "plex_smfid", EX TODVXCF "''AOSMFID "'''"

… where plex_smfid is the focal LPAR for the SYSPLEX and presumably the last one down. TODVXCF is a script that adds a few seconds to the current time, then executes a script:

In this example, the variable 'targ_smfid' is passed to the script from the TOD trap generated when the route command is received from the partner system.

Now the final two steps:

• Issue "OPER ''Z EOD'''

• Issue a "OPER 'C(ancel) &AOTASK'"

A cancel is acceptable if and only if all AF/OPERATOR resources are down. The only resources up after all other resources are down are files that may need to be closed and freed, and any other termination requirements that may be part of shutdown, such as displays of statistics. The cancel of AF/OPERATOR is the last command you can issue. The Z EOD will wait until the last task is down before executing.

Concluding thoughts
With MVS automation in the data center for over 15 years now, it's time to use it to its full potential. Issuing commands manually for almost any reason is at best slow and at worst chancy. Mainframe availability is far too important to risk error when the correct solution is easily programmed.