EXECSYMB, SYSTEM_MGMT, Executive Symbiont EXECSYMB V3.6.1 John Osudar Chemical Technology Division Argonne National Laboratory 205 L-213 9700 South Cass Avenue Argonne, IL 60439-4837 Phone: (708) 252-7505 FAX: (708) 252-5912 Electronic mail addresses: Internet: OSUDAR@CMT.ANL.GOV DECUServe: OSUDAR NOTE ==== If you would like to receive notification of future bug fixes and enhancements to EXECSYMB, please send your name, mailing and/or electronic address, and phone number to me at the above location. No guarantees or promises, of course... +---------------------------------------------------------------------+ | See bottom of document for "release notes" for the latest versions! | + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + | See the file SAMPLEPROC.COM for examples and further documentation. | +---------------------------------------------------------------------+ Introduction ------------ EXECSYMB is a VMS server symbiont that feeds queue entry information to detached processes that are designed to execute specific operations (hence the name, Executive Symbiont). It is multi-threaded (it can handle up to 32 queues at the same time); it requires one detached process and one mailbox for each queue, plus one mailbox for status information and one for detecting process exit status for each copy of EXECSYMB (i.e. one per 32 queues). EXECSYMB can be used to implement a wide variety of VMS queue-based operations. Among its present applications are: (1) Remote queuing of print jobs (and limited remote queuing of batch jobs) (2) Pre-processing and post-processing of jobs destined for other server symbionts (e.g. performing MFENET gateway operations on files queued to MFENET inbound-file symbionts) (3) Queued file transfer for DECnet and other networks This program was written with one major goal in mind: to simplify the system's interface to a user-written queue processor, eliminating (or at least hiding) the quirks that are present in the job controller-to-symbiont interface supplied by Digital. Under VMS V4 and later, symbionts operate in a rather strange environment. A symbiont is a detached process (unlike VMS V3, where it was a subprocess of the job controller). However, it lacks several useful things that most processes have: symbionts have no CLI, which means that they can't spawn commands; symbionts are always created with username SYSTEM and one privilege (SETPRV); and symbionts cannot access process-permanent logicals (SYS$INPUT, SYS$OUTPUT, SYS$ERROR) the way other processes can. A symbiont's SYS$INPUT points to a mailbox to which the job controller writes its request messages, and SYS$OUTPUT points to another mailbox from which the job controller reads status messages from the symbiont. (Incidentally, the latter has been the job controller's "achilles heel"; if your symbiont accidentally writes a bunch of stuff to SYS$OUTPUT, the job controller will respond with a series of console error messages about "invalid mailbox message received". Under some versions of VMS, this is immediately followed by a fatal "end-of-file on job controller mailbox" that aborts the queue manager, stopping ALL queues on your system and leaving active batch jobs in VMS limbo. VMS V4.4 and beyond allows the queue manager to restart after such an abort, but this usually leads to a recurrence of the problem, and a hard abort on the part of the queue manager.) EXECSYMB was designed to make the symbiont environment easier to use. Of course, EXECSYMB itself has to deal with the restrictions listed above. However, a queue handled by EXECSYMB is processed by an EXECSYMB detached process, which is created with the DCL CLI, a real command procedure as SYS$INPUT, a log file as SYS$OUTPUT, and a pair of mailboxes for talking to EXECSYMB. The protocol for passing information between the executive symbiont and the detached process involves transferring "items", which EXECSYMB parses from the job controller's request message, through a mailbox. One of the "items" is created by EXECSYMB itself -- it's called "EXEC_STEP", which is an "execute trigger", telling the detached process that a string of items has been completely transferred and an operation (execute, reset, exit) is to be performed. The detached process returns a completion status through the other mailbox, which provides EXECSYMB with a task status to return to the job controller. As a result, it is relatively easy to write a DCL command procedure to process a queue. There are just a few requirements for such a procedure, which are outlined below, in the section "sample queue processing procedure". Features of EXECSYMB -------------------- In addition to providing a safer and less hostile environment for queue processing, EXECSYMB adds some features that aren't available to ordinary user-written symbionts without additional coding. (1) EXECSYMB is multi-threaded, so that it can handle a number of different queues at the same time. The code required to do this correctly is rather complex, so concentrating it in EXECSYMB is quite practical. (2) EXECSYMB handles the AST-level delivery of job controller messages for things like stop and pause requests. (3) EXECSYMB eliminates the need for calling the symbiont support routines to parse, copy, and re-copy the request message components (request items), as is done in most user-written symbionts. EXECSYMB parses the request message itself, and keeps pointers to the individual items; only those items required by a particular queue are actually passed to the queue processor. (4) EXECSYMB provides a requeue-on-error capability that can be selected on a per-queue basis. If a queue specifies a retry interval, and a task is aborted with an error status, the job is automatically requeued to try again after the specified interval. It is also possible to specify whether the job is to be restarted from the task that aborted, or from the beginning. (Note that, if this feature is selected for a particular queue, the queue processor must do some reasonable error detection and recovery, returning an "error" status to EXECSYMB only if it really wants the job to be retried.) (5) EXECSYMB allows a queue processor to handle all copies, the first copy only, or the last copy only, for jobs that specify multiple job copies or file copies. (This is especially important when the queue processor is transferring one copy of the file, and forwarding the number-of-copies parameter for handling elsewhere.) (6) EXECSYMB allows the queue processor to be started dynamically, when the queue actually has work available. This option is selectable on a per-queue basis, by specifying a "timeout" interval. If the queue processor detached process exists but is idle, and no work is queued within the timeout period, EXECSYMB will tell the process to exit (while keeping the queue in the active state, as far as the VMS job controller/queue manager is concerned.) Running EXECSYMB ---------------- Here's what you need to know to run EXECSYMB. (Remember that a symbiont image must reside in SYS$SYSTEM.) The EXECSYMB process will change its name to "ExecSymb3.0.1_n" where n is a sequence number that starts at 1. Since each process handles up to 32 queues, you shouldn't see sequence numbers above 1 unless you have more than 32 queues processed by EXECSYMB at the same time. EXECSYMB starts with its base priority set from the /BASE_PRIORITY=n qualifier on the first queue that it starts; however, soon after starting it will set its base priority to 7. This is done so that EXECSYMB's priority will be above that of the queue processors (which are created with a base priority of DEFPRI, which is usually 4) and below that of the job controller (whose base priority is 8.) EXECSYMB sends important messages, such as startup and shutdown of the main process, to the OPER12 operator. Other information is written to the log file, which is named SYS_EXECSYMB:EXECSYMBn.LOG (where "n" is as described in the process name above). If SYS_EXECSYMB is not defined in the system logical name table, log files are placed in SYS$MANAGER:. (The "queue processor" detached processes also place their SYS$OUTPUT and SYS$ERROR log files there. As of V3.2.6, it is possible to redirect a particular queue's .OUT/.ERR files by defining a system logical name, EXECSYMB_OUTDIR_queuename, pointing to the destination directory.) EXECSYMB remains active as long as it has at least one active queue to process; upon stopping the last queue (and waiting for its queue processing detached process, if any, to exit), EXECSYMB will exit. EXECSYMB can send items to the detached process in ASCII or binary form. The ASCII form uses two writes per item, one to send the item name, the other to send its value. The binary form does one write per item, and includes the item's binary code number as the first four bytes of the record. In the ASCII form, many item values are converted from their raw form to an edited ASCII form; e.g. the UIC item will be converted from four binary bytes to a string of the form [gggggg,mmmmmm]. The binary form does no editing on item values. In addition, EXECSYMB can be told to skip sending items that have a null string as a value. This can save a lot of processing. EXECSYMB terminates the list of items for a particular task with an item that it makes up itself. This item has the name "EXEC_STEP", binary code 0, and can have one of three ASCII values: "EXECUTE", which means that the task's item list has been completely transferred, and the task is ready for execution; "RESET", which means that a partial item list was sent, which should be discarded, and the detached process should return to its initial state; and "EXIT", which means that the detached process should exit. The "EXECUTE" command expects the detached process to return a completion status, and then perform a "RESET" automatically. EXECSYMB can also send the queue processor another item that it makes up itself. This item, named "EXEC_FLAGS", binary code -1, is a character string containing a list of "task flags", indicating specific characteristics of the current task which are not directly available otherwise. This string consists of a leading slash ("/"), a list of (zero or more) uppercase-only keywords separated by slashes, and a trailing slash. Thus, to determine if a specific keyword is present in the item value, you can search for the keyword surrounded by slashes. At the present time, the only keyword that can be passed in this item is "SPOOL", which means that the file associated with this task is a spool file (i.e. a file created by writing to a spooled device). You can tell EXECSYMB how to process a queue through two queue qualifiers: the command procedure that is to be used is specified through the /LIBRARY qualifier, and the list of processing options is specified as a character string in the /SEPARATE=RESET qualifier (which corresponds to the item JOB_RESET_MODULES). Since device control libraries and job reset modules aren't too meaningful in the context of server symbionts, use of these two qualifiers works out quite well. The /LIBRARY qualifier takes a filename as its value. Although device control libraries are required to reside in SYS$SHARE and have the extension .TLB, the /LIBRARY qualifier does not place these restrictions on the value you specify. (In fact, the job controller sticks a SYS$SHARE: on the front of this value, and a .TLB on the end, but EXECSYMB strips those off!) Thus, /LIBRARY=SYS_PROCS:EXAMINEQUEUE.COM is a valid specification. Note that this field is restricted to a total of 39 characters (since the job controller thinks this is the filename part of a library name.) The /SEPARATE=RESET qualifier takes an arbitrary character string as its value. EXECSYMB requires this string to be a comma-separated list of options from the following list: TIME="d hh:mm:ss.cc" This specifies how long to wait before retrying aborted jobs. If not specified, aborted jobs are not retried. DYN="d hh:mm:ss.cc" This specifies the timeout period for a queue with a dynamically-started queue processor. If not specified, the queue processor is always active while the queue is started. ITEMS="itemlist" This specifies the list of item codes to be passed to the queue processor. Codes are represented as decimal numbers, and the syntax mm:nn can be used for a range of codes. SPOOL=[spooldir] This specifies a directory into which spool files can be entered, if it is possible for the queue to include spool files (i.e. files created from output to a spooled device.) Such a directory MUST exist on all disks that are used as secondary devices for spooled devices. ASCII or BINary This specifies the format for passing item values to the detached process. PRINTer or SERVER This specifies the queue type, and is only used to set the indicator for SHOW QUEUE (printer queues print, server queues process) NONULl or NULL This specifies whether items whose values are the null string are passed to the detached process CHECK or NOCHECK This specifies whether the queue can checkpoint jobs, i.e. whether a job consisting of several tasks in which one task aborts is restarted from the point of abort (CHECK) or from the beginning of the job (NOCHECK) COPY={ ALL, This specifies which copy of a multi-copy job, First, and which copy of a multi-copy file in a job, or Last} is to be processed by the detached process. ALL gets every copy, FIRST gets the first file copy in the first job copy, and LAST gets the last file copy in the last job copy. (Note that certain flags in the SEPARATION_CONTROL item may be set only on the FIRST or LAST copy.) USER=username If specified, this sets the username under which the queue processor detached process will run. (The default is SYSTEM.) This can be useful for accounting, or for processors that use proxies into other DECnet nodes. Note that the specified username need NOT be a valid one on the system (i.e. there doesn't have to be an entry in SYSUAF for that username)! FLAG If specified, indicates that the queue processor wants the "EXEC_FLAGS" pseudo-item to be passed prior to the "EXEC_STEP" item for each task. INIT If specified, indicates that the dynamic queue processor should be started at queue startup (so that it can do some initializations). Defaults, if a particular option is not specified, are: no requeue, non-dynamic queue processor, no ITEMS, no SPOOL directory, ASCII, SERVER, NULL, CHECK, COPY=ALL, USER=SYSTEM, no FLAG, no INIT A little more needs to be said about spool file handling. EXECSYMB can be used to process queues from spooled devices (such as spooled null printers, used for remote printer support). When output is written to a spooled device, a spool file is created on the device's secondary device (disk), and queued to the device's queue. This file is identified only by file ID, and has no directory entry. In some applications (e.g. where the file will be copied to another node with a network copy program), the file must have a full filename to be effective. EXECSYMB will detect spool files for queues that specify the SPOOL=directory option, and will enter such files in the specified directory on their device. Once such a file has been processed (and is about to be deleted automatically by the job controller), EXECSYMB removes the directory entry. Sample queue processing procedure --------------------------------- The following is a sample queue processing procedure that can be used with EXECSYMB. It is reproduced here to illustrate the required features of such a procedure; it can also be found, without comments, in this directory in the file EXAMINEQUEUE.COM (which can be used as-is with EXECSYMB.) This procedure does nothing "useful"; it produces a log file containing the items passed for each request it processes, and it indicates successful completion of each task. First, you can specify any setup commands you require. SET NOVERIFY and SET NOON are recommended. $ SET NOON $ SET NOVERIFY Next, the procedure must locate the mailboxes that it uses to talk to EXECSYMB. The input mailbox (written by EXECSYMB, read by the procedure) has a logical name of CMD_MBX_pid where "pid" is the process ID of the detached process running this procedure. The output mailbox (written by the procedure, read by EXECSYMB) is used for returning status, and has the name STAT_W_MBX_pid (where "pid", again, is the process ID). We can also determine the name of the queue we are processing by translating the logical name QUEUE_NAME_pid. Thus, the procedure must determine its own PID, and then use it to access this information. $ PID=F$GETJPI("","PID") $ QUEUENAME=F$TRNLNM("QUEUE_NAME_''PID'") Before we do any real work, we must open the input and output mailboxes for talking with EXECSYMB. (If we decide to reinitialize after detecting a mailbox I/O error, we may want to come back here as well.) $INIT: $ OPEN/WRITE/ERR=NOMBX STAT STAT_W_MBX_'PID': $ OPEN/READ/ERR=NOMBX INPUT CMD_MBX_'PID': The items sent by EXECSYMB all have names that can be used as DCL symbols. That is the technique used here. (EXECSYMB has an alternate format for items, in which a fixed-length binary item code precedes the item value; that format is intended for use by compiled programs that run in the detached process.) The item "EXEC_STEP" is used by EXECSYMB to guide the operation of the detached process; EXECSYMB can assign it the values "EXECUTE", "RESET", and "EXIT". Initially, or after an "EXECUTE" operation is complete, or when EXECSYMB explicitly requests a "RESET" (such as when a job is deleted while being processed), the procedure should reset controlling variables to initial values, and return to its input processing loop. This procedure has one controlling variable, EXEC_STEP. Here, we initialize the symbol to the value "INPUT", and subsequently use the symbol as the destination label of a GOTO. This is a rather simple, yet effective, way of implementing the control operation. (We also write out a separator line to the log file, to mark the boundary between requests.) $RESET: $ WRITE SYS$OUTPUT "========================================" $ EXEC_STEP="INPUT" Now comes the good part of the procedure. It starts with the label that was initially assigned to EXEC_STEP. The first command reads the NAME of the symbol (i.e. the item name) being passed, and the second command reads its VALUE. Thus, EXECSYMB can tell the process which item is being passed, and its value, and a symbol of that name with that value is created. $INPUT: $ READ/ERR=ERRMBX INPUT SYMBOL $ READ/ERR=ERRMBX INPUT 'SYMBOL For demonstration purposes, we echo the information to SYS$OUTPUT, which is the log file. $ WRITE SYS$OUTPUT SYMBOL $ WRITE SYS$OUTPUT 'SYMBOL Having completed the value assignment, we go to the location pointed to by EXEC_STEP. This was initially INPUT, and will remain so until EXECSYMB sends the item EXEC_STEP and a new value (EXECUTE, RESET, or EXIT) for it. $ GOTO 'EXEC_STEP When EXECSYMB says "EXECUTE", it indicates that the item sequence for this task is complete, and should be acted upon. A real symbiont processor would do something with the item list at this point (e.g. do something with the file specified in item FILE_SPECIFICATION). Once the operation is complete, EXECSYMB expects a status to be returned via the mailbox. This example always returns a success status. After it reports completion, the procedure loops to the "RESET" label to reset controlling variables and re-enter the input loop. $EXECUTE: $ WRITE STAT "%X00000001" $ GOTO RESET If an "EXIT" operation is requested, EXECSYMB wants the detached process to exit. One way to do this is to log off; you can also do an EXIT or whatever else you desire (including, for example, sending operator notification that the detached process is exiting.) $EXIT: $ LOGOUT Finally, we add some error handling for the cases where we failed to open one of the required mailboxes, or got an I/O error on a mailbox. If we can't open a mailbox, we cannot continue, so we exit (logoff). If we got an I/O error, we can close and attempt to reopen the mailbox, in which case we'll either succeed or the open-failure code will terminate us. $NOMBX: $ LOGOUT $ERRMBX: $ CLOSE STAT $ CLOSE INPUT $ GOTO INIT Release notes for EXECSYMB V02.32 ================================= 15-Jun-1989 Changes since the previous release (V02.29, 23-Feb-1988) are as follows: 1. Synchronization between AST-level and non-AST-level code was improved. Also, use of hibernation and wakeup calls was improved to cut down on the number of system service calls. 2, The mailbox message size for the status mailbox was increased from 12 to 268 bytes, to allow room for accounting, checkpoint, and device status information. 3. The status message format was extended to allow specification of accounting data, checkpoint data, and device status. The program logic was enhanced to recognize "intermediate" status messages containing checkpoint and device status information. (In other words, instead of only recognizing a status message from a stream at task completion, EXECSYMB will also accept such messages during task execution, to provide checkpoint data or device status to the job controller.) The old status message was required to have the format %Xnnnnnnnn where the n's are hex digits. The new status message can have two formats. An intermediate status message looks like this: ,{optional device status}{,optional checkpoint data string} A task completion status message looks like this: {,optional accounting data} Status values are numeric longwords, represented as either a signed decimal number, or a hex number preceded by "%X". Accounting data consists of four such numbers, representing the four accounting data fields (pages printed, QIO's to printer, GET's from source, and hundredths of CPU-seconds used.) Checkpoint data is an arbitrary character string. For more details, read the SMB section of the VMS Utility Routines manual. A note about the "device status" field: this can be used to specify various queue status or attribute values. There are also some unsupported bits in the status (at least as of VMS V5.4), which may be quite useful in some applications. Here is a brief summary of the bits in the device status field as of VMS V5.4: SMBMSG$V_name bit# mask value ============= ==== ========== LOWERCASE 0 1 Sets "Lowercase" attribute PAUSE_TASK 1 2 Initiates pause-queue REMOTE 2 4 Sets "remote" attribute SERVER 3 8 Sets "server" attribute STALLED 4 16 Indicates queue "stalled" STOP_STREAM 5 32 Initiates stop-queue TERMINAL 6 64 Sets "terminal" attribute UNAVAILABLE 7 128 Sets "device unavailable" SYM_NOTIFIES 8 256 ? SYM_REQUESTS_OPER 9 512 ? SYM_COPIES_FILE 10 1024 ? SYM_COPIES_JOB 11 2048 ? SYM_ACCEPTS_ALL_FORMS 12 4096 Allows all jobs, any /FORM SYM_NO_JOB_CHECKPOINT 13 8192 ? 4. A feature called "permanent processor support" had been added to EXECSYMB in V02.27. This was supposed to provide a way for an existing detached process to "register" itself as a queue processor (instead of EXECSYMB creating the process when the queue was started. The feature wasn't designed too well, and was only added to support one specific application (which, naturally, ended up never using it). Because permanent processor support was incompatible with the intermediate status message support described above (which *is* useful), and because, to my knowledge, nobody ever used it, permanent processor support *has been removed* from EXECSYMB as of V02.32. ---------------- If you built something using the "permanent processor" feature, it won't work with versions of EXECSYMB beyond V02.29 -- and I would like to hear from you to discuss alternative implementations. Release notes for EXECSYMB V3.0.1 ================================= 12-Jul-1989 Changes since the previous release (V02.32, 15-Jun-1989) are as follows: 1. Use (and misuse) of event flags in previous versions was completely cleaned up. This may have been at least partly responsible for "strange behavior" in past releases of EXECSYMB. 2. Cosmetic changes to process names were made (EXECSYMB process has a name like "ExecSymb3.0.1_n" instead of "SYMBIONT_EXEC_n" and queue processors are named "nx=queuename" instead of "nx_queuename".) 3. EXECSYMB determines whether it is running on VMS V4.x or a later version. If a later version, it uses the new queue manager features that were added in VMS V5.0 (e.g. queue name no longer needs to be specified when changing a particular entry's attributes.) 4. EXECSYMB now explicitly sets its base priority to 7 (one below the current level used by the job controller), instead of using whatever was specified as the /BASE_PRIORITY qualifier on the first queue started for this symbiont. Also, EXECSYMB now uses the system parameter DEFPRI to determine the base priority of the queue processors that it creates. (Thus, they will typically run with base priority 4.) 5. A bug in the protection attached to the status mailbox was fixed. Prior to this version, the status mailbox was inadvertently left with full access for all users, when it should have had full access only for SYSTEM processes. 6. Some additional stream status information was added, to reduce the amount of time EXECSYMB uses for scanning streams that have never been active (both in its "main loop" and when a status message is received.) 7. Logfile messages (and some operator messages) were shortened to eliminate extraneous text. (Since debugging symbiont problems required finding the message in the source code anyway, having a longer message didn't really provide any useful information.) 8. For several versions prior to this release, it had not been possible to get null items sent to the queue processor (i.e. all queues were effectively "NONULL" whether they specified it or not.) This bug has been fixed. (Most queue processors will still want to specify NONULL for efficiency's sake.) 9. Previous versions of EXECSYMB did no validity checking of the requeue-time or dynamic-time queue parameters; such checks are now included. 10. This isn't really a change, but should be noted somewhere: EXECSYMB has always used the LIB$FID_TO_NAME routine to convert a file-ID into a full filespec. Prior to VMS V5.0, this routine was supplied with EXECSYMB as an object module (and a Bliss-32 source file). VMS V5.0 added LIB$FID_TO_NAME to the LIBRTL, so EXECSYMB for V5.0 and beyond no longer requires the explicit object module reference in its build procedure's LINK command. 11. The (undocumented) "debug" mode was enhanced to allow it to be turned on and off dynamically, and to display all current status information. Release notes for EXECSYMB V3.1.1 ================================= 02-Mar-1990 Changes since the previous public release (V3.0.1, 12-Jul-1989) are as follows: 1. EXECSYMB now saves the "device name" specified in the "/ON=" qualifier for each queue, and creates a logical name DEVICE_NAME_pid (where "pid", as usual, is the process ID of the queue processor for that queue.) This provides a way to pass a queue-dependent parameter of up to 31 characters directly to the queue processor. Note that the SCS node name portion of "/ON=" is *not* part of this parameter. Also note that VMS imposes no syntax restrictions on the "device name", as long as the string is enclosed in quotes. An example of the use of this feature: we have several queues that use an NJE package to send print files to devices on our IBM systems. In the past, we used a separate command procedure to drive each queue processor -- now we use a single procedure, and specify the destination for each queue through the "/ON=" parameter. It makes things simpler and more flexible, and it's nice to see the information displayed in a SHOW QUEUE instead of being buried in a command procedure somewhere. 2. A bug that prevented EXECSYMB from stopping some hung queue processors has been fixed. In past releases, if a queue processor changed its UIC to something other than [1,4] and then became hung, EXECSYMB's exit timeout code would fail to stop the process because EXECSYMB didn't enable WORLD privilege. As of this release, EXECSYMB correctly enables WORLD privilege, and thus is able to stop any hung processor. (All symbiont processes, such as EXECSYMB, are started with SETPRV and no other privileges enabled -- so fixing this problem simply required adding WORLD to the list of privileges that EXECSYMB enables.) Release notes for EXECSYMB V3.1.2 ================================= 21-Jun-1990 Changes since the previous public release (V3.1.1, 02-Mar-1990) are as follows: 1. One very minor bug fix, to the code that re-converts the "TIME=" and "DYN=" queue parameters to ASCII (to reset the parameter string after the queue is started), to display the hours correctly. Release notes for EXECSYMB V3.2.4 ================================= 20-Aug-1990 Changes since the previous public release (V3.1.2, 21-Jun-1990) are as follows: 1. The ability was added to disable requeue-for-retry (i.e. to force a job to complete with an error or fatal error status) in a queue that is enabled for retry (i.e. for which a TIME= parameter is specified). This feature is implemented by making the status (written to the EXECSYMB status mailbox) the negative of the actual status. 2. EXECSYMB now defines some additional logical names in the group [1,*] logical name table (LNM$GROUP_000001). These names may be used to determine the association of queues, streams and queue processor processes related to a particular copy of the EXECSYMB symbiont process. Every time a stream is started by EXECSYMB process number N (i.e. the one whose process name is "ExecSymb3.2.4_N"), a logical name of the form EXECSYMB_Nx is created (where x is the stream number encoded as a "base 32" digit, 0-9 or A-V); its equivalence name is the name of the queue for which that stream was started. This logical will remain defined for as long as the EXECSYMB process exists (although its equivalence name may change.) Also, when a queue processor process is actually created, a name of the form EXECSYMB_PID_Nx is defined, whose equivalence name is the process ID of the queue processor. Thus, for any EXECSYMB process, you can determine the queues and active queue processors that belong to it. 3. The $DEBUG mode "EXIT" command (executed by writing the string "$DEBUGEXIT" to the symbiont's status mailbox) was changed to stop all streams' queue processors. A "QUIT" command (i.e. "$DEBUGQUIT") was added to do the "immediate exit" that was formerly what EXIT did. This change was made to provide a clean way of stopping the symbiont and its associated processes after a queue manager failure. (A sample command procedure, EXECUTIL.COM, is supplied with the source code, illustrating some of these features.) Release notes for EXECSYMB V3.2.6 ================================= 01-Apr-1991 Changes since the previous public release (V3.2.4, 20-Aug-1990) are as follows: 1. Added queue-specific locations for the *.OUT and *.ERR log files. If the logical name "EXECSYMB_OUTDIR_queuename" is defined in the system logical name table, its value is used as the device/directory where the log files for queue "queuename" are created. Release notes for EXECSYMB V3.2.7 ================================= 23-Oct-1991 Changes since the previous public release (V3.2.6, 01-Apr-1991) are as follows: 1. Added a queue parameter "INIT", which is used with the "DYN=" parameter to indicate that the dynamic queue processor is to be started when the queue is first started. (Ordinarily, when a queue is started and a dynamic queue processor is specified, the processor is not started until a job is available for processing.) This is useful if the queue processor procedure needs to do some initialization at queue startup, such as sending device status bits to the job controller. The queue processor will be started with the usual dynamic timeout. Release notes for EXECSYMB V3.3.1 ================================= 31-Mar-1992 Changes since the previous public release (V3.2.7, 23-Oct-1991) are as follows: 1. Initial update to include new items defined by VMS V5.5. (All new items are presently passed untranslated.) Note that previous releases of EXECSYMB *will not work* under VMS V5.5; however, this release of EXECSYMB *will* work with previous releases of VMS. 2. Compiling this version of EXECSYMB with a VMS V5.5-compatible SMBDEF.INC will produce a symbiont that will work with VMS V5.5 as well as previous releases of VMS. Release notes for EXECSYMB V3.4.1 ================================= 14-Oct-1992 Changes since the previous public release (V3.3.1, 31-Mar-1992) are as follows: 1. The code was cleaned up a bit: an IMPLICIT NONE declaration was added to each routine, and all previously undeclared variables were declared. 2. An occasional problem was noted in V3.3.1 (and prior versions), which appears to be due to a race condition between EXECSYMB and a queue processor whose exit has been requested. The queue processor exits before EXECSYMB flags the fact that the exit is expected. As a result, the queue processor exit is treated as an unexpected abort, and EXECSYMB stops the queue. This problem is evidenced by EXECSYMBx.LOG message sequences like this one: 17-MAR-92 09:46:34 Exitstat: stream 7 = %X10000001 17-MAR-92 09:46:34 Exit request for processor 00000703 The "Exitstat" message indicates an unexpected abort on stream 7; the next message is logged by the EXECSYMB routine that requests a queue processor exit (and which caused the exit that resulted in the first message). Although I don't see how this race condition can happen, I have now changed the exit-request code to set the "exiting" flag before the exit request message is sent out. ***NOTE*** If you still see message sequences like the one listed above after installing V3.4, please contact me with details! 3. Some additional output code was added to the debugging mode, primarily to dump the contents of messages received from the job controller. 4. This is an important change, especially in light of the instability of the V5.5-x job controller/queue manager (both "classic" and "new" versions). When the job controller aborts, in its last gasp it kills off all symbiont processes, so that there are no orphan symbionts sitting around with nobody to talk to them. EXECSYMB gets zapped this way -- but any active queue processors started by EXECSYMB will just keep running, since nobody will tell them to stop. The only way to fix this is to add a user rundown routine (which runs in kernel mode at image/process rundown) and have that routine do the cleanup. (An exit handler won't do the job, since the job controller simply does a $DELPRC on each symbiont process -- thus, an exit handler would not be called.) After several job controller crashes in recent weeks, I finally decided that it was time to write such a routine, which meant creating a user- written system service. As long as I was doing that, I took the other EXECSYMB kernel-mode code -- the SETUSER routine -- and created a user- written system service for it. The combination of SETUSER and the rundown routine are now in a separate image, EXECSYMB_KM.EXE, which must be installed /SHARE/PROT in order for EXECSYMB to work properly. Since I have no VMS V4.x systems left on which I could test this, I am only including this support in the VMS V5.x-compatible EXECSYMB. The rundown routine will deassign all logical names created by EXECSYMB and will send an exit request to each active queue processor. (It does *not* do a $DELPRC on the queue processors -- instead it allows them to do a normal exit.) 5. The "new, improved" VMS V5.5 queue manager behaves differently when it receives a TASK_STATUS message from a symbiont. (EXECSYMB sends a TASK_STATUS messages when it receives an intermediate status message from a queue processor.) In particular, it is no longer legal to send a TASK_STATUS message if there is no job active on the queue. The INIT feature (added in EXECSYMB V3.2.7) was provided specifically for the purpose of allowing a queue processor to send device status information back to the job controller (via an "intermediate status" message to EXECSYMB). This would be used to set queue characteristics (e.g. ACCEPTS_ALL_FORMS) before any jobs were actually processed. However, there was no *requirement* that the queue processor actually send an intermediate status message; the queue would still start correctly if no such message was sent. The new queue manager doesn't allow EXECSYMB to use a TASK_STATUS message to forward the device status information; instead, the device status must be sent as part of queue startup. As a result, effective with V3.4.1, if you specify the INIT flag on an EXECSYMB queue, your queue processor *must* send an "intermediate status" message when it starts up. Until the message is received by EXECSYMB, the queue will remain in the "starting" state, as EXECSYMB will be waiting to receive the device status information for forwarding to the job controller. 6. I fixed a bug in the queue processor exit processing code which would return a spurious STOP_STREAM completion message to the job controller. Ordinarily this message would be ignored without any ill effects. However, with the new V5.5 queue manager, the extra message would cause EXECSYMB to abort and produce a process dump in SYS$SYSTEM: *if* the spurious message occurred on the last active queue. Under those circumstances, the job controller thinks the symbiont has exited (since its last active queue reported a STOP_STREAM already) and so the extra message appears to come from a process that isn't a symbiont. The job controller returns an "invalid message" status which aborts the process and triggers the dump. Release notes for EXECSYMB V3.5.2 ================================= 13-Jan-1994 Changes since the previous public release (V3.4.1, 14-Oct-1992) are as follows: 1. AXP support was added. This involved no code changes to EXECSYMB.FOR itself. The MACRO-32 compiler complained about some of the code in one of the support modules (ENTREM), so I cleaned that up. EXECSYMB_KM was modified to assemble on the VAX or compile on the AXP with a suitable symbol definition; I used the templace user-written system service code (provided as C code on the AXP) but wrote the equivalent in MACRO-32 to keep the whole thing as one module (and to avoid the need for a C compiler on top of the Fortran compiler!) Note: this version of EXECSYMB *should* work with VAX VMS V6.0; apparently there were no changes to the symbiont interface in the new VMS release. (The $SMBDEF modules are identical for both releases.) I would expect that you'd at least have to relink the images under V6.0, though. 2. The new DEC Fortran compilers produce warnings for things like declared but unused variables. I cleaned up the code to eliminate all the warnings. While doing that, I also did some minor cleanup on other code sequences that I found annoying for one reason or another. 3. The error handling for queue processor process creation was somewhat inadequate. Since errors in that operation are extremely rare, I never noticed this deficiency. While testing the AXP version I managed to cause a process creation failure, and found that EXECSYMB didn't handle it too well. That problem has now been corrected. 4. I was asked whether queue processor working set parameters could be adjusted, to cut down on page faults. Since VMS supports use of the /WSDEFAULT, /WSQUOTA, and /WSEXTENT qualifiers on symbiont queues, it seems that EXECSYMB should honor these when creating the queue processor (along with the /BASE_PRIORITY qualifier). I've now added the code required to do this. Since it uses the $GETQUI system service, this code is certainly VMS version dependent; I've disabled its use if you're running a version prior to V5.0. Note -- VMS doesn't display the working set information for non-batch queues, so I've supplied a short Fortran program (SHOWQUEWS.*) that will do this. 5. During the beta testing of V3.5, we discovered that a STOP/QUEUE/RESET on an EXECSYMB queue would make the queue unstartable. (Attempts to start the queue would hang in QUEMAN, and the queue would remain in "starting" state forever.) It turns out that the VMS queue manager's handling of STOP/QUEUE/RESET changed in some recent release prior to V6.0: what had been a valid symbiont response to the queue manager's RESET request now causes the queue to hang as described above. To work around this problem, code was added to EXECSYMB to distinguish between a normal STOP request and a RESET request. 6. While I was working on the problem described in (5), I fixed some bugs in the built-in debugging code (DUMP mailbox command) and added a minor enhancement to that code (ability to dump specific stream data). Release notes for EXECSYMB V3.6.1 ================================= 14-Jul-1994 Changes since the previous public release (V3.5.2, 13-Jan-1994) are as follows: 1. Bug fix required for VAX VMS V6.x -- EXECSYMB's version detection code had a really stupid bug in it, which caused a bunch of messages in the EXECSYMB log files showing a status of 00000154 (%SYSTEM-F-IVLOGNAM), and prevented the symbiont from working. 2. Bug fix in EXECSYMB_KM; a line of code was placed in a VAX-only section that should have been in the architecture-independent section. This would have caused problems in the AXP flavor of EXECSYMB_KM. 3. One new feature: a new flag "HOLD" is recognized in the queue parameter string (i.e. the /SEPARATE=RESET string). Prior to V3.6, if you wanted jobs to be requeued for retry on a failure completion status, you had to specify a wait time via the "TIME=" queue parameter. I ran into many cases where I just wanted the jobs held indefinitely (e.g. if the queue processor modifies and requeues the job for processing elsewhere, and then releases it). HOLD enables you to do this. If you specify "HOLD", don't specify "TIME=", and a job reports an error completion status, the job will be requeued and put on indefinite hold.