mmwatch command
Administers clustered watch folder watches and lists all of the active watch folder API watches.
Synopsis
mmwatch Device list [--events] [--watch-id WatchID] [-Y]
or
mmwatch Device list --watch-id WatchID --config [-Y]
or
mmwatch all list [--events] [-Y]
or
mmwatch {Device | all} cleanup --auto-disabled
or
mmwatch all upgrade
or
mmwatch all status [-Y]
or
mmwatch Device status [--watch-id WatchID [-v | -Y]]
or
mmwatch Device enable {-F ConfigFilePath |
[--fileset FSetName] [--watch-id WatchID]
[--events {Event[,Event...] | ALL}]
--event-handler handlertype
--sink-brokers BrokerIP:Port[,BrokerIP:Port...]
--sink-topic Topic
[--sink-auth-config Path]
[--secondary-sink [--sec-sink-fileset FSetName]]
[--degraded]}
or
mmwatch Device disable --watch-id WatchID
or
mmwatch Device suspend --watch-id WatchID
or
mmwatch Device resume --watch-id WatchID
Availability
Available with IBM Spectrum Scale Advanced Edition, IBM Spectrum Scale Data Management Edition, IBM Spectrum Scale Developer Edition, or IBM Spectrum Scale Erasure Code Edition.
Description
The mmwatch command is used to enable, disable, list, and generally administer persistent and fault-tolerant clustered watches. The purpose of clustered watch folder is to monitor file system event activity and produce events to an external event handler, which is referenced as a sink. The mmwatch command is also used to list active watch folder API watches. Errors are logged to /var/adm/ras/mmwatch.log, /var/adm/ras/mmwfclient.log, and /var/log/messages.
Parameters
- Device
- all
- Lists the active clustered watches for all file systems. Note: You must specify the Device or use the all keyword for mmwatch operations.
- list [--events] [-Y]
- Displays the details of active clustered watches for the specified device. The optional
--events option lists the monitored clustered watch
folder events for each specific watch for the specified
device. The -Y parameter provides output in machine-readable (colon-delimited)
format. Note: Fields that have a colon (:) are encoded to prevent confusion. For the set of characters that might be encoded, see the command documentation of mmclidecode. Use the mmclidecode command to decode the field.
- cleanup --auto-disabled
- If more than 50% of the conduits for a clustered watch cannot send messages to the sink Kafka, the watch will be auto-disabled. Although the watch is essentially stopped, the information in the message queue services monitor database remains for later debugging. You can use the cleanup option to clear out the auto-disabled clustered watch folder information.
- upgrade
- Converts all active clustered watches to use the IBM Spectrum Scale system health infrastructure. During the conversion process, conduits are disabled and then re-enabled. In the period of time between disablement and re-enablement of the consumer conduits, some events might be missed. In addition, this command also disables any clustered watches that are currently in the suspended state.
- status
- Provides the status of the consumer conduit processes that are running on the nodes within the
kafkaClWatchConsumerServers node class, and is subject to the following:
- If all is used in place of a device name, the status is provided for all clustered watches that are enabled within the cluster. If no clustered watches are found, a statement stating this is returned but this scenario is not considered an error.
- If a Device is specified without a watch ID, then the status is provided for all clustered watches that are enabled within the cluster associated with that device. If no clustered watches exist for the device, a statement stating this is returned and this is considered an error.
- If Device and --watch-id
WatchID are both specified, then the status is provided for
the single clustered watch that is associated with the given device and watch ID. If the clustered
watch cannot be found, a statement stating this is returned and this is considered an
error.Note: The -v flag can only be used when a watch ID is given. It provides up to the last 10 entries for the given watch ID for each component.
- enable
- Enables a clustered watch for the given device. The scope of the watch is the whole file system unless the --fileset option is provided. If --fileset is provided, then the watch type will be fileset watch for dependent filesets and inode space watch for independent filesets. Enablement entails setting up and validating the configuration, creating a Kafka topic on the local Kafka cluster, starting the watch consumer processes, and applying the respective policy partition rules based on the watched events.
- -F ConfigFilePath
- Specifies the path to the configuration file.The configuration file is populated with key:value pairs, which include all of the required parameters for enable and any optional fields desired in the following format:
- WATCH_ID:<Watch ID>
- FILESET:<fileset name>
- EVENTS:<Event1>,<Event2>
- EVENT_HANDLER:kafkasink
- SINK_BROKERS:<sink broker:port>
- SINK_TOPIC:<sink topic>
- SINK_AUTH_CONFIG:</path/to/auth/config>
- DEGRADED:<true|false>
Note: If this option is given, then the other command line options that are used to enable a watch are invalid and produce a syntax error. Using the --fileset, --watch-type, --events, --event-handler, --sink-brokers, --sink-topic, or --sink-auth-config options in combination with the -F option produces a syntax error. - --fileset FSetName
- Specifies the name of the fileset within the given file system to watch. If the fileset is dependent, then the watch type is fileset. No nested filesets are watched within a dependent fileset. If the fileset is independent, then the watch type is inodespace. Only nested dependent filesets are included in the watch of an independent fileset. The --fileset option is optional.
- --watch-id WatchID
- The WatchID is the unique identifier for a clustered watch. The
WatchID is optional for the enable operation; if it is not
specified, a calculated name is given with the following value:
The WatchID of a current clustered watch is required for the disable operation.CLW<#seconds since epoch>
Note: The WatchID can only be alphanumeric. The WatchID cannot contain the keywords all, list, enable, disable, or suspend. - --config
- Displays the configuration details for the specified active watch WatchID.
- --events Event[,Event...] | ALL}
- Defines a comma-separated string of events to monitor. The --events option
is optional. The following events can be watched:
- IN_ACCESS
- IN_ATTRIB
- IN_CLOSE_NOWRITE
- IN_CLOSE_WRITE
- IN_CREATE
- IN_DELETE
- IN_DELETE_SELF
- IN_MODIFY
- IN_MOVED_FROM
- IN_MOVED_TO
- IN_MOVE_SELF
- IN_OPEN
- --event-handler handlertype
- The only type of event handler that is currently supported is kafkasink. The --event-handler option is required.
- --sink-brokers BrokerIP:Port[,BrokerIP:Port...]
- Includes a comma-separated list of broker:port pairs for the sink Kafka cluster (the external Kafka cluster where the events are sent). The --sink-brokers option is required.
- --sink-topic Topic
- The topic that producers write to in the sink Kafka cluster. The --sink-topic option is required.
- --sink-auth-config Path
- The full path to the file that contains the authentication details to the sink Kafka cluster. The --sink-auth-config option is optional. For an example of how to use this flag, see Interaction between clustered watch folder and the external Kafka sink.
- --secondary-sink [ --sec-sink-fileset FSetName ]
- The --secondary-sink option means that when the suspend operation is run
for an active clustered watch, the events received while the clustered watch is suspended are
written to files within a fileset. When the corresponding clustered watch is resumed (the resume
operation is run), events from the file system activity as well as those that were written
previously to the secondary sink fileset will be sent to the external Kafka sink.If a secondary sink fileset name is not provided, the default name will be .secondary_sink. It will be created as an independent fileset that is mounted under the corresponding clustered watch's associated device mount point. If a fileset name is provided and the fileset that the customer defines does not exist, the fileset will be created as an independent fileset on the file system that is associated with the clustered watch.Note: The secondary sink fileset must always be located within the file system that is associated with the clustered watch. In addition, the secondary sink fileset cannot be named .msgq.
- --degraded
- Uses half of the standard number of partitions (the number of brokers instead of 2X the number of brokers) to save local disk space. The --degraded option is optional.
- disable
- Disables a clustered watch for the given device. Disablement stops the watch consumer processes, removes the local Kafka topic, and deletes the policy partition rules for the watch.
- suspend
- Suspends an active clustered watch, which means that the consumer conduits are stopped and will
not send events to the external Kafka queue. Events will accumulate within the message queue.
Note: If the clustered watch is suspended for a lengthy period of time or if a lot of file system activity occurs while the clustered watch is suspended, events might be missed due to the limited amount of local disk space on the broker nodes of the message queue.
- resume
- Resumes a previously suspended clustered watch. The consumer conduits are restarted. They will read events that built up while the consumer conduits were suspended and send them to the external Kafka queue.
Exit status
- 0
- Successful completion.
- nonzero
- A failure occurs.
Security
You must have root authority to run the mmwatch command.
Examples
- To list all current clustered watches that are active, issue this
command:
mmwatch demo list
The system displays information similar to the following output:Filesystem demo has 2 watcher(s): Watcher PID Type Start Time Path ----------------------------------------------------------------------------------------------------- node1.ibm.com 7047 directory Mon Aug 20 20:39:20 2018 /demo/somedir node2.ibm.com 6225 directory Tue Aug 21 10:45:05 2018 /demo
- To list all current clustered watches with associated events, issue this
command:
mmwatch demo list --events
The system displays information similar to the following output:Filesystem demo has 2 watcher(s): Watcher PID Monitored Events --------------------------------------------------------------------------------------- node3.ibm.com 7047 IN_CLOSE_WRITE,IN_DELETE_SELF node4.ibm.com 6225 IN_ACCESS,IN_ATTRIB,IN_CLOSE_NOWRITE, IN_CLOSE_WRITE,IN_CREATE,IN_DELETE, IN_DELETE_SELF,IN_MODIFY,IN_MOVED_FROM, IN_MOVE_SELF
- To list the current configuration for a clustered watch, issue this
command:
mmwatch Device list --watch-id WatchID --config
The system displays information similar to the following output:node1:~ # mmwatch fs2 list --watch-id CLW1552340432 --config DEVICE:fs2 WATCH_ID:CLW1552340432 EVENTS:all EVENT_HANDLER:kafkasink SINK_BROKERS:ibmserver30:9092 SINK_TOPIC:ibm30topic DEGRADED:TRUE SINK_AUTH_TYPE:none node1:~ #
- You can see if clustered watches are auto-disabled in the
status:
# mmwatch all status Device Watch ID Watch State fs2 CLW1556232964 Auto Disabled Node Name Status machine1.company.com DOWN machine2.company.com DOWN machine3.company.com DOWN machine4.company.com DOWN fs1 CLW1556232741 Auto Disabled Node Name Status machine1.company.com DOWN machine2.company.com DOWN machine3.company.com DOWN machine4.company.com DOWN
To remove the corresponding database entries, use the cleanup option:# mmwatch all cleanup --auto-disabled [I] Successfully cleaned up 2 clustered watch(es).
Then, if you view the status again, you will not see the auto-disabled clustered watches:# mmwatch all status [I] No clustered watch topics found. Status cannot be obtained.
- To check the status, issue this
command:
mmwatch all status
The system displays information similar to the following output:node1:~ # mmwatch fs2 status --watch-id CLW1552340432 Device Watch ID fs2 CLW1552340432 Node Name Status node1.gpfs.ss HEALTHY node3.gpfs.ss HEALTHY node4.gpfs.ss HEALTHY node1:~ #
- To enable a clustered watch using an input configuration file, issue
this command:
mmwatch Device enable -F ConfigFilePath
The system displays information similar to the following output:node1:~ # mmwatch fs2 enable -F watch.config [I] Beginning enablement of Clustered Watch with newly created watch ID: CLW1552340432 [I] Verifying MsgQueue nodes meet minimum local space requirements for Clustered Watch to be enabled for watch: CLW1552340432. Depending on cluster size, this may take some time. [I] Successfully verified all configured MsgQueue nodes meet minimum local space requirements for Clustered Watch to be enabled for watch: CLW1552340432 [I] Verified the watch type is FSYS for filesystem fs2 [I] Successfully created Clustered Watch topic on the MsgQueue for watch: CLW1552340432 [I] Successfully added Clustered Watch configuration file into CCR for watch: CLW1552340432 [I] Successfully enabled Clustered Watch consumers for watch: CLW1552340432 [I] Successfully added Clustered Watch policy rules for watch: CLW1552340432 [I] Successfully enabled Clustered Watch: CLW1552340432 node1:~ #
- Enabling an independent fileset displays information similar to the
following
output:
node1.gpfs.ss:~ # mmwatch fs2 enable --fileset fileset10 --event-handler kafkasink --sink-brokers "sink1.server.net:9092,sink2.server.net:9092" --sink-topic sink_topic --sink-auth-config "/root/sink.auth" --degraded [I] Beginning enablement of Clustered Watch with newly created watch ID: CLW1549311488 [I] Verifying MsgQueue nodes meet minimum local space requirements for Clustered Watch to be enabled for watch: CLW1549311488. Depending on cluster size, this may take some time. [W] Broker node node1.gpfs.ss meets the minimum amount of local disk space required 10485760KB to run in degraded mode but not the recommended amount of local disk space 20971520KB [I] Successfully verified all configured MsgQueue nodes meet minimum local space requirements for Clustered Watch to be enabled for watch: CLW1549311488 [I] Successfully created Clustered Watch topic on the MsgQueue for watch: CLW1549311488 [I] Successfully added Clustered Watch configuration file into CCR for watch: CLW1549311488 [I] Successfully enabled Clustered Watch consumers for watch: CLW1549311488 [I] Successfully added Clustered Watch policy rules for watch: CLW1549311488 [I] Successfully enabled Clustered Watch: CLW1549311488 node1.gpfs.ss:~ #
- To disable a clustered watch, issue this
command:
mmwatch Device disable --watch-id WatchID
The system displays information similar to the following output:node1:~ # mmwatch fs2 disable --watch-id CLW1552340432 [I] Successfully removed Clustered Watch policy rules for watch: CLW1552340432 [I] Successfully disabled Clustered Watch consumers for watch: CLW1552340432 [I] Successfully deleted Clustered Watch topic from the MsgQueue for watch: CLW1552340432 [I] Successfully removed Clustered Watch configuration file from the CCR for watch: CLW1552340432 [I] Successfully disabled Clustered Watch: CLW1552340432 node1:~ #
- Before suspending a clustered watch, you can check the status
with the mmwatch all status
command:
To suspend a clustered watch, issue this command:# mmwatch all status Device Watch ID Watch State fs1 CLW1556906797 Active Node Name Status computer1.company.com HEALTHY computer2.company.com HEALTHY computer3.company.com HEALTHY computer4.company.com HEALTHY
mmwatch Device suspend --watch-id WatchID
The system displays information similar to the following output:# mmwatch fs1 suspend --watch-id CLW1556906797 [I] The clustered watch associated with device: fs1 and watch ID: CLW1556906797 is not currently suspended. Will attempt to suspend. [I] Successfully suspended Clustered Watch associated with device: fs1 and watch ID: CLW1556906797 # mmwatch all status Device Watch ID Watch State fs1 CLW1556906797 Suspended Node Name Status computer1.company.com SUSPENDED computer2.company.com SUSPENDED computer3.company.com SUSPENDED computer4.company.com SUSPENDED
- To resume a suspended clustered watch, issue this
command:
mmwatch Device resume --watch-id WatchID
The system displays information similar to the following output:# mmwatch fs1 resume --watch-id CLW1556906797 [I] The clustered watch associated with device: fs1 and watch ID: CLW1556906797 is currently suspended. Will attempt to resume. [I] Successfully resumed Clustered Watch associated with device: fs1 and watch ID: CLW1556906797 # mmwatch all status Device Watch ID Watch State fs1 CLW1556906797 Active Node Name Status computer1.company.com HEALTHY computer2.company.com HEALTHY computer3.company.com HEALTHY computer4.company.com HEALTHY
See also
Location
/usr/lpp/mmfs/bin