How admin scheduler for JVM (app server, agent server, integration server or health monitor) works?

Answer

Every JVM (application server, integration server, agent server or health monitor) as their own admin scheduler which runs bases on the yantra.statistics.persist.interval property which is set to 10 minutes by default in yfs.properties file.
Below are the excerpts from the same.

# Property to determine statistics logging time interval
# format of the property is Xm or Xh where X is an integer between 1 and 60
# and M/m for minutes or H/h for hours.
# If any unrecognized value or unit is specified, it will default to 10m (minutes).
# If a value of 61m or greater is specified, it will be reduced to 60m.
# If a value of 25h or greater is specified, it will be reduced to 24h.
# If the units are minutes (M/m), then the value is rounded up or down to the nearest equal divisor of 60 minutes. Possible values are 1,2,3,4,5,6,10,12,15,20 or 30.
# If the units are hours (H/h), then the value is rounded up or down to the nearest equal divisor of 24 hours. Possible values are 1,2,3,4,6,8 or 12.
yantra.statistics.persist.interval=10m

So even if this property is setup to a wrong value system will default it to 10 minutes or 60 minutes or 24 hours as per the above rule.

Now below is the process that admin scheduler follows and how system calculates the time interval to find out stale entries:
1) Say yantra.statistics.persist.interval=10
2) So to keep other offsets in mind like time precision (may be the precision is seconds or nanoseconds) or due to some other things the admin scheduler will update the LAST_HEARTBEAT column of the YFS_HEARTBEAT table in every (yantra.statistics.persist.interval)/2 time that is in 5 minutes in this the current example
3) Now system calculate (current system time - last heartbeat time) and if this is greater 1.5 * yantra.statistics.persist.interval; then the entry is qualified as stale either by health monitor or by system during cache broadcasting.

User can change this property according to their ease but this should not be very high value as it might result with the performance (eg system might end up doing broadcasting messages to actually down servers). So unless until required Sterling recommend to keep this property to its current value. If required to be overridden user should use customer_overrides.properties file to override them. Please see version 8.0, section 12.2.2 'Using the Customer_Overrides.properties File' of Installation_Guide.pdf for more information on how to use this file.

This admin scheduler should always be keep on updating the entries and very unlikely to go wrong though not foolproof.

Note: Please make sure application server and agent/integration servers are properly shut down before database is shut down. Heath monitor will be the first one to be shut down and last one to be brought up. If the DB is shut down somehow before application server then admin scheduler will not be able to update YFS_HEARTBEAT table and in this case the live server records in this table will become stale. These records then will be picked by Health Monitor and their status will be changed to 02.