Package edu.internet2.middleware.grouper.app.loader

Class GrouperLoader

java.lang.Object
edu.internet2.middleware.grouper.app.loader.GrouperLoader
public class GrouperLoader extends Object
main class to start the grouper loader

  • Field Details

    • GROUPER_LOADER_TYPE

      public static final String GROUPER_LOADER_TYPE
      group attribute name of type of the loader, must match one of the enums in GrouperLoaderType. If there is a query, and it has "group_name" before "from", then defaults to SQL_GROUP_LIST else defaults to SQL_SIMPLE
      See Also:

    • GROUPER_LOADER_DISPLAY_NAME_SYNC_TYPE

      public static final String GROUPER_LOADER_DISPLAY_NAME_SYNC_TYPE
      grouper loader display name sync type. BASE_FOLDER_NAME or LEVELS
      See Also:

    • GROUPER_LOADER_DISPLAY_NAME_SYNC_BASE_FOLDER_NAME

      public static final String GROUPER_LOADER_DISPLAY_NAME_SYNC_BASE_FOLDER_NAME
      folder name after which display names need to be synced between source and grouper
      See Also:

    • GROUPER_LOADER_DISPLAY_NAME_SYNC_LEVELS

      public static final String GROUPER_LOADER_DISPLAY_NAME_SYNC_LEVELS
      levels starting from the group after which display names need to be synced between source and grouper
      See Also:

    • GROUPER_LOADER_AND_GROUPS

      public static final String GROUPER_LOADER_AND_GROUPS
      groups to and with to restrict members (e.g. "and" with activeEmployees)
      See Also:

    • GROUPER_LOADER_GROUPS_LIKE

      public static final String GROUPER_LOADER_GROUPS_LIKE
      If you want the group (if not used from anywhere) or members deleted when no longer in loader sql results, list the sql like name, e.g. stem1:stem2:%:%org
      See Also:

    • GROUPER_LOADER_GROUP_QUERY

      public static final String GROUPER_LOADER_GROUP_QUERY
      optional group information for a group list query: e.g. to specify the display name of the group/stem when it is created
      See Also:

    • GROUPER_LOADER_GROUP_TYPES

      public static final String GROUPER_LOADER_GROUP_TYPES
      types to add to loaded groups
      See Also:

    • GROUPER_LOADER_SCHEDULE_TYPE

      public static final String GROUPER_LOADER_SCHEDULE_TYPE
      group attribute name of type of schedule, must match one of the enums in GrouperLoaderScheduleType. defaults to START_TO_START_INTERVAL if grouperLoaderQuartzCron is blank, else defaults to CRON
      See Also:

    • GROUPER_LOADER_QUERY

      public static final String GROUPER_LOADER_QUERY
      group attribute name of query, must have the required columns for the grouperLoaderType
      See Also:

    • GROUPER_LOADER_QUARTZ_CRON

      public static final String GROUPER_LOADER_QUARTZ_CRON
      group attribute name of quartz cron-like string to describe when the job should run
      See Also:

    • GROUPER_LOADER_INTERVAL_SECONDS

      public static final String GROUPER_LOADER_INTERVAL_SECONDS
      group attribute name of the interval in seconds for a schedule type like START_TO_START_INTERVAL. defaults to 86400 (1 day)
      See Also:

    • GROUPER_LOADER_PRIORITY

      public static final String GROUPER_LOADER_PRIORITY
      group attribute name of priority of job, optional, if not there, will be 5. More is better. if the threadpool is full, then this priority will help the schedule pick which job should go next
      See Also:

    • GROUPER_LOADER_FAILSAFE_USE

      public static final String GROUPER_LOADER_FAILSAFE_USE
      T or F if using failsafe. If blank use the global defaults
      See Also:

    • GROUPER_LOADER_MAX_GROUP_PERCENT_REMOVE

      public static final String GROUPER_LOADER_MAX_GROUP_PERCENT_REMOVE
      integer from 0 to 100 which specifies the maximum percent of a group which can be removed in a loader run. If not specified will use the global default grouper-loader.properties config setting: loader.failsafe.maxPercentRemove = 30
      See Also:

    • GROUPER_LOADER_MAX_OVERALL_PERCENT_GROUPS_REMOVE

      public static final String GROUPER_LOADER_MAX_OVERALL_PERCENT_GROUPS_REMOVE
      If the group list meets the criteria above and the percentage of memberships that are managed by the loader (i.e. match the groupLikeString) that currently have members in Grouper but wouldn't after the job runs is greater than this percentage, then don't remove members, log it as an error and fail the job. An admin would need to approve the failsafe or change this param in the config, and run the job manually, then change this config back. {valueType: "integer", required: true} loader.failsafe.groupList.managedGroups.maxPercentMembershipsRemove = 30
      See Also:

    • GROUPER_LOADER_MAX_OVERALL_PERCENT_MEMBERSHIPS_REMOVE

      public static final String GROUPER_LOADER_MAX_OVERALL_PERCENT_MEMBERSHIPS_REMOVE
      integer from 0 to 100 which specifies the maximum percent of all loaded groups in the job which can be removed in a loader run. If not specified will use the global default grouper-loader.properties config setting: loader.failsafe.groupList.managedGroups.maxPercentGroupsRemove = 30
      See Also:

    • GROUPER_LOADER_MIN_GROUP_SIZE

      public static final String GROUPER_LOADER_MIN_GROUP_SIZE
      minimum number of members for the group to be tracked by failsafe defaults to grouper-loader.base.properties: loader.failsafe.minGroupSize
      See Also:

    • GROUPER_LOADER_MIN_MANAGED_GROUPS

      public static final String GROUPER_LOADER_MIN_MANAGED_GROUPS
      The minimum number of managed groups for this loader job for the list of groups job to be applicable
      See Also:

    • GROUPER_LOADER_MIN_GROUP_NUMBER_OF_MEMBERS

      public static final String GROUPER_LOADER_MIN_GROUP_NUMBER_OF_MEMBERS
      The minimum group number of members for this group, a failsafe alert will trigger if the group is smaller than this amount
      See Also:

    • GROUPER_LOADER_MIN_OVERALL_NUMBER_OF_MEMBERS

      public static final String GROUPER_LOADER_MIN_OVERALL_NUMBER_OF_MEMBERS
      The minimum overall number of members for this job across all managed groups, a failsafe alert will trigger if the job's overall membership count is smaller than this amount
      See Also:

    • GROUPER_LOADER_FAILSAFE_SEND_EMAIL

      public static final String GROUPER_LOADER_FAILSAFE_SEND_EMAIL
      If an email should be sent out when a failsafe alert happens. The email will be sent to the list or group configured in grouper-loader.properties: loader.failsafe.sendEmailToAddresses, or loader.failsafe.sendEmailToGroup
      See Also:

    • GROUPER_LOADER_DB_NAME

      public static final String GROUPER_LOADER_DB_NAME
      group attribute name of the db connection where this query comes from. if the name is "grouper", then it will be the group db name. defaults to "grouper" for sql type loaders
      See Also:

    • ATTRIBUTE_LOADER_TYPE

      public static final String ATTRIBUTE_LOADER_TYPE
      Type of loader, e.g. ATTR_SQL_SIMPLE
      See Also:

    • ATTRIBUTE_LOADER_DB_NAME

      public static final String ATTRIBUTE_LOADER_DB_NAME
      DB name in grouper-loader.properties or default grouper db if blank
      See Also:

    • ATTRIBUTE_LOADER_SCHEDULE_TYPE

      public static final String ATTRIBUTE_LOADER_SCHEDULE_TYPE
      Type of schedule. Defaults to CRON if a cron schedule is entered, or START_TO_START_INTERVAL if an interval is entered
      See Also:

    • ATTRIBUTE_LOADER_QUARTZ_CRON

      public static final String ATTRIBUTE_LOADER_QUARTZ_CRON
      If a CRON schedule type, this is the cron setting string from the quartz product to run a job daily, hourly, weekly, etc. e.g. daily at 7am: 0 0 7 * * ?
      See Also:

    • ATTRIBUTE_LOADER_INTERVAL_SECONDS

      public static final String ATTRIBUTE_LOADER_INTERVAL_SECONDS
      If a START_TO_START_INTERVAL schedule type, this is the number of seconds between runs
      See Also:

    • ATTRIBUTE_LOADER_PRIORITY

      public static final String ATTRIBUTE_LOADER_PRIORITY
      Quartz has a fixed threadpool (max configured in the grouper-loader.properties), and when the max is reached, then jobs are prioritized by this integer. The higher the better, and the default if not set is 5.
      See Also:

    • ATTRIBUTE_LOADER_ATTRS_LIKE

      public static final String ATTRIBUTE_LOADER_ATTRS_LIKE
      If empty, then orphans will be left alone (for attributeDefName and attributeDefNameSets). If %, then all orphans deleted. If a SQL like string, then only ones in that like string not in loader will be deleted
      See Also:

    • ATTRIBUTE_LOADER_ATTR_QUERY

      public static final String ATTRIBUTE_LOADER_ATTR_QUERY
      SQL query with at least some of the following columns: attr_name, attr_display_name, attr_description
      See Also:

    • ATTRIBUTE_LOADER_ATTR_SET_QUERY

      public static final String ATTRIBUTE_LOADER_ATTR_SET_QUERY
      SQL query with at least the following columns: if_has_attr_name, then_has_attr_name
      See Also:

    • ATTRIBUTE_LOADER_ACTION_QUERY

      public static final String ATTRIBUTE_LOADER_ACTION_QUERY
      SQL query with at least the following column: action_name
      See Also:

    • ATTRIBUTE_LOADER_ACTION_SET_QUERY

      public static final String ATTRIBUTE_LOADER_ACTION_SET_QUERY
      SQL query with at least the following columns: if_has_action_name, then_has_action_name
      See Also:

    • ATTRIBUTE_GROUPER_LOADER_METADATA_LOADED

      public static final String ATTRIBUTE_GROUPER_LOADER_METADATA_LOADED
      True means the group was loaded from loader
      See Also:

    • ATTRIBUTE_GROUPER_LOADER_METADATA_LAODED

      @Deprecated public static final String ATTRIBUTE_GROUPER_LOADER_METADATA_LAODED
      Deprecated.
      True means the group was loaded from loader TODO remove in 2.4
      See Also:

    • ATTRIBUTE_GROUPER_LOADER_METADATA_GROUP_ID

      public static final String ATTRIBUTE_GROUPER_LOADER_METADATA_GROUP_ID
      Group id which is being populated from the loader
      See Also:

    • ATTRIBUTE_GROUPER_LOADER_METADATA_LAST_FULL_MILLIS

      public static final String ATTRIBUTE_GROUPER_LOADER_METADATA_LAST_FULL_MILLIS
      Millis since 1970 that this group was fully processed
      See Also:

    • ATTRIBUTE_GROUPER_LOADER_METADATA_LAST_INCREMENTAL_MILLIS

      public static final String ATTRIBUTE_GROUPER_LOADER_METADATA_LAST_INCREMENTAL_MILLIS
      Millis since 1970 that this group was incrementally processed
      See Also:

    • ATTRIBUTE_GROUPER_LOADER_METADATA_LAST_SUMMARY

      public static final String ATTRIBUTE_GROUPER_LOADER_METADATA_LAST_SUMMARY
      summary like count of additions, updates and removals
      See Also:

    • LOADER_METADATA_VALUE_DEF

      public static final String LOADER_METADATA_VALUE_DEF
      name of the loader metadata definition
      See Also:

  • Constructor Details

    • GrouperLoader

      public GrouperLoader()

  • Method Details

    • shutdownIfStarted

      public static void shutdownIfStarted()
      call this when exiting grouper if not the daemon which should stay running

    • main

      public static void main(String[] args)
      Parameters:
      args -

    • scheduleJobs

      public static int scheduleJobs()
      Returns:
      the number of changes made

    • printAllSupportDdlUtilsPlatforms

      public static void printAllSupportDdlUtilsPlatforms()
      print out all ddlutils platforms

    • schedulerFactory

      public static org.quartz.SchedulerFactory schedulerFactory()
      lazy load (and start the scheduler) the scheduler factory
      Returns:
      the scheduler factory

    • scheduleChangeLogJobs

      public static int scheduleChangeLogJobs()
      schedule change log jobs

    • scheduleChangeLogTempToChangeLogJob

      public static boolean scheduleChangeLogTempToChangeLogJob()
      schedule maintenance job for moving records from change log to change log temp

    • scheduleChangeLogConsumers

      public static int scheduleChangeLogConsumers()
      schedule change log consumer jobs

    • scheduleMessagingListeners

      public static int scheduleMessagingListeners()
      schedule messaging listener jobs

    • scheduleOtherJobs

      public static int scheduleOtherJobs()
      schedule other jobs

    • scheduleEsbListenerJobs

      public static int scheduleEsbListenerJobs()

    • runJobOnceForGroup

      public static String runJobOnceForGroup(GrouperSession grouperSession, Group group)
      Parameters:
      group -
      grouperSession -
      Returns:
      status

    • runJobOnceForGroup

      public static String runJobOnceForGroup(GrouperSession grouperSession, Group group, boolean runOnDaemon)
      Parameters:
      group -
      grouperSession -
      runOnDaemon -
      Returns:
      status

    • runOnceByJobName

      public static String runOnceByJobName(GrouperSession grouperSession, String jobName)
      Parameters:
      grouperSession -
      jobName -
      Returns:
      status

    • runOnceByJobName

      public static String runOnceByJobName(GrouperSession grouperSession, String jobName, boolean runOnDaemon)
      Parameters:
      grouperSession -
      jobName -
      runOnDaemon -
      Returns:
      status

    • _internal_runJobOnceForAttributeDef

      public static Hib3GrouperLoaderLog _internal_runJobOnceForAttributeDef(GrouperSession grouperSession, AttributeDef attributeDef)
      Parameters:
      attributeDef -
      grouperSession -
      Returns:
      status

    • runJobOnceForAttributeDef

      public static String runJobOnceForAttributeDef(GrouperSession grouperSession, AttributeDef attributeDef)
      Parameters:
      attributeDef -
      grouperSession -
      Returns:
      status

    • schedulePspFullSyncJob

      public static boolean schedulePspFullSyncJob()
      schedule psp full sync job

    • internal_retrieveThreadLocalGrouperLoaderDryRun

      public static GrouperLoader.GrouperLoaderDryRunBean internal_retrieveThreadLocalGrouperLoaderDryRun()
      Returns:
      the threadLocalGrouperLoaderDryRun

    • internal_assignThreadLocalGrouperLoaderDryRun

      public static void internal_assignThreadLocalGrouperLoaderDryRun(GrouperLoader.GrouperLoaderDryRunBean theThreadLocalGrouperLoaderDryRun)
      Parameters:
      theThreadLocalGrouperLoaderDryRun - the threadLocalGrouperLoaderDryRun to set

    • dryRunJobOnceForGroup

      public static String dryRunJobOnceForGroup(GrouperSession grouperSession, Group group, String fileName)
      Parameters:
      group -
      grouperSession -
      fileName - is the file where output should go
      Returns:
      status

    • isDryRun

      public static boolean isDryRun()
      Returns:
      true if dry run

    • dryRunWriteLine

      public static void dryRunWriteLine(String line)
      Parameters:
      line -

    • renameJobAndTriggerSubstring

      public static void renameJobAndTriggerSubstring(String oldJobName, String oldSubstring, String newSubstring) throws org.quartz.SchedulerException
      Rename a job and it's associated triggers where the name replaces the oldSubstring with the newSubstring
      Parameters:
      oldJobName -
      oldSubstring -
      newSubstring -
      Throws:
      org.quartz.SchedulerException

    • scheduleJobIfNeeded

      public static boolean scheduleJobIfNeeded(org.quartz.JobDetail jobDetail, org.quartz.Trigger trigger) throws org.quartz.SchedulerException
      Schedule job if new or something has changed
      Parameters:
      jobDetail -
      trigger -
      Returns:
      true if needed an update
      Throws:
      org.quartz.SchedulerException

    • isJobRunningAsRunNow

      public static boolean isJobRunningAsRunNow(String jobName)
      see if the job is running as a run now:
      Parameters:
      jobName -
      Returns:
      true if run now, e.g. OTHER_JOB_sleep, OTHER_JOB_schedulerCheckDaemon

    • isJobRunning

      public static boolean isJobRunning(String jobName, boolean checkingAtStartOfJobBeforeInsertingGrouperLoaderLog)
      Parameters:
      jobName -
      true - if checking within this job (in which case checking to see if it is running elsewhere?)
      Returns:
      true if the job appears to currently be running

    • internal_getJobStartTimeIfRunning

      public static Date internal_getJobStartTimeIfRunning(String jobName)
      Parameters:
      jobName -
      Returns:
      date or null if not running

    • isJobEnabled

      public static boolean isJobEnabled(String jobName)
      Parameters:
      jobName -
      Returns:
      true is job is enabled

    • isRunningJobOnceLocally

      public static boolean isRunningJobOnceLocally()
      If the threadlocal indicates that the current job should be running once
      Returns:
      boolean

    • waitForChangeLogTempToEntityProcess

      public static void waitForChangeLogTempToEntityProcess(int exceptionIfNotProcessedAfterSeconds)
      Retrieves the most recent ChangeLogTempToEntity entry and waits for it to be processed.
      Parameters:
      exceptionIfNotProcessedAfterSeconds - throw an exception if still not done after this amount of seconds pass. -1 to continue indefinitely

    • waitForChangeLogJobProcess

      public static void waitForChangeLogJobProcess(String jobName, boolean waitForChangeLogTempToEntityProcess, int exceptionIfNotProcessedAfterSeconds)
      Retrieves the most recent change log entry and waits for it to be processed by this change log consumer
      Parameters:
      jobName - name of the job, e.g. CHANGE_LOG_consumer_compositeMemberships
      waitForChangeLogTempToEntityProcess - wait for change log temp to clear first
      exceptionIfNotProcessedAfterSeconds - throw an exception if still not done after this amount of seconds pass. -1 to continue indefinitely