org.apache.hadoop.mapreduce.util

## Class MRJobConfUtil

- java.lang.Object
- - org.apache.hadoop.mapreduce.util.MRJobConfUtil

-     public final class MRJobConfUtil
      extends Object

  A class that contains utility methods for MR Job configuration.

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static double` | `PROGRESS_MIN_DELTA_FACTOR` Configurations to control the frequency of logging of task Attempt.
    `static String` | `REDACTION_REPLACEMENT_VAL`
    `static float` | `TASK_REPORT_INTERVAL_TO_TIMEOUT_RATIO`

  - ### Method Summary

    All Methods Static Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `static double` | `convertTaskProgressToFactor(float progress)` Coverts a progress between 0.0 to 1.0 to double format used to log the task attempt.
    `static double` | `getTaskProgressMinDeltaThreshold()` Retrieves the min delta progress required to log the task attempt current progress.
    `static long` | `getTaskProgressReportInterval(org.apache.hadoop.conf.Configuration conf)` Get the progress heartbeat interval configuration for mapreduce tasks.
    `static long` | `getTaskProgressWaitDeltaTimeThreshold()` Retrieves the min time required to log the task attempt current progress.
    `static org.apache.hadoop.conf.Configuration` | `initEncryptedIntermediateConfigsForTesting(org.apache.hadoop.conf.Configuration conf)`
    `static void` | `redact(org.apache.hadoop.conf.Configuration conf)` Redact job configuration properties.
    `static org.apache.hadoop.conf.Configuration` | `setLocalDirectoriesConfigForTesting(org.apache.hadoop.conf.Configuration conf, File testRootDir)` Set local directories so that the generated folders is subdirectory of the test directories.
    `static void` | `setTaskLogProgressDeltaThresholds(org.apache.hadoop.conf.Configuration conf)` load the values defined from a configuration file including the delta progress and the maximum time between each log message.

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### REDACTION_REPLACEMENT_VAL

          public static final String REDACTION_REPLACEMENT_VAL

      See Also:
        Constant Field Values

    - #### TASK_REPORT_INTERVAL_TO_TIMEOUT_RATIO

          public static final float TASK_REPORT_INTERVAL_TO_TIMEOUT_RATIO

      See Also:
        Constant Field Values

    - #### PROGRESS_MIN_DELTA_FACTOR

          public static final double PROGRESS_MIN_DELTA_FACTOR

      Configurations to control the frequency of logging of task Attempt.
      See Also:
        Constant Field Values

  - ### Method Detail

    - #### redact

          public static void redact(org.apache.hadoop.conf.Configuration conf)

      Redact job configuration properties.
      Parameters:
        `conf` - the job configuration to redact

    - #### getTaskProgressReportInterval

          public static long getTaskProgressReportInterval(org.apache.hadoop.conf.Configuration conf)

      Get the progress heartbeat interval configuration for mapreduce tasks. By default, the value of progress heartbeat interval is a proportion of that of task timeout.
      Parameters:
        `conf` - the job configuration to read from
      Returns:
        the value of task progress report interval

    - #### setTaskLogProgressDeltaThresholds

          public static void setTaskLogProgressDeltaThresholds(org.apache.hadoop.conf.Configuration conf)

      load the values defined from a configuration file including the delta progress and the maximum time between each log message.
      Parameters:
        `conf` -

    - #### getTaskProgressMinDeltaThreshold

          public static double getTaskProgressMinDeltaThreshold()

      Retrieves the min delta progress required to log the task attempt current progress.
      Returns:
        the defined threshold in the conf. returns the default value if `setTaskLogProgressDeltaThresholds(org.apache.hadoop.conf.Configuration)` has not been called.

    - #### getTaskProgressWaitDeltaTimeThreshold

          public static long getTaskProgressWaitDeltaTimeThreshold()

      Retrieves the min time required to log the task attempt current progress.
      Returns:
        the defined threshold in the conf. returns the default value if `setTaskLogProgressDeltaThresholds(org.apache.hadoop.conf.Configuration)` has not been called.

    - #### convertTaskProgressToFactor

          public static double convertTaskProgressToFactor(float progress)

      Coverts a progress between 0.0 to 1.0 to double format used to log the task attempt.
      Parameters:
        `progress` - of the task which is a value between 0.0 and 1.0.
      Returns:
        the double value that is less than or equal to the argument multiplied by `PROGRESS_MIN_DELTA_FACTOR`.

    - #### initEncryptedIntermediateConfigsForTesting

          public static org.apache.hadoop.conf.Configuration initEncryptedIntermediateConfigsForTesting(org.apache.hadoop.conf.Configuration conf)

    - #### setLocalDirectoriesConfigForTesting

          public static org.apache.hadoop.conf.Configuration setLocalDirectoriesConfigForTesting(org.apache.hadoop.conf.Configuration conf,
                                                                                                 File testRootDir)

      Set local directories so that the generated folders is subdirectory of the test directories.
      Parameters:
        `conf` -
        `testRootDir` -
      Returns:

Copyright © 2008–2022 Apache Software Foundation. All rights reserved.