org.apache.hadoop.mapreduce.v2.app

## Class MRAppMaster

- java.lang.Object
- - org.apache.hadoop.service.AbstractService
  - - org.apache.hadoop.service.CompositeService
    - - org.apache.hadoop.mapreduce.v2.app.MRAppMaster

- All Implemented Interfaces:
    Closeable, AutoCloseable, org.apache.hadoop.service.Service

      public class MRAppMaster
      extends org.apache.hadoop.service.CompositeService

  The Map-Reduce Application Master. The state machine is encapsulated in the implementation of Job interface. All state changes happens via Job interface. Each event results in a Finite State Transition in Job. MR AppMaster is the composition of loosely coupled services. The services interact with each other via events. The components resembles the Actors model. The component acts on received event and send out the events to other components. This keeps it highly concurrent with no or minimal synchronization needs. The events are dispatched by a central Dispatch mechanism. All components register to the Dispatcher. The information is shared across different components using AppContext.

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Class and Description
    --- | ---
    `class` | `MRAppMaster.RunningAppContext`

    - ### Nested classes/interfaces inherited from class org.apache.hadoop.service.CompositeService

      `org.apache.hadoop.service.CompositeService.CompositeServiceShutdownHook`

    - ### Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

      `org.apache.hadoop.service.Service.STATE`

  - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `protected org.apache.hadoop.security.UserGroupInformation` | `currentUser`
    `static String` | `INTERMEDIATE_DATA_ENCRYPTION_ALGO`
    `protected boolean` | `isLastAMRetry`
    `protected org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager` | `jobTokenSecretManager`
    `protected MRAppMetrics` | `metrics`
    `static int` | `SHUTDOWN_HOOK_PRIORITY` Priority of the MRAppMaster shutdown hook.
    `protected AtomicBoolean` | `successfullyUnregistered`
    `protected TaskAttemptListener` | `taskAttemptListener`

    - ### Fields inherited from class org.apache.hadoop.service.CompositeService

      `STOP_ONLY_STARTED_SERVICES`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `MRAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId, org.apache.hadoop.yarn.api.records.ContainerId containerId, String nmHost, int nmPort, int nmHttpPort, org.apache.hadoop.yarn.util.Clock clock, long appSubmitTime)`
    `MRAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId, org.apache.hadoop.yarn.api.records.ContainerId containerId, String nmHost, int nmPort, int nmHttpPort, long appSubmitTime)`

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `cleanupStagingDir()` clean up staging directories for the job.
    `protected ClientService` | `createClientService(AppContext context)`
    `protected org.apache.hadoop.yarn.event.EventHandler<CommitterEvent>` | `createCommitterEventHandler(AppContext context, org.apache.hadoop.mapreduce.OutputCommitter committer)`
    `protected ContainerAllocator` | `createContainerAllocator(ClientService clientService, AppContext context)`
    `protected ContainerLauncher` | `createContainerLauncher(AppContext context)`
    `protected org.apache.hadoop.yarn.event.Dispatcher` | `createDispatcher()`
    `protected Job` | `createJob(org.apache.hadoop.conf.Configuration conf, JobStateInternal forcedState, String diagnostic)` Create and initialize (but don't start) a single job.
    `protected org.apache.hadoop.yarn.event.EventHandler<JobFinishEvent>` | `createJobFinishEventHandler()` create an event handler that handles the job finish event.
    `protected org.apache.hadoop.yarn.event.EventHandler<JobHistoryEvent>` | `createJobHistoryHandler(AppContext context)`
    `protected AMPreemptionPolicy` | `createPreemptionPolicy(org.apache.hadoop.conf.Configuration conf)`
    `protected Speculator` | `createSpeculator(org.apache.hadoop.conf.Configuration conf, AppContext context)`
    `protected org.apache.hadoop.service.AbstractService` | `createStagingDirCleaningService()`
    `protected TaskAttemptFinishingMonitor` | `createTaskAttemptFinishingMonitor(org.apache.hadoop.yarn.event.EventHandler eventHandler)`
    `protected TaskAttemptListener` | `createTaskAttemptListener(AppContext context, AMPreemptionPolicy preemptionPolicy)`
    `List<org.apache.hadoop.mapreduce.v2.api.records.AMInfo>` | `getAllAMInfos()`
    `org.apache.hadoop.yarn.api.records.ApplicationId` | `getAppID()`
    `org.apache.hadoop.yarn.api.records.ApplicationAttemptId` | `getAttemptID()`
    `ClientService` | `getClientService()`
    `org.apache.hadoop.mapreduce.OutputCommitter` | `getCommitter()`
    `Map<org.apache.hadoop.mapreduce.v2.api.records.TaskId,org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.TaskInfo>` | `getCompletedTaskFromPreviousRun()`
    `ContainerAllocator` | `getContainerAllocator()`
    `ContainerLauncher` | `getContainerLauncher()`
    `AppContext` | `getContext()`
    `protected org.apache.hadoop.security.Credentials` | `getCredentials()`
    `org.apache.hadoop.yarn.event.Dispatcher` | `getDispatcher()`
    `protected org.apache.hadoop.fs.FileSystem` | `getFileSystem(org.apache.hadoop.conf.Configuration conf)` Create the default file System for this job.
    `org.apache.hadoop.mapreduce.v2.api.records.JobId` | `getJobId()`
    `protected RMHeartbeatHandler` | `getRMHeartbeatHandler()`
    `int` | `getStartCount()`
    `TaskAttemptListener` | `getTaskAttemptListener()`
    `protected static void` | `initAndStartAppMaster(MRAppMaster appMaster, org.apache.hadoop.mapred.JobConf conf, String jobUserName)`
    `protected void` | `initJobCredentialsAndUGI(org.apache.hadoop.conf.Configuration conf)` Obtain the tokens needed by the job and put them in the UGI
    `Boolean` | `isLastAMRetry()`
    `boolean` | `isNewApiCommitter()`
    `protected boolean` | `keepJobFiles(org.apache.hadoop.mapred.JobConf conf, String jobTempDir)`
    `static void` | `main(String[] args)`
    `void` | `notifyIsLastAMRetry(boolean isLastAMRetry)`
    `boolean` | `recovered()`
    `protected void` | `serviceInit(org.apache.hadoop.conf.Configuration conf)`
    `protected void` | `serviceStart()`
    `protected void` | `serviceStop()`
    `void` | `shutDownJob()`
    `protected void` | `startJobs()` This can be overridden to instantiate multiple jobs and create a workflow.
    `void` | `stop()`
    `protected void` | `sysexit()` Exit call.

    - ### Methods inherited from class org.apache.hadoop.service.CompositeService

      `addIfService, addService, getServices, removeService`

    - ### Methods inherited from class org.apache.hadoop.service.AbstractService

      `close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

- - ### Field Detail

    - #### SHUTDOWN_HOOK_PRIORITY

          public static final int SHUTDOWN_HOOK_PRIORITY

      Priority of the MRAppMaster shutdown hook.
      See Also:
        Constant Field Values

    - #### INTERMEDIATE_DATA_ENCRYPTION_ALGO

          public static final String INTERMEDIATE_DATA_ENCRYPTION_ALGO

      See Also:
        Constant Field Values

    - #### metrics

          protected final MRAppMetrics metrics

    - #### taskAttemptListener

          protected TaskAttemptListener taskAttemptListener

    - #### jobTokenSecretManager

          protected org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager jobTokenSecretManager

    - #### currentUser

          protected org.apache.hadoop.security.UserGroupInformation currentUser

    - #### isLastAMRetry

          protected volatile boolean isLastAMRetry

    - #### successfullyUnregistered

          protected AtomicBoolean successfullyUnregistered

  - ### Constructor Detail

    - #### MRAppMaster

          public MRAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId,
                             org.apache.hadoop.yarn.api.records.ContainerId containerId,
                             String nmHost,
                             int nmPort,
                             int nmHttpPort,
                             long appSubmitTime)

    - #### MRAppMaster

          public MRAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId,
                             org.apache.hadoop.yarn.api.records.ContainerId containerId,
                             String nmHost,
                             int nmPort,
                             int nmHttpPort,
                             org.apache.hadoop.yarn.util.Clock clock,
                             long appSubmitTime)

  - ### Method Detail

    - #### createTaskAttemptFinishingMonitor

          protected TaskAttemptFinishingMonitor createTaskAttemptFinishingMonitor(org.apache.hadoop.yarn.event.EventHandler eventHandler)

    - #### serviceInit

          protected void serviceInit(org.apache.hadoop.conf.Configuration conf)
                              throws Exception

      Overrides:
        `serviceInit` in class `org.apache.hadoop.service.CompositeService`
      Throws:
        `Exception`

    - #### createDispatcher

          protected org.apache.hadoop.yarn.event.Dispatcher createDispatcher()

    - #### createPreemptionPolicy

          protected AMPreemptionPolicy createPreemptionPolicy(org.apache.hadoop.conf.Configuration conf)

    - #### keepJobFiles

          protected boolean keepJobFiles(org.apache.hadoop.mapred.JobConf conf,
                                         String jobTempDir)

    - #### getFileSystem

          protected org.apache.hadoop.fs.FileSystem getFileSystem(org.apache.hadoop.conf.Configuration conf)
                                                           throws IOException

      Create the default file System for this job.
      Parameters:
        `conf` - the conf object
      Returns:
        the default filesystem for this job
      Throws:
        `IOException`

    - #### getCredentials

          protected org.apache.hadoop.security.Credentials getCredentials()

    - #### cleanupStagingDir

          public void cleanupStagingDir()
                                 throws IOException

      clean up staging directories for the job.
      Throws:
        `IOException`

    - #### sysexit

          protected void sysexit()

      Exit call. Just in a function call to enable testing.

    - #### shutDownJob

          public void shutDownJob()

    - #### createJobFinishEventHandler

          protected org.apache.hadoop.yarn.event.EventHandler<JobFinishEvent> createJobFinishEventHandler()

      create an event handler that handles the job finish event.
      Returns:
        the job finish event handler.

    - #### createJob

          protected Job createJob(org.apache.hadoop.conf.Configuration conf,
                                  JobStateInternal forcedState,
                                  String diagnostic)

      Create and initialize (but don't start) a single job.
      Parameters:
        `forcedState` - a state to force the job into or null for normal operation.
        `diagnostic` - a diagnostic message to include with the job.

    - #### initJobCredentialsAndUGI

          protected void initJobCredentialsAndUGI(org.apache.hadoop.conf.Configuration conf)

      Obtain the tokens needed by the job and put them in the UGI
      Parameters:
        `conf` -

    - #### createJobHistoryHandler

          protected org.apache.hadoop.yarn.event.EventHandler<JobHistoryEvent> createJobHistoryHandler(AppContext context)

    - #### createStagingDirCleaningService

          protected org.apache.hadoop.service.AbstractService createStagingDirCleaningService()

    - #### createSpeculator

          protected Speculator createSpeculator(org.apache.hadoop.conf.Configuration conf,
                                                AppContext context)

    - #### createTaskAttemptListener

          protected TaskAttemptListener createTaskAttemptListener(AppContext context,
                                                                  AMPreemptionPolicy preemptionPolicy)

    - #### createCommitterEventHandler

          protected org.apache.hadoop.yarn.event.EventHandler<CommitterEvent> createCommitterEventHandler(AppContext context,
                                                                                                          org.apache.hadoop.mapreduce.OutputCommitter committer)

    - #### createContainerAllocator

          protected ContainerAllocator createContainerAllocator(ClientService clientService,
                                                                AppContext context)

    - #### getRMHeartbeatHandler

          protected RMHeartbeatHandler getRMHeartbeatHandler()

    - #### createContainerLauncher

          protected ContainerLauncher createContainerLauncher(AppContext context)

    - #### createClientService

          protected ClientService createClientService(AppContext context)

    - #### getAppID

          public org.apache.hadoop.yarn.api.records.ApplicationId getAppID()

    - #### getAttemptID

          public org.apache.hadoop.yarn.api.records.ApplicationAttemptId getAttemptID()

    - #### getJobId

          public org.apache.hadoop.mapreduce.v2.api.records.JobId getJobId()

    - #### getCommitter

          public org.apache.hadoop.mapreduce.OutputCommitter getCommitter()

    - #### isNewApiCommitter

          public boolean isNewApiCommitter()

    - #### getStartCount

          public int getStartCount()

    - #### getContext

          public AppContext getContext()

    - #### getDispatcher

          public org.apache.hadoop.yarn.event.Dispatcher getDispatcher()

    - #### getCompletedTaskFromPreviousRun

          public Map<org.apache.hadoop.mapreduce.v2.api.records.TaskId,org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.TaskInfo> getCompletedTaskFromPreviousRun()

    - #### getAllAMInfos

          public List<org.apache.hadoop.mapreduce.v2.api.records.AMInfo> getAllAMInfos()

    - #### getContainerAllocator

          public ContainerAllocator getContainerAllocator()

    - #### getContainerLauncher

          public ContainerLauncher getContainerLauncher()

    - #### getTaskAttemptListener

          public TaskAttemptListener getTaskAttemptListener()

    - #### isLastAMRetry

          public Boolean isLastAMRetry()

    - #### serviceStart

          protected void serviceStart()
                               throws Exception

      Overrides:
        `serviceStart` in class `org.apache.hadoop.service.CompositeService`
      Throws:
        `Exception`

    - #### stop

          public void stop()

      Specified by:
        `stop` in interface `org.apache.hadoop.service.Service`
      Overrides:
        `stop` in class `org.apache.hadoop.service.AbstractService`

    - #### recovered

          public boolean recovered()

    - #### startJobs

          protected void startJobs()

      This can be overridden to instantiate multiple jobs and create a workflow. TODO: Rework the design to actually support this. Currently much of the job stuff has been moved to init() above to support uberization (MR-1220). In a typical workflow, one presumably would want to uberize only a subset of the jobs (the "small" ones), which is awkward with the current design.

    - #### main

          public static void main(String[] args)

    - #### notifyIsLastAMRetry

          public void notifyIsLastAMRetry(boolean isLastAMRetry)

    - #### initAndStartAppMaster

          protected static void initAndStartAppMaster(MRAppMaster appMaster,
                                                      org.apache.hadoop.mapred.JobConf conf,
                                                      String jobUserName)
                                               throws IOException,
                                                      InterruptedException

      Throws:
        `IOException`
        `InterruptedException`

    - #### serviceStop

          protected void serviceStop()
                              throws Exception

      Overrides:
        `serviceStop` in class `org.apache.hadoop.service.CompositeService`
      Throws:
        `Exception`

    - #### getClientService

          public ClientService getClientService()

Copyright © 2008–2022 Apache Software Foundation. All rights reserved.