org.apache.hadoop.yarn.server.nodemanager

## Class ContainerExecutor

- java.lang.Object
- - org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor

- All Implemented Interfaces:
    org.apache.hadoop.conf.Configurable
  Direct Known Subclasses:
    DefaultContainerExecutor, LinuxContainerExecutor

      public abstract class ContainerExecutor
      extends Object
      implements org.apache.hadoop.conf.Configurable

  This class is abstraction of the mechanism used to launch a container on the underlying OS. All executor implementations must extend ContainerExecutor.

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Class and Description
    --- | ---
    `static class` | `ContainerExecutor.DelayedProcessKiller` This class will signal a target container after a specified delay.
    `static class` | `ContainerExecutor.ExitCode` The container exit code.
    `static class` | `ContainerExecutor.Signal` The constants for the signals.

  - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static String` | `DIRECTORY_CONTENTS` The relative path to which debug information will be written.
    `static org.apache.hadoop.fs.permission.FsPermission` | `TASK_LAUNCH_SCRIPT_PERMISSION` The permissions to use when creating the launch script.
    `static String` | `TOKEN_FILE_NAME_FMT`
    `protected static String` | `WILDCARD`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `ContainerExecutor()`

  - ### Method Summary

    All Methods Static Methods Instance Methods Abstract Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `activateContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.fs.Path pidFilePath)` Mark the container as active.
    `void` | `cleanupBeforeRelaunch(Container container)` Perform any cleanup before the next launch of the container.
    `void` | `deactivateContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)` Mark the container as inactive.
    `abstract void` | `deleteAsUser(DeletionAsUserContext ctx)` Delete specified directories as a given user.
    `org.apache.hadoop.conf.Configuration` | `getConf()`
    `String[]` | `getIpAndHost(Container container)`
    `static String[]` | `getLocalIpAndHost(Container container)`
    `protected String` | `getNMEnvVar(String varname)`
    `protected org.apache.hadoop.fs.Path` | `getPidFilePath(org.apache.hadoop.yarn.api.records.ContainerId containerId)` Get the pidFile of the container.
    `String` | `getProcessId(org.apache.hadoop.yarn.api.records.ContainerId containerID)` Get the process-identifier for the container.
    `protected String[]` | `getRunCommand(String command, String groupId, String userName, org.apache.hadoop.fs.Path pidFile, org.apache.hadoop.conf.Configuration config)` Return a command line to execute the given command in the OS shell.
    `protected String[]` | `getRunCommand(String command, String groupId, String userName, org.apache.hadoop.fs.Path pidFile, org.apache.hadoop.conf.Configuration config, org.apache.hadoop.yarn.api.records.Resource resource)` Return a command line to execute the given command in the OS shell.
    `protected String[]` | `getRunCommandForOther(String command, org.apache.hadoop.conf.Configuration config)` Return a command line to execute the given command in the OS shell.
    `protected String[]` | `getRunCommandForWindows(String command, String groupId, String userName, org.apache.hadoop.fs.Path pidFile, org.apache.hadoop.conf.Configuration config, org.apache.hadoop.yarn.api.records.Resource resource)` Return a command line to execute the given command in the OS shell.
    `abstract void` | `init(Context nmContext)` Run the executor initialization steps.
    `protected boolean` | `isContainerActive(org.apache.hadoop.yarn.api.records.ContainerId containerId)` Return whether the container is still active.
    `abstract boolean` | `isContainerAlive(ContainerLivenessContext ctx)` Check if a container is alive.
    `abstract int` | `launchContainer(ContainerStartContext ctx)` Launch the container on the node.
    `org.apache.hadoop.fs.Path` | `localizeClasspathJar(org.apache.hadoop.fs.Path jarPath, org.apache.hadoop.fs.Path target, String owner)` This function localizes the JAR file on-demand.
    `protected void` | `logOutput(String output)` Log each line of the output string as INFO level log messages.
    `void` | `pauseContainer(Container container)` Pause the container.
    `void` | `prepareContainer(ContainerPrepareContext ctx)` Prepare the container prior to the launch environment being written.
    `int` | `reacquireContainer(ContainerReacquisitionContext ctx)` Recover an already existing container.
    `protected File[]` | `readDirAsUser(String user, org.apache.hadoop.fs.Path dir)` Return the files in the target directory.
    `abstract boolean` | `reapContainer(ContainerReapContext ctx)` Perform the steps necessary to reap the container.
    `abstract int` | `relaunchContainer(ContainerStartContext ctx)` Relaunch the container on the node.
    `void` | `resumeContainer(Container container)` Resume the container from pause state.
    `void` | `setConf(org.apache.hadoop.conf.Configuration conf)`
    `abstract boolean` | `signalContainer(ContainerSignalContext ctx)` Signal container with the specified signal.
    `abstract void` | `startLocalizer(LocalizerStartContext ctx)` Prepare the environment for containers in this application to execute.
    `abstract void` | `symLink(String target, String symlink)` Create a symlink file which points to the target.
    `void` | `writeLaunchEnv(OutputStream out, Map<String,String> environment, Map<org.apache.hadoop.fs.Path,List<String>> resources, List<String> command, org.apache.hadoop.fs.Path logDir, String user, LinkedHashSet<String> nmVars)` This method writes out the launch environment of a container to the default container launch script.
    `void` | `writeLaunchEnv(OutputStream out, Map<String,String> environment, Map<org.apache.hadoop.fs.Path,List<String>> resources, List<String> command, org.apache.hadoop.fs.Path logDir, String user, String outFilename, LinkedHashSet<String> nmVars)` This method writes out the launch environment of a container to a specified path.

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### WILDCARD

          protected static final String WILDCARD

      See Also:
        Constant Field Values

    - #### TOKEN_FILE_NAME_FMT

          public static final String TOKEN_FILE_NAME_FMT

      See Also:
        Constant Field Values

    - #### TASK_LAUNCH_SCRIPT_PERMISSION

          public static final org.apache.hadoop.fs.permission.FsPermission TASK_LAUNCH_SCRIPT_PERMISSION

      The permissions to use when creating the launch script.

    - #### DIRECTORY_CONTENTS

          public static final String DIRECTORY_CONTENTS

      The relative path to which debug information will be written.
      See Also:
        `ContainerLaunch.ShellScriptBuilder.listDebugInformation(org.apache.hadoop.fs.Path)`, Constant Field Values

  - ### Constructor Detail

    - #### ContainerExecutor

          public ContainerExecutor()

  - ### Method Detail

    - #### setConf

          public void setConf(org.apache.hadoop.conf.Configuration conf)

      Specified by:
        `setConf` in interface `org.apache.hadoop.conf.Configurable`

    - #### getConf

          public org.apache.hadoop.conf.Configuration getConf()

      Specified by:
        `getConf` in interface `org.apache.hadoop.conf.Configurable`

    - #### init

          public abstract void init(Context nmContext)
                             throws IOException

      Run the executor initialization steps. Verify that the necessary configs and permissions are in place.
      Parameters:
        `nmContext` - Context of NM
      Throws:
        `IOException` - if initialization fails

    - #### localizeClasspathJar

          public org.apache.hadoop.fs.Path localizeClasspathJar(org.apache.hadoop.fs.Path jarPath,
                                                                org.apache.hadoop.fs.Path target,
                                                                String owner)
                                                         throws IOException

      This function localizes the JAR file on-demand. On Windows the ContainerLaunch creates a temporary special JAR manifest of other JARs to workaround the CLASSPATH length. In a secure cluster this JAR must be localized so that the container has access to it. The default implementation returns the classpath passed to it, which is expected to have been created in the node manager's fprivate folder, which will not work with secure Windows clusters.
      Parameters:
        `jarPath` - the path to the JAR to localize
        `target` - the directory where the JAR file should be localized
        `owner` - the name of the user who should own the localized file
      Returns:
        the path to the localized JAR file
      Throws:
        `IOException` - if localization fails

    - #### startLocalizer

          public abstract void startLocalizer(LocalizerStartContext ctx)
                                       throws IOException,
                                              InterruptedException

      Prepare the environment for containers in this application to execute.

           For $x in local.dirs
             create $x/$user/$appId
           Copy $nmLocal/appTokens  -> $N/$user/$appId
           For $rsrc in private resources
             Copy $rsrc  -> $N/$user/filecache/[idef]
           For $rsrc in job resources
             Copy $rsrc  -> $N/$user/$appId/filecache/idef

      Parameters:
        `ctx` - LocalizerStartContext that encapsulates necessary information for starting a localizer.
      Throws:
        `IOException` - for most application init failures
        `InterruptedException` - if application init thread is halted by NM

    - #### prepareContainer

          public void prepareContainer(ContainerPrepareContext ctx)
                                throws IOException

      Prepare the container prior to the launch environment being written.
      Parameters:
        `ctx` - Encapsulates information necessary for launching containers.
      Throws:
        `IOException` - if errors occur during container preparation

    - #### launchContainer

          public abstract int launchContainer(ContainerStartContext ctx)
                                       throws IOException,
                                              org.apache.hadoop.yarn.exceptions.ConfigurationException

      Launch the container on the node. This is a blocking call and returns only when the container exits.
      Parameters:
        `ctx` - Encapsulates information necessary for launching containers.
      Returns:
        the return status of the launch
      Throws:
        `IOException` - if the container launch fails
        `org.apache.hadoop.yarn.exceptions.ConfigurationException` - if config error was found

    - #### relaunchContainer

          public abstract int relaunchContainer(ContainerStartContext ctx)
                                         throws IOException,
                                                org.apache.hadoop.yarn.exceptions.ConfigurationException

      Relaunch the container on the node. This is a blocking call and returns only when the container exits.
      Parameters:
        `ctx` - Encapsulates information necessary for relaunching containers.
      Returns:
        the return status of the relaunch
      Throws:
        `IOException` - if the container relaunch fails
        `org.apache.hadoop.yarn.exceptions.ConfigurationException` - if config error was found

    - #### signalContainer

          public abstract boolean signalContainer(ContainerSignalContext ctx)
                                           throws IOException

      Signal container with the specified signal.
      Parameters:
        `ctx` - Encapsulates information necessary for signaling containers.
      Returns:
        returns true if the operation succeeded
      Throws:
        `IOException` - if signaling the container fails

    - #### reapContainer

          public abstract boolean reapContainer(ContainerReapContext ctx)
                                         throws IOException

      Perform the steps necessary to reap the container.
      Parameters:
        `ctx` - Encapsulates information necessary for reaping containers.
      Returns:
        returns true if the operation succeeded.
      Throws:
        `IOException` - if reaping the container fails.

    - #### deleteAsUser

          public abstract void deleteAsUser(DeletionAsUserContext ctx)
                                     throws IOException,
                                            InterruptedException

      Delete specified directories as a given user.
      Parameters:
        `ctx` - Encapsulates information necessary for deletion.
      Throws:
        `IOException` - if delete fails
        `InterruptedException` - if interrupted while waiting for the deletion operation to complete

    - #### symLink

          public abstract void symLink(String target,
                                       String symlink)
                                throws IOException

      Create a symlink file which points to the target.
      Parameters:
        `target` - The target for symlink
        `symlink` - the symlink file
      Throws:
        `IOException` - Error when creating symlinks

    - #### isContainerAlive

          public abstract boolean isContainerAlive(ContainerLivenessContext ctx)
                                            throws IOException

      Check if a container is alive.
      Parameters:
        `ctx` - Encapsulates information necessary for container liveness check.
      Returns:
        true if container is still alive
      Throws:
        `IOException` - if there is a failure while checking the container status

    - #### reacquireContainer

          public int reacquireContainer(ContainerReacquisitionContext ctx)
                                 throws IOException,
                                        InterruptedException

      Recover an already existing container. This is a blocking call and returns only when the container exits. Note that the container must have been activated prior to this call.
      Parameters:
        `ctx` - encapsulates information necessary to reacquire container
      Returns:
        The exit code of the pre-existing container
      Throws:
        `IOException` - if there is a failure while reacquiring the container
        `InterruptedException` - if interrupted while waiting to reacquire the container

    - #### writeLaunchEnv

          public void writeLaunchEnv(OutputStream out,
                                     Map<String,String> environment,
                                     Map<org.apache.hadoop.fs.Path,List<String>> resources,
                                     List<String> command,
                                     org.apache.hadoop.fs.Path logDir,
                                     String user,
                                     LinkedHashSet<String> nmVars)
                              throws IOException

      This method writes out the launch environment of a container to the default container launch script. For the default container script path see `ContainerLaunch.CONTAINER_SCRIPT`.
      Parameters:
        `out` - the output stream to which the environment is written (usually a script file which will be executed by the Launcher)
        `environment` - the environment variables and their values
        `resources` - the resources which have been localized for this container. Symlinks will be created to these localized resources
        `command` - the command that will be run
        `logDir` - the log dir to which to copy debugging information
        `user` - the username of the job owner
        `nmVars` - the set of environment vars that are explicitly set by NM
      Throws:
        `IOException` - if any errors happened writing to the OutputStream, while creating symlinks

    - #### writeLaunchEnv

          public void writeLaunchEnv(OutputStream out,
                                     Map<String,String> environment,
                                     Map<org.apache.hadoop.fs.Path,List<String>> resources,
                                     List<String> command,
                                     org.apache.hadoop.fs.Path logDir,
                                     String user,
                                     String outFilename,
                                     LinkedHashSet<String> nmVars)
                              throws IOException

      This method writes out the launch environment of a container to a specified path.
      Parameters:
        `out` - the output stream to which the environment is written (usually a script file which will be executed by the Launcher)
        `environment` - the environment variables and their values
        `resources` - the resources which have been localized for this container. Symlinks will be created to these localized resources
        `command` - the command that will be run
        `logDir` - the log dir to which to copy debugging information
        `user` - the username of the job owner
        `outFilename` - the path to which to write the launch environment
        `nmVars` - the set of environment vars that are explicitly set by NM
      Throws:
        `IOException` - if any errors happened writing to the OutputStream, while creating symlinks

    - #### readDirAsUser

          protected File[] readDirAsUser(String user,
                                         org.apache.hadoop.fs.Path dir)

      Return the files in the target directory. If retrieving the list of files requires specific access rights, that access will happen as the specified user. The list will not include entries for "." or "..".
      Parameters:
        `user` - the user as whom to access the target directory
        `dir` - the target directory
      Returns:
        a list of files in the target directory

    - #### logOutput

          protected void logOutput(String output)

      Log each line of the output string as INFO level log messages.
      Parameters:
        `output` - the output string to log

    - #### getPidFilePath

          protected org.apache.hadoop.fs.Path getPidFilePath(org.apache.hadoop.yarn.api.records.ContainerId containerId)

      Get the pidFile of the container.
      Parameters:
        `containerId` - the container ID
      Returns:
        the path of the pid-file for the given containerId.

    - #### getRunCommand

          protected String[] getRunCommand(String command,
                                           String groupId,
                                           String userName,
                                           org.apache.hadoop.fs.Path pidFile,
                                           org.apache.hadoop.conf.Configuration config)

      Return a command line to execute the given command in the OS shell. On Windows, the {code}groupId{code} parameter can be used to launch and associate the given GID with a process group. On non-Windows hosts, the {code}groupId{code} parameter is ignored.
      Parameters:
        `command` - the command to execute
        `groupId` - the job owner's GID
        `userName` - the job owner's username
        `pidFile` - the path to the container's PID file
        `config` - the configuration
      Returns:
        the command line to execute

    - #### getRunCommand

          protected String[] getRunCommand(String command,
                                           String groupId,
                                           String userName,
                                           org.apache.hadoop.fs.Path pidFile,
                                           org.apache.hadoop.conf.Configuration config,
                                           org.apache.hadoop.yarn.api.records.Resource resource)

      Return a command line to execute the given command in the OS shell. On Windows, the {code}groupId{code} parameter can be used to launch and associate the given GID with a process group. On non-Windows hosts, the {code}groupId{code} parameter is ignored.
      Parameters:
        `command` - the command to execute
        `groupId` - the job owner's GID for Windows. On other operating systems it is ignored.
        `userName` - the job owner's username for Windows. On other operating systems it is ignored.
        `pidFile` - the path to the container's PID file on Windows. On other operating systems it is ignored.
        `config` - the configuration
        `resource` - on Windows this parameter controls memory and CPU limits. If null, no limits are set. On other operating systems it is ignored.
      Returns:
        the command line to execute

    - #### getRunCommandForWindows

          protected String[] getRunCommandForWindows(String command,
                                                     String groupId,
                                                     String userName,
                                                     org.apache.hadoop.fs.Path pidFile,
                                                     org.apache.hadoop.conf.Configuration config,
                                                     org.apache.hadoop.yarn.api.records.Resource resource)

      Return a command line to execute the given command in the OS shell. The {code}groupId{code} parameter can be used to launch and associate the given GID with a process group.
      Parameters:
        `command` - the command to execute
        `groupId` - the job owner's GID
        `userName` - the job owner's username
        `pidFile` - the path to the container's PID file
        `config` - the configuration
        `resource` - this parameter controls memory and CPU limits. If null, no limits are set.
      Returns:
        the command line to execute

    - #### getRunCommandForOther

          protected String[] getRunCommandForOther(String command,
                                                   org.apache.hadoop.conf.Configuration config)

      Return a command line to execute the given command in the OS shell.
      Parameters:
        `command` - the command to execute
        `config` - the configuration
      Returns:
        the command line to execute

    - #### isContainerActive

          protected boolean isContainerActive(org.apache.hadoop.yarn.api.records.ContainerId containerId)

      Return whether the container is still active.
      Parameters:
        `containerId` - the target container's ID
      Returns:
        true if the container is active

    - #### getNMEnvVar

          protected String getNMEnvVar(String varname)

    - #### activateContainer

          public void activateContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                                        org.apache.hadoop.fs.Path pidFilePath)

      Mark the container as active.
      Parameters:
        `containerId` - the container ID
        `pidFilePath` - the path where the executor should write the PID of the launched process

    - #### getIpAndHost

          public String[] getIpAndHost(Container container)
                                throws ContainerExecutionException

      Throws:
        `ContainerExecutionException`

    - #### getLocalIpAndHost

          public static String[] getLocalIpAndHost(Container container)

    - #### deactivateContainer

          public void deactivateContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)

      Mark the container as inactive. For inactive containers this method has no effect.
      Parameters:
        `containerId` - the container ID

    - #### pauseContainer

          public void pauseContainer(Container container)

      Pause the container. The default implementation is to raise a kill event. Specific executor implementations can override this behavior.
      Parameters:
        `container` - the Container

    - #### resumeContainer

          public void resumeContainer(Container container)

      Resume the container from pause state. The default implementation ignores this event. Specific implementations can override this behavior.
      Parameters:
        `container` - the Container

    - #### cleanupBeforeRelaunch

          public void cleanupBeforeRelaunch(Container container)
                                     throws IOException,
                                            InterruptedException

      Perform any cleanup before the next launch of the container.
      Parameters:
        `container` - container
      Throws:
        `IOException`
        `InterruptedException`

    - #### getProcessId

          public String getProcessId(org.apache.hadoop.yarn.api.records.ContainerId containerID)

      Get the process-identifier for the container.
      Parameters:
        `containerID` - the container ID
      Returns:
        the process ID of the container if it has already launched, or null otherwise

Copyright © 2008–2022 Apache Software Foundation. All rights reserved.