org.apache.hadoop.util

## Class Shell

- java.lang.Object
- - org.apache.hadoop.util.Shell

- Direct Known Subclasses:
    Shell.ShellCommandExecutor

      @InterfaceAudience.Public
       @InterfaceStability.Evolving
      public abstract class Shell
      extends Object

  A base class for running a Shell command. `Shell` can be used to run shell commands like `du` or `df`. It also offers facilities to gate commands by time-intervals.

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Class and Description
    --- | ---
    `static interface` | `Shell.CommandExecutor`
    `static class` | `Shell.ExitCodeException` This is an IOException with exit code added.
    `static class` | `Shell.OSType`
    `static class` | `Shell.ShellCommandExecutor` A simple shell command executor.

  - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static String` | `ENV_HADOOP_HOME` Environment variable for Hadoop's home dir: "HADOOP_HOME".
    `static String` | `ENV_NAME_REGEX` Regular expression for environment variables: "[A-Za-z_][A-Za-z0-9_]*".
    `static boolean` | `FREEBSD`
    `protected boolean` | `inheritParentEnv` Indicates if the parent env vars should be inherited or not
    `static boolean` | `isSetsidAvailable` Flag which is true if setsid exists.
    `static String` | `LINK_COMMAND` a Unix command to create a link: "ln".
    `static boolean` | `LINUX`
    `static org.slf4j.Logger` | `LOG`
    `static boolean` | `MAC`
    `static Shell.OSType` | `osType` Get the type of the operating system, as determined from parsing the `os.name` property.
    `static boolean` | `OTHER`
    `static boolean` | `PPC_64`
    `static String` | `READ_LINK_COMMAND` a Unix command to get a link target: "readlink".
    `static String` | `SET_GROUP_COMMAND` a Unix command to set the change user's groups list: "chgrp".
    `static String` | `SET_OWNER_COMMAND` a Unix command to set owner: "chown".
    `static String` | `SET_PERMISSION_COMMAND` a Unix command to set permission: "chmod".
    `static boolean` | `SOLARIS`
    `static String` | `SYSPROP_HADOOP_HOME_DIR` System property for the Hadoop home directory: "hadoop.home.dir".
    `protected long` | `timeOutInterval` Time after which the executing script would be timedout.
    `static String` | `TOKEN_SEPARATOR_REGEX` Token separator regex used to parse Shell tool outputs.
    `static String` | `USER_NAME_COMMAND` a Unix command to get the current user's name: "whoami".
    `static boolean` | `WINDOWS`
    `static int` | `WINDOWS_MAX_SHELL_LENGHT` Deprecated. use the correctly spelled constant.
    `static int` | `WINDOWS_MAX_SHELL_LENGTH` Maximum command line length in Windows KB830473 documents this as 8191
    `static Object` | `WindowsProcessLaunchLock` Windows `CreateProcess` synchronization object.
    `static String` | `WINUTILS` Deprecated. use one of the exception-raising getter methods, specifically `getWinUtilsPath()` or `getWinUtilsFile()`

  - ### Constructor Summary

    Constructors
    Modifier | Constructor and Description
    --- | ---
    `protected` | `Shell()` Create an instance with no minimum interval between runs; stderr is not merged with stdout.
    `protected` | `Shell(long interval)` Create an instance with a minimum interval between executions; stderr is not merged with stdout.
    `protected` | `Shell(long interval, boolean redirectErrorStream)` Create a shell instance which can be re-executed when the `run()` method is invoked with a given elapsed time between calls.

  - ### Method Summary

    All Methods Static Methods Instance Methods Abstract Methods Concrete Methods Deprecated Methods
    Modifier and Type | Method and Description
    --- | ---
    `static File` | `appendScriptExtension(File parent, String basename)` Returns a File referencing a script with the given basename, inside the given parent directory.
    `static String` | `appendScriptExtension(String basename)` Returns a script file name with the given basename.
    `static boolean` | `checkIsBashSupported()`
    `static void` | `checkWindowsCommandLineLength(String... commands)` Checks if a given command (String[]) fits in the Windows maximum command line length Note that the input is expected to already include space delimiters, no extra count will be added for delimiters.
    `static void` | `destroyAllShellProcesses()` Static method to destroy all running `Shell` processes.
    `static String` | `execCommand(Map<String,String> env, String... cmd)` Static method to execute a shell command.
    `static String` | `execCommand(Map<String,String> env, String[] cmd, long timeout)` Static method to execute a shell command.
    `static String` | `execCommand(String... cmd)` Static method to execute a shell command.
    `static Set<Shell>` | `getAllShells()` Static method to return a Set of all `Shell` objects.
    `static String[]` | `getCheckProcessIsAliveCommand(String pid)` Return a command for determining if process with specified pid is alive.
    `String` | `getEnvironment(String env)` Get an environment variable.
    `static String` | `getEnvironmentVariableRegex()` Return a regular expression string that match environment variables.
    `protected abstract String[]` | `getExecString()` return an array containing the command name and its parameters.
    `int` | `getExitCode()` get the exit code.
    `static String[]` | `getGetPermissionCommand()` Return a command to get permission information.
    `static String[]` | `getGroupsCommand()` a Unix command to get the current user's groups list.
    `static String[]` | `getGroupsForUserCommand(String user)` A command to get a given user's groups list.
    `static String[]` | `getGroupsIDForUserCommand(String user)` A command to get a given user's group id list.
    `static String` | `getHadoopHome()` Get the Hadoop home directory.
    `static Long` | `getMemlockLimit(Long ulimit)` Static method to return the memory lock limit for datanode.
    `Process` | `getProcess()` get the current sub-process executing the given command.
    `static File` | `getQualifiedBin(String executable)` Fully qualify the path to a binary that should be in a known hadoop bin location.
    `static String` | `getQualifiedBinPath(String executable)` Fully qualify the path to a binary that should be in a known hadoop bin location.
    `static String[]` | `getReadlinkCommand(String link)` Return a command to read the target of the a symbolic link.
    `static String[]` | `getRunScriptCommand(File script)` Returns a command to run the given script.
    `static String[]` | `getSetOwnerCommand(String owner)` Return a command to set owner.
    `static String[]` | `getSetPermissionCommand(String perm, boolean recursive)` Return a command to set permission.
    `static String[]` | `getSetPermissionCommand(String perm, boolean recursive, String file)` Return a command to set permission for specific file.
    `static String[]` | `getSignalKillCommand(int code, String pid)` Return a command to send a signal to a given pid.
    `static String[]` | `getSymlinkCommand(String target, String link)` Return a command to create symbolic links.
    `static String[]` | `getUsersForNetgroupCommand(String netgroup)` A command to get a given netgroup's user list.
    `Thread` | `getWaitingThread()` get the thread that is waiting on this instance of `Shell`.
    `static File` | `getWinUtilsFile()` Get a file reference to winutils.
    `static String` | `getWinUtilsPath()` Locate the winutils binary, or fail with a meaningful exception and stack trace as an RTE.
    `static boolean` | `hasWinutilsPath()` Predicate to indicate whether or not the path to winutils is known.
    `static boolean` | `isJava7OrAbove()` Deprecated. This call isn't needed any more: please remove uses of it.
    `static boolean` | `isJavaVersionAtLeast(int version)` Query to see if major version of Java specification of the system is equal or greater than the parameter.
    `boolean` | `isTimedOut()` To check if the passed script to shell command executor timed out or not.
    `protected abstract void` | `parseExecResult(BufferedReader lines)` Parse the execution result
    `protected void` | `run()` Check to see if a command needs to be executed and execute if needed.
    `protected void` | `setEnvironment(Map<String,String> env)` Set the environment for the command.
    `protected void` | `setWorkingDirectory(File dir)` Set the working directory.

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### LOG

          public static final org.slf4j.Logger LOG

    - #### SYSPROP_HADOOP_HOME_DIR

          public static final String SYSPROP_HADOOP_HOME_DIR

      System property for the Hadoop home directory: "hadoop.home.dir".
      See Also:
        Constant Field Values

    - #### ENV_HADOOP_HOME

          public static final String ENV_HADOOP_HOME

      Environment variable for Hadoop's home dir: "HADOOP_HOME".
      See Also:
        Constant Field Values

    - #### WINDOWS_MAX_SHELL_LENGTH

          public static final int WINDOWS_MAX_SHELL_LENGTH

      Maximum command line length in Windows KB830473 documents this as 8191
      See Also:
        Constant Field Values

    - #### WINDOWS_MAX_SHELL_LENGHT

          @Deprecated
          public static final int WINDOWS_MAX_SHELL_LENGHT

      Deprecated. use the correctly spelled constant.
      mis-spelling of `WINDOWS_MAX_SHELL_LENGTH`.
      See Also:
        Constant Field Values

    - #### USER_NAME_COMMAND

          public static final String USER_NAME_COMMAND

      a Unix command to get the current user's name: "whoami".
      See Also:
        Constant Field Values

    - #### WindowsProcessLaunchLock

          public static final Object WindowsProcessLaunchLock

      Windows `CreateProcess` synchronization object.

    - #### osType

          public static final Shell.OSType osType

      Get the type of the operating system, as determined from parsing the `os.name` property.

    - #### WINDOWS

          public static final boolean WINDOWS

    - #### SOLARIS

          public static final boolean SOLARIS

    - #### MAC

          public static final boolean MAC

    - #### FREEBSD

          public static final boolean FREEBSD

    - #### LINUX

          public static final boolean LINUX

    - #### OTHER

          public static final boolean OTHER

    - #### PPC_64

          public static final boolean PPC_64

    - #### ENV_NAME_REGEX

          public static final String ENV_NAME_REGEX

      Regular expression for environment variables: "[A-Za-z_][A-Za-z0-9_]*".
      See Also:
        Constant Field Values

    - #### SET_PERMISSION_COMMAND

          public static final String SET_PERMISSION_COMMAND

      a Unix command to set permission: "chmod".
      See Also:
        Constant Field Values

    - #### SET_OWNER_COMMAND

          public static final String SET_OWNER_COMMAND

      a Unix command to set owner: "chown".
      See Also:
        Constant Field Values

    - #### SET_GROUP_COMMAND

          public static final String SET_GROUP_COMMAND

      a Unix command to set the change user's groups list: "chgrp".
      See Also:
        Constant Field Values

    - #### LINK_COMMAND

          public static final String LINK_COMMAND

      a Unix command to create a link: "ln".
      See Also:
        Constant Field Values

    - #### READ_LINK_COMMAND

          public static final String READ_LINK_COMMAND

      a Unix command to get a link target: "readlink".
      See Also:
        Constant Field Values

    - #### timeOutInterval

          protected long timeOutInterval

      Time after which the executing script would be timedout.

    - #### inheritParentEnv

          protected boolean inheritParentEnv

      Indicates if the parent env vars should be inherited or not

    - #### WINUTILS

          @Deprecated
          public static final String WINUTILS

      Deprecated. use one of the exception-raising getter methods, specifically `getWinUtilsPath()` or `getWinUtilsFile()`
      Location of winutils as a string; null if not found.

      Important: caller must check for this value being null. The lack of such checks has led to many support issues being raised.

    - #### isSetsidAvailable

          public static final boolean isSetsidAvailable

      Flag which is true if setsid exists.

    - #### TOKEN_SEPARATOR_REGEX

          public static final String TOKEN_SEPARATOR_REGEX

      Token separator regex used to parse Shell tool outputs.

  - ### Constructor Detail

    - #### Shell

          protected Shell()

      Create an instance with no minimum interval between runs; stderr is not merged with stdout.

    - #### Shell

          protected Shell(long interval)

      Create an instance with a minimum interval between executions; stderr is not merged with stdout.
      Parameters:
        `interval` - interval in milliseconds between command executions.

    - #### Shell

          protected Shell(long interval,
                          boolean redirectErrorStream)

      Create a shell instance which can be re-executed when the `run()` method is invoked with a given elapsed time between calls.
      Parameters:
        `interval` - the minimum duration in milliseconds to wait before re-executing the command. If set to 0, there is no minimum.
        `redirectErrorStream` - should the error stream be merged with the normal output stream?

  - ### Method Detail

    - #### isJava7OrAbove

          @Deprecated
          public static boolean isJava7OrAbove()

      Deprecated. This call isn't needed any more: please remove uses of it.
      query to see if system is Java 7 or later. Now that Hadoop requires Java 7 or later, this always returns true.
      Returns:
        true, always.

    - #### isJavaVersionAtLeast

          public static boolean isJavaVersionAtLeast(int version)

      Query to see if major version of Java specification of the system is equal or greater than the parameter.
      Parameters:
        `version` - 8, 9, 10 etc.
      Returns:
        comparison with system property, always true for 8

    - #### checkWindowsCommandLineLength

          public static void checkWindowsCommandLineLength(String... commands)
                                                    throws IOException

      Checks if a given command (String[]) fits in the Windows maximum command line length Note that the input is expected to already include space delimiters, no extra count will be added for delimiters.
      Parameters:
        `commands` - command parts, including any space delimiters
      Throws:
        `IOException`

    - #### getGroupsCommand

          public static String[] getGroupsCommand()

      a Unix command to get the current user's groups list.

    - #### getGroupsForUserCommand

          public static String[] getGroupsForUserCommand(String user)

      A command to get a given user's groups list. If the OS is not WINDOWS, the command will get the user's primary group first and finally get the groups list which includes the primary group. i.e. the user's primary group will be included twice.

    - #### getGroupsIDForUserCommand

          public static String[] getGroupsIDForUserCommand(String user)

      A command to get a given user's group id list. The command will get the user's primary group first and finally get the groups list which includes the primary group. i.e. the user's primary group will be included twice. This command does not support Windows and will only return group names.

    - #### getUsersForNetgroupCommand

          public static String[] getUsersForNetgroupCommand(String netgroup)

      A command to get a given netgroup's user list.

    - #### getGetPermissionCommand

          public static String[] getGetPermissionCommand()

      Return a command to get permission information.

    - #### getSetPermissionCommand

          public static String[] getSetPermissionCommand(String perm,
                                                         boolean recursive)

      Return a command to set permission.

    - #### getSetPermissionCommand

          public static String[] getSetPermissionCommand(String perm,
                                                         boolean recursive,
                                                         String file)

      Return a command to set permission for specific file.
      Parameters:
        `perm` - String permission to set
        `recursive` - boolean true to apply to all sub-directories recursively
        `file` - String file to set
      Returns:
        String[] containing command and arguments

    - #### getSetOwnerCommand

          public static String[] getSetOwnerCommand(String owner)

      Return a command to set owner.

    - #### getSymlinkCommand

          public static String[] getSymlinkCommand(String target,
                                                   String link)

      Return a command to create symbolic links.

    - #### getReadlinkCommand

          public static String[] getReadlinkCommand(String link)

      Return a command to read the target of the a symbolic link.

    - #### getCheckProcessIsAliveCommand

          public static String[] getCheckProcessIsAliveCommand(String pid)

      Return a command for determining if process with specified pid is alive.
      Parameters:
        `pid` - process ID
      Returns:
        a `kill -0` command or equivalent

    - #### getSignalKillCommand

          public static String[] getSignalKillCommand(int code,
                                                      String pid)

      Return a command to send a signal to a given pid.

    - #### getEnvironmentVariableRegex

          public static String getEnvironmentVariableRegex()

      Return a regular expression string that match environment variables.

    - #### appendScriptExtension

          public static File appendScriptExtension(File parent,
                                                   String basename)

      Returns a File referencing a script with the given basename, inside the given parent directory. The file extension is inferred by platform: `".cmd"` on Windows, or `".sh"` otherwise.
      Parameters:
        `parent` - File parent directory
        `basename` - String script file basename
      Returns:
        File referencing the script in the directory

    - #### appendScriptExtension

          public static String appendScriptExtension(String basename)

      Returns a script file name with the given basename. The file extension is inferred by platform: `".cmd"` on Windows, or `".sh"` otherwise.
      Parameters:
        `basename` - String script file basename
      Returns:
        String script file name

    - #### getRunScriptCommand

          public static String[] getRunScriptCommand(File script)

      Returns a command to run the given script. The script interpreter is inferred by platform: cmd on Windows or bash otherwise.
      Parameters:
        `script` - File script to run
      Returns:
        String[] command to run the script

    - #### getHadoopHome

          public static String getHadoopHome()
                                      throws IOException

      Get the Hadoop home directory. Raises an exception if not found
      Returns:
        the home dir
      Throws:
        `IOException` - if the home directory cannot be located.

    - #### getQualifiedBin

          public static File getQualifiedBin(String executable)
                                      throws FileNotFoundException

      Fully qualify the path to a binary that should be in a known hadoop bin location. This is primarily useful for disambiguating call-outs to executable sub-components of Hadoop to avoid clashes with other executables that may be in the path. Caveat: this call doesn't just format the path to the bin directory. It also checks for file existence of the composed path. The output of this call should be cached by callers.
      Parameters:
        `executable` - executable
      Returns:
        executable file reference
      Throws:
        `FileNotFoundException` - if the path does not exist

    - #### getQualifiedBinPath

          public static String getQualifiedBinPath(String executable)
                                            throws IOException

      Fully qualify the path to a binary that should be in a known hadoop bin location. This is primarily useful for disambiguating call-outs to executable sub-components of Hadoop to avoid clashes with other executables that may be in the path. Caveat: this call doesn't just format the path to the bin directory. It also checks for file existence of the composed path. The output of this call should be cached by callers.
      Parameters:
        `executable` - executable
      Returns:
        executable file reference
      Throws:
        `FileNotFoundException` - if the path does not exist
        `IOException` - on path canonicalization failures

    - #### hasWinutilsPath

          public static boolean hasWinutilsPath()

      Predicate to indicate whether or not the path to winutils is known. If true, then `WINUTILS` is non-null, and both `getWinUtilsPath()` and `getWinUtilsFile()` will successfully return this value. Always false on non-windows systems.
      Returns:
        true if there is a valid path to the binary

    - #### getWinUtilsPath

          public static String getWinUtilsPath()

      Locate the winutils binary, or fail with a meaningful exception and stack trace as an RTE. This method is for use in methods which don't explicitly throw an `IOException`.
      Returns:
        the path to `WINUTILS_EXE`
      Throws:
        `RuntimeException` - if the path is not resolvable

    - #### getWinUtilsFile

          public static File getWinUtilsFile()
                                      throws FileNotFoundException

      Get a file reference to winutils. Always raises an exception if there isn't one
      Returns:
        the file instance referring to the winutils bin.
      Throws:
        `FileNotFoundException` - on any failure to locate that file.

    - #### checkIsBashSupported

          public static boolean checkIsBashSupported()
                                              throws InterruptedIOException

      Throws:
        `InterruptedIOException`

    - #### setEnvironment

          protected void setEnvironment(Map<String,String> env)

      Set the environment for the command.
      Parameters:
        `env` - Mapping of environment variables

    - #### setWorkingDirectory

          protected void setWorkingDirectory(File dir)

      Set the working directory.
      Parameters:
        `dir` - The directory where the command will be executed

    - #### run

          protected void run()
                      throws IOException

      Check to see if a command needs to be executed and execute if needed.
      Throws:
        `IOException`

    - #### getExecString

          protected abstract String[] getExecString()

      return an array containing the command name and its parameters.

    - #### parseExecResult

          protected abstract void parseExecResult(BufferedReader lines)
                                           throws IOException

      Parse the execution result
      Throws:
        `IOException`

    - #### getEnvironment

          public String getEnvironment(String env)

      Get an environment variable.
      Parameters:
        `env` - the environment var
      Returns:
        the value or null if it was unset.

    - #### getProcess

          public Process getProcess()

      get the current sub-process executing the given command.
      Returns:
        process executing the command

    - #### getExitCode

          public int getExitCode()

      get the exit code.
      Returns:
        the exit code of the process

    - #### getWaitingThread

          public Thread getWaitingThread()

      get the thread that is waiting on this instance of `Shell`.
      Returns:
        the thread that ran runCommand() that spawned this shell or null if no thread is waiting for this shell to complete

    - #### isTimedOut

          public boolean isTimedOut()

      To check if the passed script to shell command executor timed out or not.
      Returns:
        if the script timed out.

    - #### execCommand

          public static String execCommand(String... cmd)
                                    throws IOException

      Static method to execute a shell command. Covers most of the simple cases without requiring the user to implement the `Shell` interface.
      Parameters:
        `cmd` - shell command to execute.
      Returns:
        the output of the executed command.
      Throws:
        `IOException`

    - #### execCommand

          public static String execCommand(Map<String,String> env,
                                           String[] cmd,
                                           long timeout)
                                    throws IOException

      Static method to execute a shell command. Covers most of the simple cases without requiring the user to implement the `Shell` interface.
      Parameters:
        `env` - the map of environment key=value
        `cmd` - shell command to execute.
        `timeout` - time in milliseconds after which script should be marked timeout
      Returns:
        the output of the executed command.
      Throws:
        `IOException` - on any problem.

    - #### execCommand

          public static String execCommand(Map<String,String> env,
                                           String... cmd)
                                    throws IOException

      Static method to execute a shell command. Covers most of the simple cases without requiring the user to implement the `Shell` interface.
      Parameters:
        `env` - the map of environment key=value
        `cmd` - shell command to execute.
      Returns:
        the output of the executed command.
      Throws:
        `IOException` - on any problem.

    - #### destroyAllShellProcesses

          public static void destroyAllShellProcesses()

      Static method to destroy all running `Shell` processes. Iterates through a map of all currently running `Shell` processes and destroys them one by one. This method is thread safe

    - #### getAllShells

          public static Set<Shell> getAllShells()

      Static method to return a Set of all `Shell` objects.

    - #### getMemlockLimit

          public static Long getMemlockLimit(Long ulimit)

      Static method to return the memory lock limit for datanode.
      Parameters:
        `ulimit` - max value at which memory locked should be capped.
      Returns:
        long value specifying the memory lock limit.

Copyright © 2008–2022 Apache Software Foundation. All rights reserved.