org.apache.hadoop.yarn.logaggregation.filecontroller

## Class LogAggregationFileController

- java.lang.Object
- - org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileController

-     @InterfaceAudience.Public
       @InterfaceStability.Unstable
      public abstract class LogAggregationFileController
      extends Object

  Base class to implement Log Aggregation File Controller.

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `protected static FsPermission` | `APP_DIR_PERMISSIONS` Permissions for the Application directory.
    `protected static FsPermission` | `APP_LOG_FILE_UMASK` Umask for the log file.
    `protected Configuration` | `conf`
    `protected String` | `fileControllerName`
    `protected boolean` | `fsSupportsChmod`
    `static String` | `LOG_AGGREGATION_FS_SUPPORT_APPEND`
    `protected Path` | `remoteRootLogDir`
    `protected String` | `remoteRootLogDirSuffix`
    `protected int` | `retentionSize`
    `protected static FsPermission` | `TLDIR_PERMISSIONS` Permissions for the top level directory under which app directories will be created.

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `LogAggregationFileController()`

  - ### Method Summary

    All Methods Instance Methods Abstract Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `protected String` | `aggregatedLogSuffix(String fileName)` Create the aggregated log suffix.
    `protected boolean` | `belongsToAppAttempt(ApplicationAttemptId appAttemptId, String containerIdStr)`
    `protected boolean` | `checkExists(FileSystem fs, Path path, FsPermission fsPerm)`
    `protected void` | `cleanOldLogs(Path remoteNodeLogFileForApp, NodeId nodeId, UserGroupInformation userUgi)`
    `protected void` | `closePrintStream(OutputStream out)`
    `abstract void` | `closeWriter()` Close the writer.
    `void` | `createAppDir(String user, ApplicationId appId, UserGroupInformation userUgi)` Create remote Application directory for log aggregation.
    `protected void` | `createDir(FileSystem fs, Path path, FsPermission fsPerm)`
    `abstract Map<ApplicationAccessType,String>` | `getApplicationAcls(Path aggregatedLogPath, ApplicationId appId)` Returns ACLs for the application.
    `abstract String` | `getApplicationOwner(Path aggregatedLogPath, ApplicationId appId)` Returns the owner of the application.
    `protected FileSystem` | `getFileSystem(Configuration conf)`
    `Path` | `getRemoteAppLogDir(ApplicationId appId, String appOwner)` Get the remote application directory for log aggregation.
    `Path` | `getRemoteNodeLogFileForApp(ApplicationId appId, String user, NodeId nodeId)` Get the remote aggregated log path.
    `Path` | `getRemoteRootLogDir()` Get the remote root log directory.
    `String` | `getRemoteRootLogDirSuffix()` Get the log aggregation directory suffix.
    `void` | `initialize(Configuration conf, String controllerName)` Initialize the log file controller.
    `abstract void` | `initializeWriter(org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileControllerContext context)` Initialize the writer.
    `protected abstract void` | `initInternal(Configuration conf)` Derived classes initialize themselves using this method.
    `boolean` | `isFsSupportsChmod()`
    `abstract void` | `postWrite(org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileControllerContext record)` Operations needed after write the log content.
    `abstract boolean` | `readAggregatedLogs(org.apache.hadoop.yarn.logaggregation.ContainerLogsRequest logRequest, OutputStream os)` Output container log.
    `abstract List<org.apache.hadoop.yarn.logaggregation.ContainerLogMeta>` | `readAggregatedLogsMeta(org.apache.hadoop.yarn.logaggregation.ContainerLogsRequest logRequest)` Return a list of `ContainerLogMeta` for an application from Remote FileSystem.
    `abstract void` | `renderAggregatedLogsBlock(org.apache.hadoop.yarn.webapp.view.HtmlBlock.Block html, org.apache.hadoop.yarn.webapp.View.ViewContext context)` Render Aggregated Logs block.
    `void` | `verifyAndCreateRemoteLogDir()` Verify and create the remote log directory.
    `abstract void` | `write(AggregatedLogFormat.LogKey logKey, org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat.LogValue logValue)` Write the log content.

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### TLDIR_PERMISSIONS

          protected static final FsPermission TLDIR_PERMISSIONS

      Permissions for the top level directory under which app directories will be created.

    - #### APP_DIR_PERMISSIONS

          protected static final FsPermission APP_DIR_PERMISSIONS

      Permissions for the Application directory.

    - #### APP_LOG_FILE_UMASK

          protected static final FsPermission APP_LOG_FILE_UMASK

      Umask for the log file.

    - #### LOG_AGGREGATION_FS_SUPPORT_APPEND

          public static final String LOG_AGGREGATION_FS_SUPPORT_APPEND

      See Also:
        Constant Field Values

    - #### conf

          protected Configuration conf

    - #### remoteRootLogDir

          protected Path remoteRootLogDir

    - #### remoteRootLogDirSuffix

          protected String remoteRootLogDirSuffix

    - #### retentionSize

          protected int retentionSize

    - #### fileControllerName

          protected String fileControllerName

    - #### fsSupportsChmod

          protected boolean fsSupportsChmod

  - ### Constructor Detail

    - #### LogAggregationFileController

          public LogAggregationFileController()

  - ### Method Detail

    - #### initialize

          public void initialize(Configuration conf,
                                 String controllerName)

      Initialize the log file controller.
      Parameters:
        `conf` - the Configuration
        `controllerName` - the log controller class name

    - #### initInternal

          protected abstract void initInternal(Configuration conf)

      Derived classes initialize themselves using this method.
      Parameters:
        `conf` - the Configuration

    - #### getRemoteRootLogDir

          public Path getRemoteRootLogDir()

      Get the remote root log directory.
      Returns:
        the remote root log directory path

    - #### getRemoteRootLogDirSuffix

          public String getRemoteRootLogDirSuffix()

      Get the log aggregation directory suffix.
      Returns:
        the log aggregation directory suffix

    - #### initializeWriter

          public abstract void initializeWriter(org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileControllerContext context)
                                         throws IOException

      Initialize the writer.
      Parameters:
        `context` - the `LogAggregationFileControllerContext`
      Throws:
        `IOException` - if fails to initialize the writer

    - #### closeWriter

          public abstract void closeWriter()
                                    throws org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationDFSException

      Close the writer.
      Throws:
        `org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationDFSException` - if the closing of the writer fails (for example due to HDFS quota being exceeded)

    - #### write

          public abstract void write(AggregatedLogFormat.LogKey logKey,
                                     org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat.LogValue logValue)
                              throws IOException

      Write the log content.
      Parameters:
        `logKey` - the log key
        `logValue` - the log content
      Throws:
        `IOException` - if fails to write the logs

    - #### postWrite

          public abstract void postWrite(org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileControllerContext record)
                                  throws Exception

      Operations needed after write the log content.
      Parameters:
        `record` - the `LogAggregationFileControllerContext`
      Throws:
        `Exception` - if anything fails

    - #### closePrintStream

          protected void closePrintStream(OutputStream out)

    - #### readAggregatedLogs

          public abstract boolean readAggregatedLogs(org.apache.hadoop.yarn.logaggregation.ContainerLogsRequest logRequest,
                                                     OutputStream os)
                                              throws IOException

      Output container log.
      Parameters:
        `logRequest` - `ContainerLogsRequest`
        `os` - the output stream
      Returns:
        true if we can read the aggregated logs successfully
      Throws:
        `IOException` - if we can not access the log file.

    - #### readAggregatedLogsMeta

          public abstract List<org.apache.hadoop.yarn.logaggregation.ContainerLogMeta> readAggregatedLogsMeta(org.apache.hadoop.yarn.logaggregation.ContainerLogsRequest logRequest)
                                                                                                       throws IOException

      Return a list of `ContainerLogMeta` for an application from Remote FileSystem.
      Parameters:
        `logRequest` - `ContainerLogsRequest`
      Returns:
        a list of `ContainerLogMeta`
      Throws:
        `IOException` - if there is no available log file

    - #### renderAggregatedLogsBlock

          public abstract void renderAggregatedLogsBlock(org.apache.hadoop.yarn.webapp.view.HtmlBlock.Block html,
                                                         org.apache.hadoop.yarn.webapp.View.ViewContext context)

      Render Aggregated Logs block.
      Parameters:
        `html` - the html
        `context` - the ViewContext

    - #### getApplicationOwner

          public abstract String getApplicationOwner(Path aggregatedLogPath,
                                                     ApplicationId appId)
                                              throws IOException

      Returns the owner of the application.
      Parameters:
        `aggregatedLogPath` - the aggregatedLog path
        `appId` - the ApplicationId
      Returns:
        the application owner
      Throws:
        `IOException` - if we can not get the application owner

    - #### getApplicationAcls

          public abstract Map<ApplicationAccessType,String> getApplicationAcls(Path aggregatedLogPath,
                                                                               ApplicationId appId)
                                                                        throws IOException

      Returns ACLs for the application. An empty map is returned if no ACLs are found.
      Parameters:
        `aggregatedLogPath` - the aggregatedLog path.
        `appId` - the ApplicationId
      Returns:
        a map of the Application ACLs.
      Throws:
        `IOException` - if we can not get the application acls

    - #### verifyAndCreateRemoteLogDir

          public void verifyAndCreateRemoteLogDir()

      Verify and create the remote log directory.

    - #### createAppDir

          public void createAppDir(String user,
                                   ApplicationId appId,
                                   UserGroupInformation userUgi)

      Create remote Application directory for log aggregation.
      Parameters:
        `user` - the user
        `appId` - the application ID
        `userUgi` - the UGI

    - #### getFileSystem

          protected FileSystem getFileSystem(Configuration conf)
                                      throws IOException

      Throws:
        `IOException`

    - #### createDir

          protected void createDir(FileSystem fs,
                                   Path path,
                                   FsPermission fsPerm)
                            throws IOException

      Throws:
        `IOException`

    - #### checkExists

          protected boolean checkExists(FileSystem fs,
                                        Path path,
                                        FsPermission fsPerm)
                                 throws IOException

      Throws:
        `IOException`

    - #### getRemoteNodeLogFileForApp

          public Path getRemoteNodeLogFileForApp(ApplicationId appId,
                                                 String user,
                                                 NodeId nodeId)

      Get the remote aggregated log path.
      Parameters:
        `appId` - the ApplicationId
        `user` - the Application Owner
        `nodeId` - the NodeManager Id
      Returns:
        the remote aggregated log path

    - #### getRemoteAppLogDir

          public Path getRemoteAppLogDir(ApplicationId appId,
                                         String appOwner)
                                  throws IOException

      Get the remote application directory for log aggregation.
      Parameters:
        `appId` - the Application ID
        `appOwner` - the Application Owner
      Returns:
        the remote application directory
      Throws:
        `IOException` - if can not find the remote application directory

    - #### cleanOldLogs

          protected void cleanOldLogs(Path remoteNodeLogFileForApp,
                                      NodeId nodeId,
                                      UserGroupInformation userUgi)

    - #### aggregatedLogSuffix

          protected String aggregatedLogSuffix(String fileName)

      Create the aggregated log suffix. The LogAggregationFileController should call this to get the suffix and append the suffix to the end of each log. This would keep the aggregated log format consistent.
      Parameters:
        `fileName` - the File Name
      Returns:
        the aggregated log suffix String

    - #### isFsSupportsChmod

          public boolean isFsSupportsChmod()

    - #### belongsToAppAttempt

          protected boolean belongsToAppAttempt(ApplicationAttemptId appAttemptId,
                                                String containerIdStr)

Copyright © 2022 Apache Software Foundation. All rights reserved.