org.apache.hadoop.yarn.client.api.async

## Class AMRMClientAsync<T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest>

- java.lang.Object
- - org.apache.hadoop.service.AbstractService
  - - org.apache.hadoop.yarn.client.api.async.AMRMClientAsync<T>

- All Implemented Interfaces:
    Closeable, AutoCloseable, Service

      @InterfaceAudience.Public
       @InterfaceStability.Stable
      public abstract class AMRMClientAsync<T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest>
      extends AbstractService

  `AMRMClientAsync` handles communication with the ResourceManager and provides asynchronous updates on events such as container allocations and completions. It contains a thread that sends periodic heartbeats to the ResourceManager. It should be used by implementing a CallbackHandler:

       class MyCallbackHandler extends AMRMClientAsync.AbstractCallbackHandler {
         public void onContainersAllocated(List<Container> containers) {
           [run tasks on the containers]
         }

         public void onContainersUpdated(List<Container> containers) {
           [determine if resource allocation of containers have been increased in
            the ResourceManager, and if so, inform the NodeManagers to increase the
            resource monitor/enforcement on the containers]
         }

         public void onContainersCompleted(List<ContainerStatus> statuses) {
           [update progress, check whether app is done]
         }

         public void onNodesUpdated(List<NodeReport> updated) {}

         public void onReboot() {}
       }

  The client's lifecycle should be managed similarly to the following:

       AMRMClientAsync asyncClient =
           createAMRMClientAsync(appAttId, 1000, new MyCallbackhandler());
       asyncClient.init(conf);
       asyncClient.start();
       RegisterApplicationMasterResponse response = asyncClient
          .registerApplicationMaster(appMasterHostname, appMasterRpcPort,
             appMasterTrackingUrl);
       asyncClient.addContainerRequest(containerRequest);
       [... wait for application to complete]
       asyncClient.unregisterApplicationMaster(status, appMsg, trackingUrl);
       asyncClient.stop();

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `protected AMRMClient<T>` | `client`
    `protected org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler` | `handler`
    `protected AtomicInteger` | `heartbeatIntervalMs`

  - ### Constructor Summary

    Constructors
    Modifier | Constructor and Description
    --- | ---
    `protected` | `AMRMClientAsync(AMRMClient<T> client, int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.AbstractCallbackHandler callbackHandler)`
    `protected` | `AMRMClientAsync(AMRMClient<T> client, int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler callbackHandler)` Deprecated.
    `protected` | `AMRMClientAsync(int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.AbstractCallbackHandler callbackHandler)`
    `protected` | `AMRMClientAsync(int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler callbackHandler)` Deprecated.

  - ### Method Summary

    All Methods Static Methods Instance Methods Abstract Methods Concrete Methods Deprecated Methods
    Modifier and Type | Method and Description
    --- | ---
    `abstract void` | `addContainerRequest(T req)` Request containers for resources before calling `allocate`
    `void` | `addSchedulingRequests(Collection<SchedulingRequest> schedulingRequests)` Add a Collection of SchedulingRequests.
    `static <T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest> AMRMClientAsync<T>` | `createAMRMClientAsync(AMRMClient<T> client, int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.AbstractCallbackHandler callbackHandler)` Create a new instance of AMRMClientAsync.
    `static <T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest> AMRMClientAsync<T>` | `createAMRMClientAsync(AMRMClient<T> client, int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler callbackHandler)` Deprecated. Use `createAMRMClientAsync(AMRMClient, int, AMRMClientAsync.AbstractCallbackHandler)` instead.
    `static <T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest> AMRMClientAsync<T>` | `createAMRMClientAsync(int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.AbstractCallbackHandler callbackHandler)` Create a new instance of AMRMClientAsync.
    `static <T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest> AMRMClientAsync<T>` | `createAMRMClientAsync(int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler callbackHandler)` Deprecated. Use `createAMRMClientAsync(int, AMRMClientAsync.AbstractCallbackHandler)` instead.
    `abstract Resource` | `getAvailableResources()` Get the currently available resources in the cluster.
    `abstract int` | `getClusterNodeCount()` Get the current number of nodes in the cluster.
    `Collection<T>` | `getMatchingRequests(long allocationRequestId)` Returns all matching ContainerRequests that match the given AllocationRequestId.
    `List<? extends Collection<T>>` | `getMatchingRequests(Priority priority, String resourceName, ExecutionType executionType, Resource capability)` Returns all matching ContainerRequests that match the given Priority, ResourceName, ExecutionType and Capability.
    `abstract List<? extends Collection<T>>` | `getMatchingRequests(Priority priority, String resourceName, Resource capability)`
    `org.apache.hadoop.yarn.client.api.TimelineV2Client` | `getRegisteredTimelineV2Client()` Get registered timeline client.
    `abstract RegisterApplicationMasterResponse` | `registerApplicationMaster(String appHostName, int appHostPort, String appTrackingUrl)` Registers this application master with the resource manager.
    `RegisterApplicationMasterResponse` | `registerApplicationMaster(String appHostName, int appHostPort, String appTrackingUrl, Map<Set<String>,PlacementConstraint> placementConstraints)` Register the application master.
    `void` | `registerTimelineV2Client(org.apache.hadoop.yarn.client.api.TimelineV2Client timelineClient)` Register TimelineClient to AMRMClient.
    `abstract void` | `releaseAssignedContainer(ContainerId containerId)` Release containers assigned by the Resource Manager.
    `abstract void` | `removeContainerRequest(T req)` Remove previous container request.
    `void` | `requestContainerResourceChange(Container container, Resource capability)` Deprecated. use `requestContainerUpdate(Container, UpdateContainerRequest)`
    `abstract void` | `requestContainerUpdate(Container container, UpdateContainerRequest updateContainerRequest)` Request a container update before calling `allocate`.
    `void` | `setHeartbeatInterval(int interval)`
    `abstract void` | `unregisterApplicationMaster(FinalApplicationStatus appStatus, String appMessage, String appTrackingUrl)` Unregister the application master.
    `abstract void` | `updateBlacklist(List<String> blacklistAdditions, List<String> blacklistRemovals)` Update application's blacklist with addition or removal resources.
    `void` | `updateTrackingUrl(String trackingUrl)` Update application's tracking url on next heartbeat.
    `void` | `waitFor(Supplier<Boolean> check)` Wait for `check` to return true for each 1000 ms.
    `void` | `waitFor(Supplier<Boolean> check, int checkEveryMillis)` Wait for `check` to return true for each `checkEveryMillis` ms.
    `void` | `waitFor(Supplier<Boolean> check, int checkEveryMillis, int logInterval)` Wait for `check` to return true for each `checkEveryMillis` ms.

    - ### Methods inherited from class org.apache.hadoop.service.AbstractService

      `close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, serviceInit, serviceStart, serviceStop, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

- - ### Field Detail

    - #### client

          protected final AMRMClient<T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest> client

    - #### handler

          protected final org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler handler

    - #### heartbeatIntervalMs

          protected final AtomicInteger heartbeatIntervalMs

  - ### Constructor Detail

    - #### AMRMClientAsync

          protected AMRMClientAsync(int intervalMs,
                                    org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.AbstractCallbackHandler callbackHandler)

    - #### AMRMClientAsync

          @InterfaceAudience.Private
          protected AMRMClientAsync(AMRMClient<T> client,
                                                               int intervalMs,
                                                               org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.AbstractCallbackHandler callbackHandler)

    - #### AMRMClientAsync

          @Deprecated
          protected AMRMClientAsync(int intervalMs,
                                                org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler callbackHandler)

      Deprecated.

    - #### AMRMClientAsync

          @InterfaceAudience.Private
           @Deprecated
          protected AMRMClientAsync(AMRMClient<T> client,
                                                                            int intervalMs,
                                                                            org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler callbackHandler)

      Deprecated.

  - ### Method Detail

    - #### createAMRMClientAsync

          public static <T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest> AMRMClientAsync<T> createAMRMClientAsync(int intervalMs,
                                                                                                                                           org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.AbstractCallbackHandler callbackHandler)

      Create a new instance of AMRMClientAsync.

      Parameters:
        `intervalMs` - heartbeat interval in milliseconds between AM and RM
        `callbackHandler` - callback handler that processes responses from the `ResourceManager`

    - #### createAMRMClientAsync

          public static <T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest> AMRMClientAsync<T> createAMRMClientAsync(AMRMClient<T> client,
                                                                                                                                           int intervalMs,
                                                                                                                                           org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.AbstractCallbackHandler callbackHandler)

      Create a new instance of AMRMClientAsync.

      Parameters:
        `client` - the AMRMClient instance
        `intervalMs` - heartbeat interval in milliseconds between AM and RM
        `callbackHandler` - callback handler that processes responses from the `ResourceManager`

    - #### createAMRMClientAsync

          @Deprecated
          public static <T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest> AMRMClientAsync<T> createAMRMClientAsync(int intervalMs,
                                                                                                                                                       org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler callbackHandler)

      Deprecated. Use `createAMRMClientAsync(int, AMRMClientAsync.AbstractCallbackHandler)` instead.

    - #### createAMRMClientAsync

          @Deprecated
          public static <T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest> AMRMClientAsync<T> createAMRMClientAsync(AMRMClient<T> client,
                                                                                                                                                       int intervalMs,
                                                                                                                                                       org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler callbackHandler)

      Deprecated. Use `createAMRMClientAsync(AMRMClient, int, AMRMClientAsync.AbstractCallbackHandler)` instead.

    - #### setHeartbeatInterval

          public void setHeartbeatInterval(int interval)

    - #### getMatchingRequests

          public abstract List<? extends Collection<T>> getMatchingRequests(Priority priority,
                                                                            String resourceName,
                                                                            Resource capability)

    - #### addSchedulingRequests

          @InterfaceAudience.Public
           @InterfaceStability.Unstable
          public void addSchedulingRequests(Collection<SchedulingRequest> schedulingRequests)

      Add a Collection of SchedulingRequests. The AMRMClient will ensure that all requests in the same batch are sent in the same allocate call.
      Parameters:
        `schedulingRequests` - Collection of Scheduling Requests.

    - #### getMatchingRequests

          public List<? extends Collection<T>> getMatchingRequests(Priority priority,
                                                                   String resourceName,
                                                                   ExecutionType executionType,
                                                                   Resource capability)

      Returns all matching ContainerRequests that match the given Priority, ResourceName, ExecutionType and Capability. NOTE: This matches only requests that were made by the client WITHOUT the allocationRequestId specified.
      Parameters:
        `priority` - Priority.
        `resourceName` - Location.
        `executionType` - ExecutionType.
        `capability` - Capability.
      Returns:
        All matching ContainerRequests

    - #### getMatchingRequests

          public Collection<T> getMatchingRequests(long allocationRequestId)

      Returns all matching ContainerRequests that match the given AllocationRequestId. NOTE: This matches only requests that were made by the client WITH the allocationRequestId specified.
      Parameters:
        `allocationRequestId` - AllocationRequestId.
      Returns:
        All matching ContainerRequests

    - #### registerApplicationMaster

          public abstract RegisterApplicationMasterResponse registerApplicationMaster(String appHostName,
                                                                                      int appHostPort,
                                                                                      String appTrackingUrl)
                                                                               throws YarnException,
                                                                                      IOException

      Registers this application master with the resource manager. On successful registration, starts the heartbeating thread.
      Throws:
        `YarnException`
        `IOException`

    - #### registerApplicationMaster

          @InterfaceAudience.Public
           @InterfaceStability.Unstable
          public RegisterApplicationMasterResponse registerApplicationMaster(String appHostName,
                                                                                                                                     int appHostPort,
                                                                                                                                     String appTrackingUrl,
                                                                                                                                     Map<Set<String>,PlacementConstraint> placementConstraints)
                                                                                                                              throws YarnException,
                                                                                                                                     IOException

      Register the application master. This must be called before any other interaction
      Parameters:
        `appHostName` - Name of the host on which master is running
        `appHostPort` - Port master is listening on
        `appTrackingUrl` - URL at which the master info can be seen
        `placementConstraints` - Placement Constraints mappings.
      Returns:
        `RegisterApplicationMasterResponse`
      Throws:
        `YarnException`
        `IOException`

    - #### unregisterApplicationMaster

          public abstract void unregisterApplicationMaster(FinalApplicationStatus appStatus,
                                                           String appMessage,
                                                           String appTrackingUrl)
                                                    throws YarnException,
                                                           IOException

      Unregister the application master. This must be called in the end.
      Parameters:
        `appStatus` - Success/Failure status of the master
        `appMessage` - Diagnostics message on failure
        `appTrackingUrl` - New URL to get master info
      Throws:
        `YarnException`
        `IOException`

    - #### addContainerRequest

          public abstract void addContainerRequest(T req)

      Request containers for resources before calling `allocate`
      Parameters:
        `req` - Resource request

    - #### removeContainerRequest

          public abstract void removeContainerRequest(T req)

      Remove previous container request. The previous container request may have already been sent to the ResourceManager. So even after the remove request the app must be prepared to receive an allocation for the previous request even after the remove request
      Parameters:
        `req` - Resource request

    - #### requestContainerResourceChange

          @Deprecated
          public void requestContainerResourceChange(Container container,
                                                                 Resource capability)

      Deprecated. use `requestContainerUpdate(Container, UpdateContainerRequest)`
      Request container resource change before calling `allocate`. Any previous pending resource change request of the same container will be removed. Application that calls this method is expected to maintain the `Container`s that are returned from previous successful allocations or resource changes. By passing in the existing container and a target resource capability to this method, the application requests the ResourceManager to change the existing resource allocation to the target resource allocation.
      Parameters:
        `container` - The container returned from the last successful resource allocation or resource change
        `capability` - The target resource capability of the container

    - #### requestContainerUpdate

          public abstract void requestContainerUpdate(Container container,
                                                      UpdateContainerRequest updateContainerRequest)

      Request a container update before calling `allocate`. Any previous pending update request of the same container will be removed.
      Parameters:
        `container` - The container returned from the last successful resource allocation or update
        `updateContainerRequest` - The `UpdateContainerRequest`.

    - #### releaseAssignedContainer

          public abstract void releaseAssignedContainer(ContainerId containerId)

      Release containers assigned by the Resource Manager. If the app cannot use the container or wants to give up the container then it can release them. The app needs to make new requests for the released resource capability if it still needs it. eg. it released non-local resources
      Parameters:
        `containerId` -

    - #### getAvailableResources

          public abstract Resource getAvailableResources()

      Get the currently available resources in the cluster. A valid value is available after a call to allocate has been made
      Returns:
        Currently available resources

    - #### getClusterNodeCount

          public abstract int getClusterNodeCount()

      Get the current number of nodes in the cluster. A valid values is available after a call to allocate has been made
      Returns:
        Current number of nodes in the cluster

    - #### registerTimelineV2Client

          public void registerTimelineV2Client(org.apache.hadoop.yarn.client.api.TimelineV2Client timelineClient)
                                        throws YarnException

      Register TimelineClient to AMRMClient.
      Parameters:
        `timelineClient` -
      Throws:
        `YarnException` - when this method is invoked even when ATS V2 is not configured.

    - #### getRegisteredTimelineV2Client

          public org.apache.hadoop.yarn.client.api.TimelineV2Client getRegisteredTimelineV2Client()

      Get registered timeline client.
      Returns:
        the registered timeline client

    - #### updateBlacklist

          public abstract void updateBlacklist(List<String> blacklistAdditions,
                                               List<String> blacklistRemovals)

      Update application's blacklist with addition or removal resources.
      Parameters:
        `blacklistAdditions` - list of resources which should be added to the application blacklist
        `blacklistRemovals` - list of resources which should be removed from the application blacklist

    - #### updateTrackingUrl

          @InterfaceAudience.Public
           @InterfaceStability.Unstable
          public void updateTrackingUrl(String trackingUrl)

      Update application's tracking url on next heartbeat.
      Parameters:
        `trackingUrl` - new tracking url for this application

    - #### waitFor

          public void waitFor(Supplier<Boolean> check)
                       throws InterruptedException

      Wait for `check` to return true for each 1000 ms. See also `waitFor(java.util.function.Supplier, int)` and `waitFor(java.util.function.Supplier, int, int)`
      Parameters:
        `check` - the condition for which it should wait
      Throws:
        `InterruptedException`

    - #### waitFor

          public void waitFor(Supplier<Boolean> check,
                              int checkEveryMillis)
                       throws InterruptedException

      Wait for `check` to return true for each `checkEveryMillis` ms. See also `waitFor(java.util.function.Supplier, int, int)`
      Parameters:
        `check` - user defined checker
        `checkEveryMillis` - interval to call `check`
      Throws:
        `InterruptedException`

    - #### waitFor

          public void waitFor(Supplier<Boolean> check,
                              int checkEveryMillis,
                              int logInterval)
                       throws InterruptedException

      Wait for `check` to return true for each `checkEveryMillis` ms. In the main loop, this method will log the message "waiting in main loop" for each `logInterval` times iteration to confirm the thread is alive.
      Parameters:
        `check` - user defined checker
        `checkEveryMillis` - interval to call `check`
        `logInterval` - interval to log for each
      Throws:
        `InterruptedException`

Copyright © 2022 Apache Software Foundation. All rights reserved.