org.apache.hadoop.mapreduce.v2.app.rm.preemption

## Class CheckpointAMPreemptionPolicy

- java.lang.Object
- - org.apache.hadoop.mapreduce.v2.app.rm.preemption.CheckpointAMPreemptionPolicy

- All Implemented Interfaces:
    AMPreemptionPolicy

      public class CheckpointAMPreemptionPolicy
      extends Object
      implements AMPreemptionPolicy

  This policy works in combination with an implementation of task checkpointing. It computes the tasks to be preempted in response to the RM request for preemption. For strict requests, it maps containers to corresponding tasks; for fungible requests, it attempts to pick the best containers to preempt (reducers in reverse allocation order). The TaskAttemptListener will interrogate this policy when handling a task heartbeat to check whether the task should be preempted or not. When handling fungible requests, the policy discount the RM ask by the amount of currently in-flight preemptions (i.e., tasks that are checkpointing). This class it is also used to maintain the list of checkpoints for existing tasks. Centralizing this functionality here, allows us to have visibility on preemption and checkpoints in a single location, thus coordinating preemption and checkpoint management decisions in a single policy.

- - ### Nested Class Summary

    - ### Nested classes/interfaces inherited from interface org.apache.hadoop.mapreduce.v2.app.rm.preemption.AMPreemptionPolicy

      `AMPreemptionPolicy.Context`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `CheckpointAMPreemptionPolicy()`

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `org.apache.hadoop.mapreduce.checkpoint.TaskCheckpointID` | `getCheckpointID(org.apache.hadoop.mapreduce.v2.api.records.TaskId taskId)` Method to retrieve the latest checkpoint for a given `TaskId`
    `void` | `handleCompletedContainer(org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptId attemptID)` Callback informing the policy of containers exiting cleanly.
    `void` | `handleFailedContainer(org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptId attemptID)` Callback informing the policy of containers exiting with a failure.
    `void` | `init(AppContext context)`
    `boolean` | `isPreempted(org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptId yarnAttemptID)` This method is invoked by components interested to learn whether a certain task is being preempted.
    `void` | `preempt(AMPreemptionPolicy.Context ctxt, org.apache.hadoop.yarn.api.records.PreemptionMessage preemptionRequests)` Callback informing the policy of ResourceManager.
    `void` | `reportSuccessfulPreemption(org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptId taskAttemptID)` This method is used to report to the policy that a certain task has been successfully preempted (for bookeeping, counters, etc..)
    `void` | `setCheckpointID(org.apache.hadoop.mapreduce.v2.api.records.TaskId taskId, org.apache.hadoop.mapreduce.checkpoint.TaskCheckpointID cid)` Method to store the latest `CheckpointID` for a given `TaskId`.

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Constructor Detail

    - #### CheckpointAMPreemptionPolicy

          public CheckpointAMPreemptionPolicy()

  - ### Method Detail

    - #### init

          public void init(AppContext context)

      Specified by:
        `init` in interface `AMPreemptionPolicy`

    - #### preempt

          public void preempt(AMPreemptionPolicy.Context ctxt,
                              org.apache.hadoop.yarn.api.records.PreemptionMessage preemptionRequests)

      Description copied from interface: `AMPreemptionPolicy`
      Callback informing the policy of ResourceManager. requests for resources to return to the cluster. The policy may take arbitrary action to satisfy requests by checkpointing task state, returning containers, or ignoring requests. The RM may elect to enforce these requests by forcibly killing containers not returned after some duration.
      Specified by:
        `preempt` in interface `AMPreemptionPolicy`
      Parameters:
        `ctxt` - Handle to the current state of running containers
        `preemptionRequests` - Request from RM for resources to return.

    - #### handleFailedContainer

          public void handleFailedContainer(org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptId attemptID)

      Description copied from interface: `AMPreemptionPolicy`
      Callback informing the policy of containers exiting with a failure. This allows the policy to implemnt cleanup/compensating actions.
      Specified by:
        `handleFailedContainer` in interface `AMPreemptionPolicy`
      Parameters:
        `attemptID` - Task attempt that failed

    - #### handleCompletedContainer

          public void handleCompletedContainer(org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptId attemptID)

      Description copied from interface: `AMPreemptionPolicy`
      Callback informing the policy of containers exiting cleanly. This is reported to the policy for bookeeping purposes.
      Specified by:
        `handleCompletedContainer` in interface `AMPreemptionPolicy`
      Parameters:
        `attemptID` - Task attempt that completed

    - #### isPreempted

          public boolean isPreempted(org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptId yarnAttemptID)

      Description copied from interface: `AMPreemptionPolicy`
      This method is invoked by components interested to learn whether a certain task is being preempted.
      Specified by:
        `isPreempted` in interface `AMPreemptionPolicy`
      Parameters:
        `yarnAttemptID` - Task attempt to query
      Returns:
        true if this attempt is being preempted

    - #### reportSuccessfulPreemption

          public void reportSuccessfulPreemption(org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptId taskAttemptID)

      Description copied from interface: `AMPreemptionPolicy`
      This method is used to report to the policy that a certain task has been successfully preempted (for bookeeping, counters, etc..)
      Specified by:
        `reportSuccessfulPreemption` in interface `AMPreemptionPolicy`
      Parameters:
        `taskAttemptID` - Task attempt that preempted

    - #### getCheckpointID

          public org.apache.hadoop.mapreduce.checkpoint.TaskCheckpointID getCheckpointID(org.apache.hadoop.mapreduce.v2.api.records.TaskId taskId)

      Description copied from interface: `AMPreemptionPolicy`
      Method to retrieve the latest checkpoint for a given `TaskId`
      Specified by:
        `getCheckpointID` in interface `AMPreemptionPolicy`
      Parameters:
        `taskId` - TaskID
      Returns:
        CheckpointID associated with this task or null

    - #### setCheckpointID

          public void setCheckpointID(org.apache.hadoop.mapreduce.v2.api.records.TaskId taskId,
                                      org.apache.hadoop.mapreduce.checkpoint.TaskCheckpointID cid)

      Description copied from interface: `AMPreemptionPolicy`
      Method to store the latest `CheckpointID` for a given `TaskId`. Assigning a null is akin to remove all previous checkpoints for this task.
      Specified by:
        `setCheckpointID` in interface `AMPreemptionPolicy`
      Parameters:
        `taskId` - TaskID
        `cid` - Checkpoint to assign or `null` to remove it.

Copyright © 2008–2022 Apache Software Foundation. All rights reserved.