##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Compatibility with 1.x
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Using CGroups
- Secure Containers
- Registry
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Submarine

- Index
- QuickStart
- Examples

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- OpenStack Swift

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# class `org.apache.hadoop.fs.FSDataOutputStreamBuilder`

- Invariants
- Implementation-agnostic parameters.
  - FSDataOutputStreamBuilder create()
  - FSDataOutputStreamBuilder append()
  - FSDataOutputStreamBuilder overwrite(boolean overwrite)
  - FSDataOutputStreamBuilder permission(FsPermission permission)
  - FSDataOutputStreamBuilder bufferSize(int bufSize)
  - FSDataOutputStreamBuilder replication(short replica)
  - FSDataOutputStreamBuilder blockSize(long size)
  - FSDataOutputStreamBuilder recursive()
  - FSDataOutputStreamBuilder progress(Progresable prog)
  - FSDataOutputStreamBuilder checksumOpt(ChecksumOpt chksumOpt)
  - Set optional or mandatory parameters
- HDFS-specific parameters.
  - FSDataOutpuStreamBuilder favoredNodes(InetSocketAddress[] nodes)
  - FSDataOutputStreamBuilder syncBlock()
  - FSDataOutputStreamBuilder lazyPersist()
  - FSDataOutputStreamBuilder newBlock()
  - FSDataOutputStreamBuilder noLocalWrite()
  - FSDataOutputStreamBuilder ecPolicyName()
  - FSDataOutputStreamBuilder replicate()
- Builder interface
  - FSDataOutputStream build()

Builder pattern for `FSDataOutputStream` and its subclasses. It is used to create a new file or open an existing file on `FileSystem` for write.

## Invariants

The `FSDataOutputStreamBuilder` interface does not validate parameters and modify the state of `FileSystem` until `build()` is invoked.

## Implementation-agnostic parameters.

### `FSDataOutputStreamBuilder create()`

Specify `FSDataOutputStreamBuilder` to create a file on `FileSystem`, equivalent to `CreateFlag#CREATE`.

### `FSDataOutputStreamBuilder append()`

Specify `FSDataOutputStreamBuilder` to append to an existing file on `FileSystem`, equivalent to `CreateFlag#APPEND`.

### `FSDataOutputStreamBuilder overwrite(boolean overwrite)`

Specify `FSDataOutputStreamBuilder` to overwrite an existing file or not. If giving `overwrite==true`, it truncates an existing file, equivalent to `CreateFlag#OVERWITE`.

### `FSDataOutputStreamBuilder permission(FsPermission permission)`

Set permission for the file.

### `FSDataOutputStreamBuilder bufferSize(int bufSize)`

Set the size of the buffer to be used.

### `FSDataOutputStreamBuilder replication(short replica)`

Set the replication factor.

### `FSDataOutputStreamBuilder blockSize(long size)`

Set block size in bytes.

### `FSDataOutputStreamBuilder recursive()`

Create parent directories if they do not exist.

### `FSDataOutputStreamBuilder progress(Progresable prog)`

Set the facility of reporting progress.

### `FSDataOutputStreamBuilder checksumOpt(ChecksumOpt chksumOpt)`

Set checksum opt.

### Set optional or mandatory parameters

    FSDataOutputStreamBuilder opt(String key, ...)
    FSDataOutputStreamBuilder must(String key, ...)

Set optional or mandatory parameters to the builder. Using `opt()` or `must()`, client can specify FS-specific parameters without inspecting the concrete type of `FileSystem`.

    // Don't
    if (fs instanceof FooFileSystem) {
        FooFileSystem fs = (FooFileSystem) fs;
        out = dfs.createFile(path)
            .optionA()
            .optionB("value")
            .cache()
            .build()
    } else if (fs instanceof BarFileSystem) {
        ...
    }

    // Do
    out = fs.createFile(path)
        .permission(perm)
        .bufferSize(bufSize)
        .opt("foofs:option.a", true)
        .opt("foofs:option.b", "value")
        .opt("barfs:cache", true)
        .must("foofs:cache", true)
        .must("barfs:cache-size", 256 * 1024 * 1024)
        .build();

#### Implementation Notes

The concrete `FileSystem` and/or `FSDataOutputStreamBuilder` implementation MUST verify that implementation-agnostic parameters (i.e., “syncable”) or implementation-specific parameters (i.e., “foofs:cache”) are supported. `FileSystem` will satisfy optional parameters (via `opt(key, ...)`) on best effort. If the mandatory parameters (via `must(key, ...)`) can not be satisfied in the `FileSystem`, `IllegalArgumentException` should be thrown in `build()`.

The behavior of resolving the conflicts between the parameters set by builder methods (i.e., `bufferSize()`) and `opt()`/`must()` is undefined.

## HDFS-specific parameters.

`HdfsDataOutputStreamBuilder extends FSDataOutputStreamBuilder` provides additional HDFS-specific parameters, for further customize file creation / append behavior.

### `FSDataOutpuStreamBuilder favoredNodes(InetSocketAddress[] nodes)`

Set favored DataNodes for new blocks.

### `FSDataOutputStreamBuilder syncBlock()`

Force closed blocks to the disk device. See `CreateFlag#SYNC_BLOCK`

### `FSDataOutputStreamBuilder lazyPersist()`

Create the block on transient storage if possible.

### `FSDataOutputStreamBuilder newBlock()`

Append data to a new block instead of the end of the last partial block.

### `FSDataOutputStreamBuilder noLocalWrite()`

Advise that a block replica NOT be written to the local DataNode.

### `FSDataOutputStreamBuilder ecPolicyName()`

Enforce the file to be a striped file with erasure coding policy ‘policyName’, no matter what its parent directory’s replication or erasure coding policy is.

### `FSDataOutputStreamBuilder replicate()`

Enforce the file to be a replicated file, no matter what its parent directory’s replication or erasure coding policy is.

## Builder interface

### `FSDataOutputStream build()`

Create a new file or append an existing file on the underlying `FileSystem`, and return `FSDataOutputStream` for write.

#### Preconditions

The following combinations of parameters are not supported:

    if APPEND|OVERWRITE: raise HadoopIllegalArgumentException
    if CREATE|APPEND|OVERWRITE: raise HadoopIllegalArgumentExdeption

`FileSystem` may reject the request for other reasons and throw `IOException`, see `FileSystem#create(path, ...)` and `FileSystem#append()`.

#### Postconditions

    FS' where :
       FS'.Files'[p] == []
       ancestors(p) is-subset-of FS'.Directories'

    result = FSDataOutputStream

The result is `FSDataOutputStream` to be used to write data to filesystem.