# Package org.apache.avro.mapred

Run Hadoop MapReduce jobs over Avro data, with map and reduce functions written in Java.

See: Description

- Class Summary
  Class | Description
  --- | ---
  AvroAsTextInputFormat | An `InputFormat` for Avro data files, which converts each datum to string form in the input key.
  AvroCollector<T> | A collector for map and reduce output.
  AvroInputFormat<T> | An `InputFormat` for Avro data files.
  AvroJob | Setters to configure jobs for Avro data.
  AvroKey<T> | The wrapper of keys for jobs configured with `AvroJob` .
  AvroKeyComparator<T> | The `RawComparator` used by jobs configured with `AvroJob`.
  AvroMapper<IN,OUT> | A mapper for Avro data.
  AvroMultipleInputs | This class supports Avro-MapReduce jobs that have multiple input paths with a different `Schema` and `AvroMapper` for each path.
  AvroMultipleOutputs | The AvroMultipleOutputs class simplifies writing Avro output data to multiple outputs
  AvroOutputFormat<T> | An `OutputFormat` for Avro data files.
  AvroRecordReader<T> | An `RecordReader` for Avro data files.
  AvroReducer<K,V,OUT> | A reducer for Avro data.
  AvroSerialization<T> | The `Serialization` used by jobs configured with `AvroJob`.
  AvroTextOutputFormat<K,V> | The equivalent of `TextOutputFormat` for writing to Avro Data Files with a `"bytes"` schema.
  AvroUtf8InputFormat | An `InputFormat` for text files.
  AvroValue<T> | The wrapper of values for jobs configured with `AvroJob` .
  AvroWrapper<T> | The wrapper of data for jobs configured with `AvroJob` .
  FsInput | Adapt an `FSDataInputStream` to `SeekableInput`.
  Pair<K,V> | A key/value pair.
  SequenceFileInputFormat<K,V> | An `InputFormat` for sequence files.
  SequenceFileReader<K,V> | A `FileReader` for sequence files.
  SequenceFileRecordReader<K,V> | A `RecordReader` for sequence files.

## Package org.apache.avro.mapred Description

Run Hadoop MapReduce jobs over Avro data, with map and reduce functions written in Java.

Avro data files do not contain key/value pairs as expected by Hadoop's MapReduce API, but rather just a sequence of values. Thus we provide here a layer on top of Hadoop's MapReduce API.

In all cases, input and output paths are set and jobs are submitted as with standard Hadoop jobs:

- Specify input files with `FileInputFormat.setInputPaths(org.apache.hadoop.mapred.JobConf, java.lang.String)`
- Specify an output directory with `FileOutputFormat.setOutputPath(org.apache.hadoop.mapred.JobConf, org.apache.hadoop.fs.Path)`
- Run your job with `JobClient.runJob(org.apache.hadoop.mapred.JobConf)`

For jobs whose input and output are Avro data files:

- Call `AvroJob.setInputSchema(org.apache.hadoop.mapred.JobConf, org.apache.avro.Schema)` and `AvroJob.setOutputSchema(org.apache.hadoop.mapred.JobConf, org.apache.avro.Schema)` with your job's input and output schemas.
- Subclass `AvroMapper` and specify this as your job's mapper with `AvroJob.setMapperClass(org.apache.hadoop.mapred.JobConf, java.lang.Class<? extends org.apache.avro.mapred.AvroMapper>)`
- Subclass `AvroReducer` and specify this as your job's reducer and perhaps combiner, with `AvroJob.setReducerClass(org.apache.hadoop.mapred.JobConf, java.lang.Class<? extends org.apache.avro.mapred.AvroReducer>)` and `AvroJob.setCombinerClass(org.apache.hadoop.mapred.JobConf, java.lang.Class<? extends org.apache.avro.mapred.AvroReducer>)`

For jobs whose input is an Avro data file and which use an `AvroMapper`, but whose reducer is a non-Avro `Reducer` and whose output is a non-Avro format:

- Call `AvroJob.setInputSchema(org.apache.hadoop.mapred.JobConf, org.apache.avro.Schema)` with your job's input schema.
- Subclass `AvroMapper` and specify this as your job's mapper with `AvroJob.setMapperClass(org.apache.hadoop.mapred.JobConf, java.lang.Class<? extends org.apache.avro.mapred.AvroMapper>)`
- Implement `Reducer` and specify your job's reducer with `JobConf.setReducerClass(java.lang.Class<? extends org.apache.hadoop.mapred.Reducer>)`. The input key and value types should be `AvroKey` and `AvroValue`.
- Optionally implement `Reducer` and specify your job's combiner with `JobConf.setCombinerClass(java.lang.Class<? extends org.apache.hadoop.mapred.Reducer>)`. You will be unable to re-use the same Reducer class as the Combiner, as the Combiner will need input and output key to be `AvroKey`, and input and output value to be `AvroValue`.
- Specify your job's output key and value types `JobConf.setOutputKeyClass(java.lang.Class<?>)` and `JobConf.setOutputValueClass(java.lang.Class<?>)`.
- Specify your job's output format `JobConf.setOutputFormat(java.lang.Class<? extends org.apache.hadoop.mapred.OutputFormat>)`.

For jobs whose input is non-Avro data file and which use a non-Avro `Mapper`, but whose reducer is an `AvroReducer` and whose output is an Avro data file:

- Set your input file format with `JobConf.setInputFormat(java.lang.Class<? extends org.apache.hadoop.mapred.InputFormat>)`.
- Implement `Mapper` and specify your job's mapper with `JobConf.setMapperClass(java.lang.Class<? extends org.apache.hadoop.mapred.Mapper>)`. The output key and value type should be `AvroKey` and `AvroValue`.
- Subclass `AvroReducer` and specify this as your job's reducer and perhaps combiner, with `AvroJob.setReducerClass(org.apache.hadoop.mapred.JobConf, java.lang.Class<? extends org.apache.avro.mapred.AvroReducer>)` and `AvroJob.setCombinerClass(org.apache.hadoop.mapred.JobConf, java.lang.Class<? extends org.apache.avro.mapred.AvroReducer>)`
- Call `AvroJob.setOutputSchema(org.apache.hadoop.mapred.JobConf, org.apache.avro.Schema)` with your job's output schema.

For jobs whose input is non-Avro data file and which use a non-Avro `Mapper` and no reducer, i.e., a map-only job:

- Set your input file format with `JobConf.setInputFormat(java.lang.Class<? extends org.apache.hadoop.mapred.InputFormat>)`.
- Implement `Mapper` and specify your job's mapper with `JobConf.setMapperClass(java.lang.Class<? extends org.apache.hadoop.mapred.Mapper>)`. The output key and value type should be `AvroWrapper` and `NullWritable`.
- Call `JobConf.setNumReduceTasks(int)` with zero.
- Call `AvroJob.setOutputSchema(org.apache.hadoop.mapred.JobConf, org.apache.avro.Schema)` with your job's output schema.

Copyright © 2009–2024 The Apache Software Foundation. All rights reserved.