reducer_estimation

Type Members

1. trait BasicRuntimeReducerEstimator extends HistoryReducerEstimator

   

2. case class FallbackEstimator(first: ReducerEstimator, fallback: ReducerEstimator) extends ReducerEstimator with Product with Serializable

   

3. final case class FlowStepHistory(keys: FlowStepKeys, submitTime: Long, launchTime: Long, finishTime: Long, totalMaps: Long, totalReduces: Long, finishedMaps: Long, finishedReduces: Long, failedMaps: Long, failedReduces: Long, mapFileBytesRead: Long, mapFileBytesWritten: Long, reduceFileBytesRead: Long, hdfsBytesRead: Long, hdfsBytesWritten: Long, mapperTimeMillis: Long, reducerTimeMillis: Long, reduceShuffleBytes: Long, cost: Double, tasks: Seq[Task]) extends Product with Serializable

   

   Info about a prior FlowStep, provided by implementers of HistoryService

4. final case class FlowStepKeys(jobName: String, user: String, priority: String, status: String, version: String, queue: String) extends Product with Serializable

   

5. case class FlowStrategyInfo(flow: Flow[JobConf], predecessorSteps: Seq[FlowStep[JobConf]], step: FlowStep[JobConf]) extends Product with Serializable

   

6. trait HistoryReducerEstimator extends ReducerEstimator

   

7. trait HistoryService extends AnyRef

   

   Provider of information about prior runs.

8. trait InputScaledRuntimeReducerEstimator extends HistoryReducerEstimator

   

9. class InputSizeReducerEstimator extends ReducerEstimator

   

   Estimator that uses the input size and a fixed "bytesPerReducer" target.

   Estimator that uses the input size and a fixed "bytesPerReducer" target.

   Bytes per reducer can be configured with configuration parameter, defaults to 4 GB.

10. abstract class RatioBasedEstimator extends ReducerEstimator

    

11. trait ReducerEstimator extends AnyRef

    

12. trait RuntimeEstimationScheme extends AnyRef

    

    Estimator that uses the input size and a fixed "bytesPerReducer" target.

    Estimator that uses the input size and a fixed "bytesPerReducer" target.

    Bytes per reducer can be configured with configuration parameter, defaults to 1 GB.

13. trait RuntimeReducerEstimator extends HistoryReducerEstimator

    

14. final case class Task(taskType: String, status: String, startTime: Long, finishTime: Long) extends Product with Serializable