scalding

Type Members

1. trait ExternalService[K, +V] extends Service[K, V]

   This represents a service that is *external* to the current job.

2. type FactoryInput = (Interval[Timestamp], Mode)

   The Platform recursively passes this input around to describe a step forward: requested input time span, and scalding Mode

3. type FailureReason = String

   These are printed/logged only when we can't make any progress

4. type FlowInput = (FlowDef, Mode)

   When it is time to run build the final flow, this is what scalding needs.

   When it is time to run build the final flow, this is what scalding needs. It is modified in the Reader[FlowInput, T]

5. case class FlowPlanException(errs: List[FailureReason]) extends Exception with Product with Serializable

   Puts a type on the final exception thrown by the scalding Planner

6. type FlowProducer[+T] = Reader[(FlowDef, Mode), T]

   This is a function that modifies a flow to return T generally T will be some kind of TypedPipe

7. type FlowToPipe[+T] = Reader[(FlowDef, Mode), TimedPipe[T]]

   We so commonly talk about producing TimedPipe we define this

8. sealed trait InternalService[K, +V] extends Service[K, V]

   This represents a join against data that is materialized by a store in the current job

9. type KeyValuePipe[+K, +V] = TypedPipe[(Timestamp, (K, V))]

10. class LocalIterableSource[+T] extends IterableSource[T]

11. class LoopState[T] extends WaitingState[T]

12. class Memo[T] extends Serializable

13. class MockMappable[T] extends scalding.Source with Mappable[T]

14. type PipeFactory[+T] = StateWithError[(Interval[Timestamp], Mode), List[FailureReason], FlowToPipe[T]]

    We are usually producing Pipes in the Planner

15. class PipeFactoryOps[+T] extends AnyRef

16. type PlannerOutput[+T] = StateWithError[(Interval[Timestamp], Mode), List[FailureReason], T]

    The recursive planner produces these objects which are Monads

17. class Scalding extends Platform[Scalding] with Serializable

18. case class ScaldingEnv(jobName: String, inargs: Array[String]) extends Env with Product with Serializable

19. trait ScaldingExecutionConfig extends ChillExecutionConfig[Scalding]

20. sealed trait Service[K, +V] extends Serializable

21. trait Sink[T] extends AnyRef

22. trait Store[K, V] extends Serializable

23. case class StoreIntermediateData[K, V](sink: Sink[(K, V)]) extends Serializable with Product with Serializable

24. case class StoreService[K, V](store: BatchedStore[K, V]) extends InternalService[K, V] with Store[K, V] with Product with Serializable

25. class TestService[K, V] extends BatchedService[K, V]

26. class TestSink[T] extends Sink[T]

    This is a test sink that assumes single threaded testing with cascading local mode

27. class TestStore[K, V] extends BatchedStore[K, V]

28. class TestStoreService[K, V] extends StoreService[K, V]

29. type TimedPipe[+T] = TypedPipe[(Timestamp, T)]

    How we represent the streams in scalding

30. type Try[+T] = Either[List[FailureReason], T]

31. case class WriteDot(filename: String) extends Product with Serializable

    Use this option to write the logical graph that cascading produces before Map/Reduce planning.

32. case class WriteStepsDot(filename: String) extends Product with Serializable

    Use this option to write map/reduce graph that cascading produces Use the job name as the key