com.twitter.summingbird.batch.state

HDFSCheckpointStore

class HDFSCheckpointStore extends CheckpointStore[Iterable[BatchID]]

Source
HDFSState.scala
Linear Supertypes
CheckpointStore[Iterable[BatchID]], AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. HDFSCheckpointStore
  2. CheckpointStore
  3. AnyRef
  4. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new HDFSCheckpointStore(config: Config)(implicit batcher: Batcher)

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  7. implicit val batcher: Batcher

    Definition Classes
    HDFSCheckpointStoreCheckpointStore
  8. def checkpointBatchStart(intersection: Intersection[InclusiveLower, ExclusiveUpper, Timestamp]): Iterable[BatchID]

    is called when the batch is started.

    is called when the batch is started. notice the intersection is not necessarily the same as [startBatch..endBatch), since there could be only part of the data available given a requested time range. It returns a batch token of typ T, which will be provided when the job is completed or failed for checkpointing.

    Definition Classes
    HDFSCheckpointStoreCheckpointStore
  9. def checkpointFailure(runningBatches: Iterable[BatchID], err: Throwable): Unit

    is called when the scalding job failed

    is called when the scalding job failed

    Definition Classes
    HDFSCheckpointStoreCheckpointStore
  10. def checkpointPlanFailure(err: Throwable): Nothing

    is called when the planning is failed to start the scalding job

    is called when the planning is failed to start the scalding job

    Definition Classes
    HDFSCheckpointStoreCheckpointStore
  11. def checkpointSuccessfulRun(runningBatches: Iterable[BatchID]): Unit

    is called when the batches are finished successfully

    is called when the batches are finished successfully

    Definition Classes
    HDFSCheckpointStoreCheckpointStore
  12. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  13. val config: Config

  14. val endBatch: ExclusiveUpper[BatchID]

    usually it's startBatch + numBatchesToRun, it's used to calculate requested time interval for the job

    usually it's startBatch + numBatchesToRun, it's used to calculate requested time interval for the job

    Definition Classes
    HDFSCheckpointStoreCheckpointStore
  15. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  16. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  17. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  18. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  19. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  20. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  21. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  22. final def notify(): Unit

    Definition Classes
    AnyRef
  23. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  24. val startBatch: InclusiveLower[BatchID]

    should get startBatch by checking the previous batch run, it's used to calculate requested time interval for the job

    should get startBatch by checking the previous batch run, it's used to calculate requested time interval for the job

    Definition Classes
    HDFSCheckpointStoreCheckpointStore
  25. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  26. def toString(): String

    Definition Classes
    AnyRef → Any
  27. lazy val versionedStore: FileVersionTracking

    Attributes
    protected
  28. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  29. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  30. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from CheckpointStore[Iterable[BatchID]]

Inherited from AnyRef

Inherited from Any

Ungrouped