com.twitter.summingbird.scalding

TestStore

class TestStore[K, V] extends BatchedStore[K, V]

Source
TestStore.scala
Linear Supertypes
BatchedStore[K, V], Store[K, V], Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. TestStore
  2. BatchedStore
  3. Store
  4. Serializable
  5. AnyRef
  6. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new TestStore(store: String, inBatcher: Batcher, initBatch: BatchID, initStore: Iterable[(K, V)], lastBatch: BatchID, pruning: PrunedSpace[(K, V)])(implicit ord: Ordering[K], tset: TupleSetter[(K, V)], tconv: TupleConverter[(K, V)])

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  7. val batcher: Batcher

    The batcher for this store

    The batcher for this store

    Definition Classes
    TestStoreBatchedStore
  8. val batches: Map[BatchID, Mappable[(K, V)]]

  9. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  10. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  11. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  12. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  13. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  14. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  15. val initBatch: BatchID

  16. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  17. def lastToIterable: Iterable[(K, V)]

  18. final def merge(delta: PipeFactory[(K, V)], sg: Semigroup[V], commutativity: Commutativity, reducers: Int): PipeFactory[(K, (Option[V], V))]

    instances of this trait MAY NOT change the logic here.

    instances of this trait MAY NOT change the logic here. This always follows the rule that we look for existing data (avoiding reading deltas in that case), then we fall back to the last checkpointed output by calling readLast. In that case, we compute the results by rolling forward

    Definition Classes
    BatchedStoreStore
  19. def mockFor(b: BatchID): Mappable[(K, V)]

  20. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  21. final def notify(): Unit

    Definition Classes
    AnyRef
  22. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  23. val ordering: Ordering[K]

    Definition Classes
    TestStoreBatchedStore
  24. def partialMerge[K1](delta: PipeFactory[(K1, V)], sg: Semigroup[V], commutativity: Commutativity): PipeFactory[(K1, V)]

    For each batch, collect up values with the same key on mapside before the keys are expanded.

    For each batch, collect up values with the same key on mapside before the keys are expanded.

    Definition Classes
    BatchedStoreStore
  25. final def planReadLast: PlannerOutput[(BatchID, FlowProducer[TypedPipe[(K, V)]])]

    This is the monadic version of readLast, returns the BatchID actually on disk

    This is the monadic version of readLast, returns the BatchID actually on disk

    Definition Classes
    BatchedStore
  26. val pruning: PrunedSpace[(K, V)]

    Override this to set up store pruning, by default, no (key,value) pairs are pruned.

    Override this to set up store pruning, by default, no (key,value) pairs are pruned. This is a house keeping function to permanently remove entries matching a criteria.

    Definition Classes
    TestStoreBatchedStore
  27. final def readAfterLastBatch[T](input: PipeFactory[T]): PlannerOutput[(BatchID, FlowProducer[TypedPipe[(K, V)]], FlowToPipe[T])]

    Reads the input data after the last batch written.

    Reads the input data after the last batch written.

    Returns: - the BatchID of the last batch written - the snapshot of the store just before this state - the data from this input covering all the time SINCE the last snapshot

    Definition Classes
    BatchedStore
  28. def readDeltaLog(delta: PipeFactory[(K, V)]): PipeFactory[(K, V)]

    This combines the current inputs along with the last checkpoint on disk to get a log of all deltas with a timestamp This is useful to leftJoin against a store.

    This combines the current inputs along with the last checkpoint on disk to get a log of all deltas with a timestamp This is useful to leftJoin against a store. TODO: This should not limit to batch boundaries, the batch store should handle only writing the data for full batches, but we can materialize more data if it is needed downstream. Note: the returned time interval NOT include the time of the snapshot data point (which is exactly 1 millisecond before the start of the interval).

    Definition Classes
    BatchedStore
  29. def readLast(exclusiveUB: BatchID, mode: Mode): Try[(BatchID, FlowProducer[TypedPipe[(K, V)]])]

    Get the most recent last batch and the ID (strictly less than the input ID) The "Last" is the stream with only the newest value for each key, within the batch combining the last from batchID and the deltas from batchID.

    Get the most recent last batch and the ID (strictly less than the input ID) The "Last" is the stream with only the newest value for each key, within the batch combining the last from batchID and the deltas from batchID.next you get the stream for batchID.next

    Definition Classes
    TestStoreBatchedStore
  30. def select(b: List[BatchID]): List[BatchID]

    Override select if you don't want to materialize every batch.

    Override select if you don't want to materialize every batch. Note that select MUST return a list containing the final batch in the supplied list; otherwise data would be lost.

    Definition Classes
    BatchedStore
  31. val sourceToBuffer: Map[scalding.Source, Buffer[Tuple]]

  32. def sumByBatches[K1, V](ins: TypedPipe[(Timestamp, (K1, V))], capturedBatcher: Batcher, commutativity: Commutativity)(implicit arg0: Semigroup[V]): TypedPipe[((K1, BatchID), (Timestamp, V))]

    Attributes
    protected
    Definition Classes
    BatchedStore
  33. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  34. final def timeSpanToBatches: PlannerOutput[List[BatchID]]

    This gives the batches needed to cover the requested input This will always be non-empty

    This gives the batches needed to cover the requested input This will always be non-empty

    Definition Classes
    BatchedStore
  35. def toString(): String

    Definition Classes
    AnyRef → Any
  36. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  37. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  38. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  39. def withInitialBatch(firstNonZero: BatchID): BatchedStore[K, V]

    For (firstNonZero - 1) we read empty.

    For (firstNonZero - 1) we read empty. For all before we error on read. For all later, we proxy On write, we throw if batchID is less than firstNonZero

    Definition Classes
    BatchedStore
  40. def writeLast(batchID: BatchID, lastVals: TypedPipe[(K, V)])(implicit flowDef: FlowDef, mode: Mode): Unit

    Instances may choose to write out the last or just compute it from the stream

    Instances may choose to write out the last or just compute it from the stream

    Definition Classes
    TestStoreBatchedStore
  41. var writtenBatches: Set[BatchID]

Inherited from BatchedStore[K, V]

Inherited from Store[K, V]

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped