com.twitter.summingbird.store

ClientStore

class ClientStore[K, V] extends ReadableStore[K, V]

The multiGet uses the "for" syntax internally to chain a bunch of computations from T => Future[U].

At a high level, the computation performed by the multiGet is the following:

- Look up the set of requested keys in the offlineStore. The offlineStore holds K -> (BatchID, V).

- For each key, use the returned BatchID and the current BatchID (calculated by the batcher) to generate a sequence of BatchIDs that the onlineStore is holding. The onlineStore holds (K, BatchID) -> V, so a join between this BatchID sequence and the K provides a keyset to use for a multiGet to the onlineStore.

- Perform this multiGet to the online store.

- PIVOT the BatchIDs out of the online store's key into a sequence in the value -- then a monoid merge with the offline store will append the offline value onto the beginning of the sequence of (BatchID, V).

- Finally, reduce this list by monoid-merging together all (BatchID, V) pairs. If any BatchID is missing from the sequence (if there are any holes, for example), that particular merged value's computation in the result will be a Future.exception vs a defined future. - Drop the final BatchID off of all successfully aggregated values (since this BatchID will be the current batch in all successful cases).

The onlineKeyFilter allows only a subset of the keys to be fetched from the realtime layer. This is a useful optimization in, for example, time series data, where many of the keys that are fetched are historical and therefore only need to be fetched from batch.

TODO (https://github.com/twitter/summingbird/issues/72): This filter needs to be generalized correctly, and is probably incorrect at the level of just supplying a boolean function. For example, in most cases a function K => T would help tie in batching logic more easily.

Source
ClientStore.scala
Linear Supertypes
ReadableStore[K, V], Closable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. ClientStore
  2. ReadableStore
  3. Closable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new ClientStore(offlineStore: ReadableStore[K, (BatchID, V)], onlineStore: ReadableStore[(K, BatchID), V], batcher: Batcher, batchesToKeep: Int, onlineKeyFilter: (K) ⇒ Boolean, collector: FutureCollector[(K, Iterable[BatchID])])(implicit arg0: Semigroup[V])

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  7. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  8. def close(time: Time): Future[Unit]

    Definition Classes
    ReadableStore → Closable
  9. def close(after: Duration): Future[Unit]

    Definition Classes
    Closable
  10. final def close(): Future[Unit]

    Definition Classes
    Closable
  11. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  12. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  13. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  14. def get(k: K): Future[Option[V]]

    Definition Classes
    ReadableStore
  15. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  16. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  17. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  18. def multiGet[K1 <: K](ks: Set[K1]): Map[K1, FOpt[V]]

    Definition Classes
    ClientStore → ReadableStore
  19. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  20. final def notify(): Unit

    Definition Classes
    AnyRef
  21. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  22. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  23. def toString(): String

    Definition Classes
    AnyRef → Any
  24. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  25. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  26. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from ReadableStore[K, V]

Inherited from Closable

Inherited from AnyRef

Inherited from Any

Ungrouped