com.twitter.summingbird.scalding

LookupJoin

object LookupJoin extends Serializable

lookupJoin simulates the behavior of a realtime system attempting to leftJoin (K, V) pairs against some other value type (JoinedV) by performing realtime lookups on a key-value Store.

An example would join (K, V) pairs of (URL, Username) against a service of (URL, ImpressionCount). The result of this join would be a pipe of (ShortenedURL, (Username, Option[ImpressionCount])).

To simulate this behavior, lookupJoin accepts pipes of key-value pairs with an explicit time value T attached. T must have some sensical ordering. The semantics are, if one were to hit the right pipe's simulated realtime service at any time between T(tuple) T(tuple + 1), one would receive Some((K, JoinedV)(tuple)).

The entries in the left pipe's tuples have the following meaning:

T: The the time at which the (K, W) lookup occurred. K: the join key. W: the current value for the join key.

The right pipe's entries have the following meaning:

T: The time at which the "service" was fed an update K: the join K. V: value of the key at time T

Before the time T in the right pipe's very first entry, the simulated "service" will return None. After this time T, the right side will return None only if the key is absent, else, the service will return Some(joinedV).

Source
LookupJoin.scala
Linear Supertypes
Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. LookupJoin
  2. Serializable
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. def apply[T, K, V, JoinedV](left: TypedPipe[(T, (K, V))], right: TypedPipe[(T, (K, JoinedV))], reducers: Option[Int] = None)(implicit arg0: Ordering[T], arg1: Ordering[K]): TypedPipe[(T, (K, (V, Option[JoinedV])))]

    This is the "infinite history" join and always joins regardless of how much time is between the left and the right

  7. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  8. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  9. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  10. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  11. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  12. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  13. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  14. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  15. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  16. final def notify(): Unit

    Definition Classes
    AnyRef
  17. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  18. def rightSumming[T, K, V, JoinedV](left: TypedPipe[(T, (K, V))], right: TypedPipe[(T, (K, JoinedV))], reducers: Option[Int] = None)(implicit arg0: Ordering[T], arg1: Ordering[K], arg2: Semigroup[JoinedV]): TypedPipe[(T, (K, (V, Option[JoinedV])))]

    In this case, the right pipe is fed through a scanLeft doing a Semigroup.

    In this case, the right pipe is fed through a scanLeft doing a Semigroup.plus before joined to the left

  19. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  20. def toString(): String

    Definition Classes
    AnyRef → Any
  21. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  22. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  23. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. def withWindow[T, K, V, JoinedV](left: TypedPipe[(T, (K, V))], right: TypedPipe[(T, (K, JoinedV))], reducers: Option[Int] = None)(gate: (T, T) ⇒ Boolean)(implicit arg0: Ordering[T], arg1: Ordering[K]): TypedPipe[(T, (K, (V, Option[JoinedV])))]

    This ensures that gate(Tleft, Tright) == true, else the None is emitted as the joined value.

    This ensures that gate(Tleft, Tright) == true, else the None is emitted as the joined value. Useful for bounding the time of the join to a recent window

  25. def withWindowRightSumming[T, K, V, JoinedV](left: TypedPipe[(T, (K, V))], right: TypedPipe[(T, (K, JoinedV))], reducers: Option[Int] = None)(gate: (T, T) ⇒ Boolean)(implicit arg0: Ordering[T], arg1: Ordering[K], arg2: Semigroup[JoinedV]): TypedPipe[(T, (K, (V, Option[JoinedV])))]

    This ensures that gate(Tleft, Tright) == true, else the None is emitted as the joined value, and sums are only done as long as they they come within the gate interval as well

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped