com.twitter.summingbird

Summer

case class Summer[P <: Platform[P], K, V](producer: Producer[P, (K, V)], store: P.Store[K, V], semigroup: Semigroup[V]) extends KeyedProducer[P, K, (Option[V], V)] with TailProducer[P, (K, (Option[V], V))] with Product with Serializable

Source
Producer.scala
Linear Supertypes
Serializable, Serializable, Product, Equals, TailProducer[P, (K, (Option[V], V))], KeyedProducer[P, K, (Option[V], V)], Producer[P, (K, (Option[V], V))], AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. Summer
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. TailProducer
  7. KeyedProducer
  8. Producer
  9. AnyRef
  10. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new Summer(producer: Producer[P, (K, V)], store: P.Store[K, V], semigroup: Semigroup[V])

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. def ++[U >: (K, (Option[V], V))](r: Producer[P, U]): Producer[P, U]

    Exactly the same as merge.

    Exactly the same as merge. Here by analogy with the scala.collections API

    Definition Classes
    Producer
  5. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  6. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  7. def also[R](that: Producer[P, R]): Producer[P, R]

    Definition Classes
    TailProducer
  8. def also[R](that: TailProducer[P, R])(implicit ev: DummyImplicit): TailProducer[P, R]

    Ensure this is scheduled, but return something equivalent to the argument like the function par in Haskell.

    Ensure this is scheduled, but return something equivalent to the argument like the function par in Haskell. This can be used to combine two independent Producers in a way that ensures that the Platform will plan both into a single Plan.

    Definition Classes
    TailProducer
  9. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  10. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  11. def collect[U](fn: PartialFunction[(K, (Option[V], V)), U]): Producer[P, U]

    Prefer to flatMap for transforming a subset of items like optionMap but convenient with case syntax in scala prod.

    Prefer to flatMap for transforming a subset of items like optionMap but convenient with case syntax in scala prod.collect { case x if fn(x) => g(x) }

    Definition Classes
    Producer
  12. def collectKeys[K2](pf: PartialFunction[K, K2]): KeyedProducer[P, K2, (Option[V], V)]

    Builds a new KeyedProvider by applying a partial function to keys of elements of this one on which the function is defined.

    Builds a new KeyedProvider by applying a partial function to keys of elements of this one on which the function is defined.

    Definition Classes
    KeyedProducer
  13. def collectValues[V2](pf: PartialFunction[(Option[V], V), V2]): KeyedProducer[P, K, V2]

    Builds a new KeyedProvider by applying a partial function to values of elements of this one on which the function is defined.

    Builds a new KeyedProvider by applying a partial function to values of elements of this one on which the function is defined.

    Definition Classes
    KeyedProducer
  14. def either[U](other: Producer[P, U]): Producer[P, Either[(K, (Option[V], V)), U]]

    Merge a different type of Producer into a single stream

    Merge a different type of Producer into a single stream

    Definition Classes
    Producer
  15. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  16. def filter(fn: ((K, (Option[V], V))) ⇒ Boolean): Producer[P, (K, (Option[V], V))]

    Keep only the items that satisfy the fn

    Keep only the items that satisfy the fn

    Definition Classes
    Producer
  17. def filterKeys(pred: (K) ⇒ Boolean): KeyedProducer[P, K, (Option[V], V)]

    Prefer this to filter or flatMap/flatMapKeys if you are filtering.

    Prefer this to filter or flatMap/flatMapKeys if you are filtering. This may be optimized in the future with an intrinsic node in the Producer graph. We know this never increases the number of items, and we know it does not rekey the partition.

    Definition Classes
    KeyedProducer
  18. def filterValues(pred: ((Option[V], V)) ⇒ Boolean): KeyedProducer[P, K, (Option[V], V)]

    Prefer this to filter or flatMap/flatMapValues if you are filtering.

    Prefer this to filter or flatMap/flatMapValues if you are filtering. This may be optimized in the future with an intrinsic node in the Producer graph. We know this never increases the number of items, and we know it does not rekey the partition.

    Definition Classes
    KeyedProducer
  19. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  20. def flatMap[U](fn: ((K, (Option[V], V))) ⇒ TraversableOnce[U]): Producer[P, U]

    Only use this function if you may return more than 1 item sometimes.

    Only use this function if you may return more than 1 item sometimes. otherwise use collect or optionMap, which can be pushed up the graph

    Definition Classes
    Producer
  21. def flatMapKeys[K2](fn: (K) ⇒ TraversableOnce[K2]): KeyedProducer[P, K2, (Option[V], V)]

    Prefer to call this method to flatMap if you are expanding only keys.

    Prefer to call this method to flatMap if you are expanding only keys. It may trigger optimizations, that can significantly improve performance

    Definition Classes
    KeyedProducer
  22. def flatMapValues[U](fn: ((Option[V], V)) ⇒ TraversableOnce[U]): KeyedProducer[P, K, U]

    Prefer this to a raw map as this may be optimized to avoid a key reshuffle

    Prefer this to a raw map as this may be optimized to avoid a key reshuffle

    Definition Classes
    KeyedProducer
  23. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  24. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  25. def keys: Producer[P, K]

    Return just the keys

    Return just the keys

    Definition Classes
    KeyedProducer
  26. def leftJoin[RightV](stream: KeyedProducer[P, K, RightV], buffer: KeyedProducer.leftJoin.buffer._11.type.Service[K, RightV] with KeyedProducer.leftJoin.buffer._11.type.Sink[(K, RightV)] forSome {val _11: P}): KeyedProducer[P, K, ((Option[V], V), Option[RightV])]

    Do a windowed join on a stream.

    Do a windowed join on a stream. You need to provide a sink that manages the buffer. Offline, this might be a bounded HDFS partition. Online it might be a cache that evicts after a period of time.

    Definition Classes
    KeyedProducer
  27. def leftJoin[RightV](service: P.Service[K, RightV]): KeyedProducer[P, K, ((Option[V], V), Option[RightV])]

    Do a lookup/join on a service.

    Do a lookup/join on a service. This is how you trigger async computation is summingbird. Any remote API call, DB lookup, etc... happens here

    Definition Classes
    KeyedProducer
  28. def lookup[U >: (K, (Option[V], V)), V](service: P.Service[U, V]): KeyedProducer[P, U, Option[V]]

    This is identical to a certain leftJoin: map((_, ())).

    This is identical to a certain leftJoin: map((_, ())).leftJoin(srv).mapValues{case (_, v) => v} Useful when you are looking up values from say a stream of inputs, such as IDs.

    Definition Classes
    Producer
  29. def map[U](fn: ((K, (Option[V], V))) ⇒ U): Producer[P, U]

    Map each item to a new value

    Map each item to a new value

    Definition Classes
    Producer
  30. def mapKeys[K2](fn: (K) ⇒ K2): KeyedProducer[P, K2, (Option[V], V)]

    Prefer to call this method to flatMap/map if you are mapping only keys.

    Prefer to call this method to flatMap/map if you are mapping only keys. It may trigger optimizations, that can significantly improve performance

    Definition Classes
    KeyedProducer
  31. def mapValues[U](fn: ((Option[V], V)) ⇒ U): KeyedProducer[P, K, U]

    Prefer this to a raw map as this may be optimized to avoid a key reshuffle

    Prefer this to a raw map as this may be optimized to avoid a key reshuffle

    Definition Classes
    KeyedProducer
  32. def merge[U >: (K, (Option[V], V))](r: Producer[P, U]): Producer[P, U]

    Combine the output into one Producer

    Combine the output into one Producer

    Definition Classes
    Producer
  33. def name(id: String): TailProducer[P, (K, (Option[V], V))]

    Naming a node is so that you may give Options for that node that may change the run-time performance of the job (parameter tuning, etc.

    Naming a node is so that you may give Options for that node that may change the run-time performance of the job (parameter tuning, etc...)

    Definition Classes
    TailProducerProducer
  34. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  35. final def notify(): Unit

    Definition Classes
    AnyRef
  36. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  37. def optionMap[U](fn: ((K, (Option[V], V))) ⇒ Option[U]): Producer[P, U]

    Prefer this or collect to flatMap if you are always emitting 0 or 1 items

    Prefer this or collect to flatMap if you are always emitting 0 or 1 items

    Definition Classes
    Producer
  38. val producer: Producer[P, (K, V)]

  39. val semigroup: Semigroup[V]

  40. val store: P.Store[K, V]

  41. def sumByKey(store: P.Store[K, (Option[V], V)])(implicit semigroup: Semigroup[(Option[V], V)]): Summer[P, K, (Option[V], V)]

    emits a KeyedProducer with a value that is the store value, just BEFORE a merge, and the right is a new delta (which may include, depending on the Platform, Store and Options, more than a single aggregated item).

    emits a KeyedProducer with a value that is the store value, just BEFORE a merge, and the right is a new delta (which may include, depending on the Platform, Store and Options, more than a single aggregated item).

    so, the sequence out of this has the property that: (v0, vdelta1), (v0 + vdelta1, vdelta2), (v0 + vdelta1 + vdelta2, vdelta3), ...

    Definition Classes
    KeyedProducer
  42. def swap: KeyedProducer[P, (Option[V], V), K]

    Exchange values for keys

    Exchange values for keys

    Definition Classes
    KeyedProducer
  43. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  44. def values: Producer[P, (Option[V], V)]

    Keep only the values

    Keep only the values

    Definition Classes
    KeyedProducer
  45. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  46. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  47. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  48. def write[U >: (K, (Option[V], V))](sink: P.Sink[U]): TailProducer[P, (K, (Option[V], V))]

    Cause some side effect on the sink, but pass through the values so they can be consumed downstream

    Cause some side effect on the sink, but pass through the values so they can be consumed downstream

    Definition Classes
    Producer

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from TailProducer[P, (K, (Option[V], V))]

Inherited from KeyedProducer[P, K, (Option[V], V)]

Inherited from Producer[P, (K, (Option[V], V))]

Inherited from AnyRef

Inherited from Any

Ungrouped