com.twitter

summingbird

package summingbird

Visibility
  1. Public
  2. All

Type Members

  1. abstract class AbstractJob extends Serializable

  2. case class AlsoProducer[P <: Platform[P], +T, +R](ensure: TailProducer[P, T], result: Producer[P, R]) extends Producer[P, R] with Product with Serializable

    This is a special node that ensures that the first argument is planned, but produces values equivalent to the result.

  3. class AlsoTailProducer[P <: Platform[P], +T, +R] extends AlsoProducer[P, T, R] with TailProducer[P, R]

  4. case class Counter(group: Group, name: Name)(implicit jobID: JobId) extends Product with Serializable

  5. trait CounterIncrementor extends AnyRef

  6. case class Dependants[P <: Platform[P]](tail: Producer[P, Any]) extends DependantGraph[Producer[P, Any]] with Product with Serializable

    Producers are Directed Acyclic Graphs by the fact that they are immutable.

  7. abstract class Env extends Serializable

    Any environment data an AbstractJob may need.

  8. trait ExecutionConfig[P <: Platform[P]] extends AnyRef

  9. case class FlatMappedProducer[P <: Platform[P], T, +U](producer: Producer[P, T], fn: (T) ⇒ TraversableOnce[U]) extends Producer[P, U] with Product with Serializable

  10. abstract class FlatMapper[-T, +U] extends (T) ⇒ TraversableOnce[U]

  11. class FunctionFlatMapper[T, U] extends FlatMapper[T, U]

  12. final case class Group(getString: String) extends AnyVal with Product with Serializable

  13. case class IdentityKeyedProducer[P <: Platform[P], K, V](producer: Producer[P, (K, V)]) extends KeyedProducer[P, K, V] with Product with Serializable

  14. case class KeyFlatMappedProducer[P <: Platform[P], K, V, K2](producer: Producer[P, (K, V)], fn: (K) ⇒ TraversableOnce[K2]) extends KeyedProducer[P, K2, V] with Product with Serializable

  15. sealed trait KeyedProducer[P <: Platform[P], K, V] extends Producer[P, (K, V)]

    This has the methods on Key-Value streams.

  16. case class LeftJoinedProducer[P <: Platform[P], K, V, JoinedV](left: Producer[P, (K, V)], joined: P.Service[K, JoinedV]) extends KeyedProducer[P, K, (V, Option[JoinedV])] with Product with Serializable

  17. case class MergedProducer[P <: Platform[P], +T](left: Producer[P, T], right: Producer[P, T]) extends Producer[P, T] with Product with Serializable

  18. trait MutableStringConfig extends AnyRef

  19. final case class Name(getString: String) extends AnyVal with Product with Serializable

  20. case class NamedProducer[P <: Platform[P], +T](producer: Producer[P, T], id: String) extends Producer[P, T] with Product with Serializable

  21. case class OptionMappedProducer[P <: Platform[P], T, +U](producer: Producer[P, T], fn: (T) ⇒ Option[U]) extends Producer[P, U] with Product with Serializable

    Represents filters and maps which may be optimized differently Note that "option-mapping" is closed under composition and hence useful to call out

  22. class OptionalPlatform2[P1 <: Platform[P1], P2 <: Platform[P2]] extends Platform[OptionalPlatform2[P1, P2]]

    Platform capable of planning and executing at most 2 underlying platforms in parallel.

  23. case class OptionalUnzip2[P1 <: Platform[P1], P2 <: Platform[P2]]() extends Product with Serializable

  24. class Options extends AnyRef

  25. trait Platform[P <: Platform[P]] extends AnyRef

    Base trait for summingbird compilers.

  26. class Platform2[P1 <: Platform[P1], P2 <: Platform[P2]] extends Platform[Platform2[P1, P2]]

    Platform capable of planning and executing two underlying platforms in parallel.

  27. trait PlatformStatProvider extends AnyRef

  28. sealed trait Producer[P <: Platform[P], +T] extends AnyRef

    A Producer is a node in our tree, able to generate new items and have operations applied to it.

  29. trait ReadableMap extends AnyRef

  30. case class Source[P <: Platform[P], T](source: P.Source[T]) extends Producer[P, T] with Product with Serializable

    Wraps the sources of the given Platform

  31. case class Summer[P <: Platform[P], K, V](producer: Producer[P, (K, V)], store: P.Store[K, V], semigroup: Semigroup[V]) extends KeyedProducer[P, K, (Option[V], V)] with TailProducer[P, (K, (Option[V], V))] with Product with Serializable

  32. trait SummingbirdConfig extends AnyRef

  33. class TPNamedProducer[P <: Platform[P], +T] extends NamedProducer[P, T] with TailProducer[P, T]

  34. sealed trait TailProducer[P <: Platform[P], +T] extends Producer[P, T]

    Only TailProducers can be planned.

  35. class TestGraphs[P <: Platform[P], T, K, V] extends AnyRef

  36. trait TimeExtractor[T] extends Serializable

    This cannot be a subclass of function and use the pattern of implicit dependencies, since then you get an implicit function.

  37. case class Unzip2[P1 <: Platform[P1], P2 <: Platform[P2]]() extends Product with Serializable

    I tried to make Unzip2 an object with an apply method that took all three type parameters, but ran into issues:

  38. case class ValueFlatMappedProducer[P <: Platform[P], K, V, V2](producer: Producer[P, (K, V)], fn: (V) ⇒ TraversableOnce[V2]) extends KeyedProducer[P, K, V2] with Product with Serializable

  39. case class WrappingConfig(backingConfig: ReadableMap, updates: Map[String, AnyRef], removes: Set[String]) extends SummingbirdConfig with Product with Serializable

  40. case class WrittenProducer[P <: Platform[P], T, U >: T](producer: Producer[P, T], sink: P.Sink[U]) extends TailProducer[P, T] with Product with Serializable

Value Members

  1. object AbstractJob extends Serializable

    Base class for all Summingbird jobs; all summingbird jobs should extend AbstractJob.

  2. object Env extends Serializable

  3. object FlatMapper

  4. object JobCounters

  5. object MemoryArbitraries

  6. object Options

    intra-graph options.

  7. object Predef

    Aliases for types and objects commonly used in Summingbird jobs.

  8. object Producer

  9. object SummingbirdRuntimeStats

  10. object TestGraphGenerators

  11. object TestGraphs

    Helpful functions and test graphs designed to flex Summingbird planners.

  12. object TimeExtractor extends Serializable

    TimeExtractor is really just a function, but we've used a specialized type for implicit resolution and serializability.

  13. object WrappingConfig extends Serializable

  14. package batch

  15. package builder

  16. package chill

  17. package example

    The "example" package contains all of the code and configuration necessary to run a basic Summingbird job locally that consumes Tweets from the public streaming API and generates counts of the number of times each word appears per tweet.

  18. package graph

    Collection of graph algorithms

  19. package javaapi

  20. package memory

  21. package online

  22. package option

  23. package planner

  24. package scalding

  25. package service

  26. package sink

  27. package source

  28. package spark

  29. package store

  30. package storm

    Package containing the Summingbird Storm platform.

  31. package viz

Ungrouped