Object/Trait

com.twitter.scalding

Execution

Related Docs: trait Execution | package scalding

Permalink

object Execution extends Serializable

Execution has many methods for creating Execution[T] instances, which are the preferred way to compose computations in scalding libraries.

Source
Execution.scala
Linear Supertypes
Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Execution
  2. Serializable
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. implicit object ExecutionMonad extends Monad[Execution]

    Permalink

    This is an instance of Monad for execution so it can be used in functions that apply to all Monads

  5. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  6. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  7. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  8. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  9. def failFastSequence[T](t: Iterable[Future[T]])(implicit cec: scala.concurrent.ExecutionContext): Future[List[T]]

    Permalink

    Use our internal faster failing zip function rather than the standard one due to waiting

  10. def failFastZip[T, U](ft: Future[T], fu: Future[U])(implicit cec: scala.concurrent.ExecutionContext): Future[(T, U)]

    Permalink

    Standard scala zip waits forever on the left side, even if the right side fails

  11. def failed(t: Throwable): Execution[Nothing]

    Permalink

    This creates a definitely failed Execution.

  12. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  13. def from[T](t: ⇒ T): Execution[T]

    Permalink

    This makes a constant execution that runs no job.

    This makes a constant execution that runs no job. Note this is a lazy parameter that is evaluated every time run is called and does so in the ExecutionContext given to run

  14. def fromFn(fn: (Config, Mode) ⇒ FlowDef): Execution[Unit]

    Permalink

    This converts a function into an Execution monad.

    This converts a function into an Execution monad. The flowDef returned is never mutated.

  15. def fromFuture[T](fn: (scala.concurrent.ExecutionContext) ⇒ Future[T]): Execution[T]

    Permalink

    The call to fn will happen when the run method on the result is called.

    The call to fn will happen when the run method on the result is called. The ConcurrentExecutionContext will be the same one used on run. This is intended for cases where you need to make asynchronous calls in the middle or end of execution. Presumably this is used with flatMap either before or after

  16. def fromTry[T](t: ⇒ Try[T]): Execution[T]

    Permalink

    This evaluates the argument every time run is called, and does so in the ExecutionContext given to run

  17. def getArgs: Execution[Args]

    Permalink

    Convenience method to get the Args

  18. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  19. def getConfig: Execution[Config]

    Permalink

    Use this to read the configuration, which may contain Args or options which describe input on which to run

  20. def getConfigMode: Execution[(Config, Mode)]

    Permalink

    Use this to get the config and mode.

  21. def getMode: Execution[Mode]

    Permalink

    Use this to get the mode, which may contain the job conf

  22. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  23. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  24. implicit def monoid[T](implicit arg0: Monoid[T]): Monoid[Execution[T]]

    Permalink

    This is the standard monoid on an Applicative (zip, then inside the Execution do plus) useful to combine unit Executions: Monoid.sum(ex1, ex2, ex3, ex4): Execution[Unit] where each are exi are Execution[Unit]

  25. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  26. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  27. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  28. def run[C](flow: Flow[C]): Future[JobStats]

    Permalink
  29. implicit def semigroup[T](implicit arg0: Semigroup[T]): Semigroup[Execution[T]]

    Permalink

    This is the standard semigroup on an Applicative (zip, then inside the Execution do plus)

  30. def sequence[T](exs: Seq[Execution[T]]): Execution[Seq[T]]

    Permalink
  31. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  32. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  33. val unit: Execution[Unit]

    Permalink

    Returns a constant Execution[Unit]

  34. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  35. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  36. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  37. def waitFor[C](flow: Flow[C]): Try[JobStats]

    Permalink
  38. def withArgs[T](fn: (Args) ⇒ Execution[T]): Execution[T]

    Permalink

    This is convenience method only here to make it slightly cleaner to get Args, which are in the Config

  39. def withConfig[T](ex: Execution[T])(c: (Config) ⇒ Config): Execution[T]

    Permalink
  40. def withId[T](fn: (UniqueID) ⇒ Execution[T]): Execution[T]

    Permalink

    Use this to use counters/stats with Execution.

    Use this to use counters/stats with Execution. You do this: Execution.withId { implicit uid => val myStat = Stat("myStat") // uid is implicitly pulled in pipe.map { t => if(someCase(t)) myStat.inc fn(t) } .writeExecution(mySink) }

  41. def withNewCache[T](ex: Execution[T]): Execution[T]

    Permalink

    This function allows running the passed execution with its own cache.

    This function allows running the passed execution with its own cache. This will mean anything inside won't benefit from Execution's global attempts to avoid repeated executions.

    The main use case here is when generating a lot of Execution results which are large. Executions caching in this case can lead to out of memory errors as the cache keeps references to many heap objects.

    Ex. (0 until 1000).map { _ => Execution.withNewCache(myLargeObjectProducingExecution)}

  42. def withParallelism[T](executions: Seq[Execution[T]], parallelism: Int): Execution[Seq[T]]

    Permalink

    Run a sequence of executions but only permitting parallelism amount to run at the same time.

    Run a sequence of executions but only permitting parallelism amount to run at the same time.

    executions

    List of executions to run

    parallelism

    Number to run in parallel

    returns

    Execution Seq

  43. def zip[A, B, C, D, E](ax: Execution[A], bx: Execution[B], cx: Execution[C], dx: Execution[D], ex: Execution[E]): Execution[(A, B, C, D, E)]

    Permalink

    combine several executions and run them in parallel when .run is called

  44. def zip[A, B, C, D](ax: Execution[A], bx: Execution[B], cx: Execution[C], dx: Execution[D]): Execution[(A, B, C, D)]

    Permalink

    combine several executions and run them in parallel when .run is called

  45. def zip[A, B, C](ax: Execution[A], bx: Execution[B], cx: Execution[C]): Execution[(A, B, C)]

    Permalink

    combine several executions and run them in parallel when .run is called

  46. def zip[A, B](ax: Execution[A], bx: Execution[B]): Execution[(A, B)]

    Permalink

    combine several executions and run them in parallel when .run is called

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped