Trait

com.twitter.scalding

CounterVerification

Related Doc: package scalding

Permalink

trait CounterVerification extends Job

Allows custom counter verification logic when the job completes.

Source
Job.scala
Linear Supertypes
Job, Serializable, FieldConversions, LowPriorityFieldConversions, AnyRef, Any
Type Hierarchy
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. CounterVerification
  2. Job
  3. Serializable
  4. FieldConversions
  5. LowPriorityFieldConversions
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def verifyCounters(counters: Map[StatKey, Long]): Try[Unit]

    Permalink

    Verify counter values.

    Verify counter values. The job will fail if this returns false or throws an exception.

Concrete Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. implicit def _implicitJobArgs: Args

    Permalink
    Attributes
    protected
    Definition Classes
    Job
  5. def anyToFieldArg(f: Any): Comparable[_]

    Permalink
    Attributes
    protected
    Definition Classes
    LowPriorityFieldConversions
  6. val args: Args

    Permalink
    Definition Classes
    Job
  7. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  8. def asList(f: Fields): List[Comparable[_]]

    Permalink
    Definition Classes
    FieldConversions
  9. def asSet(f: Fields): Set[Comparable[_]]

    Permalink
    Definition Classes
    FieldConversions
  10. def buildFlow: Flow[_]

    Permalink

    combine the config, flowDef and the Mode to produce a flow

    combine the config, flowDef and the Mode to produce a flow

    Definition Classes
    Job
  11. def classIdentifier: String

    Permalink
    Definition Classes
    Job
  12. def clear(): Unit

    Permalink
    Definition Classes
    Job
  13. def clone(nextargs: Args): Job

    Permalink

    Copy this job By default, this uses reflection and the single argument Args constructor

    Copy this job By default, this uses reflection and the single argument Args constructor

    Definition Classes
    Job
  14. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  15. def config: Map[AnyRef, AnyRef]

    Permalink

    This is the exact config that is passed to the Cascading FlowConnector.

    This is the exact config that is passed to the Cascading FlowConnector. By default: if there are no spill thresholds in mode.config, we replace with defaultSpillThreshold we overwrite io.serializations with ioSerializations we overwrite cascading.tuple.element.comparator.default to defaultComparator we add some scalding keys for debugging/logging

    Tip: override this method, call super, and ++ your additional map to add or overwrite more options

    This returns Map[AnyRef, AnyRef] for compatibility with older code

    Definition Classes
    Job
  16. implicit def dateParser: DateParser

    Permalink

    Override this to control how dates are parsed

    Override this to control how dates are parsed

    Definition Classes
    Job
  17. def defaultComparator: Option[Class[_ <: Comparator[_]]]

    Permalink

    Override this if you want to customize comparisons/hashing for your job the config method overwrites using this before sending to cascading The one we use by default is needed used to make Joins in the Fields-API more robust to Long vs Int differences.

    Override this if you want to customize comparisons/hashing for your job the config method overwrites using this before sending to cascading The one we use by default is needed used to make Joins in the Fields-API more robust to Long vs Int differences. If you only use the Typed-API, consider changing this to return None

    Definition Classes
    Job
  18. def defaultMode(fromFields: Fields, toFields: Fields): Fields

    Permalink

    Rather than give the full power of cascading's selectors, we have a simpler set of rules encoded below: 1) if the input is non-definite (ALL, GROUP, ARGS, etc...) ALL is the output.

    Rather than give the full power of cascading's selectors, we have a simpler set of rules encoded below: 1) if the input is non-definite (ALL, GROUP, ARGS, etc...) ALL is the output. Perhaps only fromFields=ALL will make sense 2) If one of from or to is a strict super set of the other, SWAP is used. 3) If they are equal, REPLACE is used. 4) Otherwise, ALL is used.

    Definition Classes
    FieldConversions
  19. def defaultSpillThreshold: Int

    Permalink

    Keep 100k tuples in memory by default before spilling Turn this up as high as you can without getting OOM.

    Keep 100k tuples in memory by default before spilling Turn this up as high as you can without getting OOM.

    This is ignored if there is a value set in the incoming jobConf on Hadoop

    Definition Classes
    Job
  20. final def ensureUniqueFields(left: Fields, right: Fields, rightPipe: Pipe): (Fields, Pipe)

    Permalink
    Definition Classes
    FieldConversions
  21. implicit def enumValueToFields(x: Value): Fields

    Permalink
    Definition Classes
    FieldConversions
  22. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  23. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  24. implicit def fieldFields[T <: TraversableOnce[Field[_]]](f: T): RichFields

    Permalink
    Definition Classes
    FieldConversions
  25. implicit def fieldToFields(f: Field[_]): RichFields

    Permalink
    Definition Classes
    FieldConversions
  26. implicit def fields[T <: TraversableOnce[Symbol]](f: T): Fields

    Permalink
    Definition Classes
    FieldConversions
  27. implicit def fieldsToRichFields(fields: Fields): RichFields

    Permalink

    We can't set the field Manifests because cascading doesn't (yet) expose field type information in the Fields API.

    We can't set the field Manifests because cascading doesn't (yet) expose field type information in the Fields API.

    Definition Classes
    FieldConversions
  28. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  29. implicit val flowDef: FlowDef

    Permalink
    Attributes
    protected
    Definition Classes
    Job
  30. implicit def fromEnum[T <: Enumeration](enumeration: T): Fields

    Permalink

    Multi-entry fields.

    Multi-entry fields. This are higher priority than Product conversions so that List will not conflict with Product.

    Definition Classes
    FieldConversions
  31. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  32. def getField(f: Fields, idx: Int): Fields

    Permalink
    Definition Classes
    FieldConversions
  33. def handleStats(statsData: CascadingStats): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Job
  34. def hasInts(f: Fields): Boolean

    Permalink
    Definition Classes
    FieldConversions
  35. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  36. implicit def intFields[T <: TraversableOnce[Int]](f: T): Fields

    Permalink
    Definition Classes
    FieldConversions
  37. implicit def intToFields(x: Int): Fields

    Permalink
    Definition Classes
    FieldConversions
  38. implicit def integerToFields(x: Integer): Fields

    Permalink
    Definition Classes
    FieldConversions
  39. def ioSerializations: List[Class[_ <: Serialization[_]]]

    Permalink

    These are user-defined serializations IN-ADDITION to (but deduped) with the required serializations

    These are user-defined serializations IN-ADDITION to (but deduped) with the required serializations

    Definition Classes
    Job
  40. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  41. implicit def iterableToRichPipe[T](iter: Iterable[T])(implicit set: TupleSetter[T], conv: TupleConverter[T]): RichPipe

    Permalink
    Definition Classes
    Job
  42. def keepAlive(): Unit

    Permalink

    Use this if a map or reduce phase takes a while before emitting tuples.

    Use this if a map or reduce phase takes a while before emitting tuples.

    Definition Classes
    Job
  43. def listeners: List[FlowListener]

    Permalink
    Definition Classes
    CounterVerificationJob
  44. implicit def mode: Mode

    Permalink
    Definition Classes
    Job
  45. def name: String

    Permalink
    Definition Classes
    Job
  46. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  47. final def newSymbol(avoid: Set[Symbol], guess: Symbol, trial: Int = 0): Symbol

    Permalink
    Definition Classes
    FieldConversions
    Annotations
    @tailrec()
  48. def next: Option[Job]

    Permalink

    Implement this method if you want some other jobs to run after the current job.

    Implement this method if you want some other jobs to run after the current job. These will not execute until the current job has run successfully.

    Definition Classes
    Job
  49. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  50. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  51. implicit def parseAnySeqToFields[T <: TraversableOnce[Any]](anyf: T): Fields

    Permalink

    Useful to convert f : Any* to Fields.

    Useful to convert f : Any* to Fields. This handles mixed cases ("hey", 'you). Not sure we should be this flexible, but given that Cascading will throw an exception before scheduling the job, I guess this is okay.

    Definition Classes
    FieldConversions
  52. implicit def pipeToRichPipe(pipe: Pipe): RichPipe

    Permalink

    you should never call this directly, it is here to make the DSL work.

    you should never call this directly, it is here to make the DSL work. Just know, you can treat a Pipe as a RichPipe within a Job

    Definition Classes
    Job
  53. implicit def productToFields(f: Product): Fields

    Permalink

    Handles treating any TupleN as a Fields object.

    Handles treating any TupleN as a Fields object. This is low priority because List is also a Product, but this method will not work for List (because List is Product2(head, tail) and so productIterator won't work as expected. Lists are handled by an implicit in FieldConversions, which have higher priority.

    Definition Classes
    LowPriorityFieldConversions
  54. implicit def read(src: Source): Pipe

    Permalink

    This is implicit so that a Source can be used as the argument to a join or other method that accepts Pipe.

    This is implicit so that a Source can be used as the argument to a join or other method that accepts Pipe.

    Definition Classes
    Job
  55. def run(): Boolean

    Permalink
    Definition Classes
    Job
  56. implicit def scaldingConfig: Config

    Permalink

    This is here so that Mappable.toIterator can find an implicit config

    This is here so that Mappable.toIterator can find an implicit config

    Attributes
    protected
    Definition Classes
    Job
  57. def skipStrategy: Option[FlowSkipStrategy]

    Permalink
    Definition Classes
    Job
  58. implicit def sourceToRichPipe(src: Source): RichPipe

    Permalink

    This implicit is to enable RichPipe methods directly on Source objects, such as map/flatMap, etc...

    This implicit is to enable RichPipe methods directly on Source objects, such as map/flatMap, etc...

    Note that Mappable is a subclass of Source, and Mappable already has mapTo and flatMapTo BUT WITHOUT incoming fields used (see the Mappable trait). This creates some confusion when using these methods (this is an unfortunate mistake in our design that was not noticed until later). To remove ambiguity, explicitly call .read on any Source that you begin operating with a mapTo/flatMapTo.

    Definition Classes
    Job
  59. def stepListeners: List[FlowStepListener]

    Permalink
    Definition Classes
    Job
  60. def stepStrategy: Option[FlowStepStrategy[_]]

    Permalink

    Specify a callback to run before the start of each flow step.

    Specify a callback to run before the start of each flow step.

    Defaults to what Config.getReducerEstimator specifies.

    Definition Classes
    Job
    See also

    ExecutionContext.buildFlow

  61. implicit def strFields[T <: TraversableOnce[String]](f: T): Fields

    Permalink
    Definition Classes
    FieldConversions
  62. implicit def stringToFields(x: String): Fields

    Permalink
    Definition Classes
    FieldConversions
  63. implicit def symbolToFields(x: Symbol): Fields

    Permalink

    '* means Fields.ALL, otherwise we take the .name

    '* means Fields.ALL, otherwise we take the .name

    Definition Classes
    FieldConversions
  64. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  65. def timeout[T](timeout: AbsoluteDuration)(t: ⇒ T): Option[T]

    Permalink
    Definition Classes
    Job
  66. implicit def toPipe[T](iter: Iterable[T])(implicit set: TupleSetter[T], conv: TupleConverter[T]): Pipe

    Permalink
    Definition Classes
    Job
  67. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  68. implicit def tuple2ToFieldsPair[T, U](pair: (T, U))(implicit tf: (T) ⇒ Fields, uf: (U) ⇒ Fields): (Fields, Fields)

    Permalink
    Definition Classes
    FieldConversions
  69. implicit def unitToFields(u: Unit): Fields

    Permalink
    Definition Classes
    FieldConversions
  70. def validate(): Unit

    Permalink
    Definition Classes
    Job
  71. def verifyCountersInTest: Boolean

    Permalink

    Override this to false to skip counter verification in tests.

  72. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  73. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  74. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  75. def write(pipe: Pipe, src: Source): Unit

    Permalink

    This is only here for Java jobs which cannot automatically access the implicit Pipe => RichPipe which makes: pipe.write( ) convenient

    This is only here for Java jobs which cannot automatically access the implicit Pipe => RichPipe which makes: pipe.write( ) convenient

    Definition Classes
    Job

Inherited from Job

Inherited from Serializable

Inherited from FieldConversions

Inherited from AnyRef

Inherited from Any

Ungrouped