Trait

com.twitter.scalding

UtcDateRangeJob

Related Doc: package scalding

Permalink

trait UtcDateRangeJob extends Job with DefaultDateRangeJob

Source
Job.scala
Linear Supertypes
Type Hierarchy
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. UtcDateRangeJob
  2. DefaultDateRangeJob
  3. Job
  4. Serializable
  5. FieldConversions
  6. LowPriorityFieldConversions
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. implicit def _implicitJobArgs: Args

    Permalink
    Attributes
    protected
    Definition Classes
    Job
  5. def anyToFieldArg(f: Any): Comparable[_]

    Permalink
    Attributes
    protected
    Definition Classes
    LowPriorityFieldConversions
  6. val args: Args

    Permalink
    Definition Classes
    Job
  7. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  8. def asList(f: Fields): List[Comparable[_]]

    Permalink
    Definition Classes
    FieldConversions
  9. def asSet(f: Fields): Set[Comparable[_]]

    Permalink
    Definition Classes
    FieldConversions
  10. def buildFlow: Flow[_]

    Permalink

    combine the config, flowDef and the Mode to produce a flow

    combine the config, flowDef and the Mode to produce a flow

    Definition Classes
    Job
  11. def classIdentifier: String

    Permalink
    Definition Classes
    Job
  12. def clear(): Unit

    Permalink
    Definition Classes
    Job
  13. def clone(nextargs: Args): Job

    Permalink

    Copy this job By default, this uses reflection and the single argument Args constructor

    Copy this job By default, this uses reflection and the single argument Args constructor

    Definition Classes
    Job
  14. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  15. def config: Map[AnyRef, AnyRef]

    Permalink

    This is the exact config that is passed to the Cascading FlowConnector.

    This is the exact config that is passed to the Cascading FlowConnector. By default: if there are no spill thresholds in mode.config, we replace with defaultSpillThreshold we overwrite io.serializations with ioSerializations we overwrite cascading.tuple.element.comparator.default to defaultComparator we add some scalding keys for debugging/logging

    Tip: override this method, call super, and ++ your additional map to add or overwrite more options

    This returns Map[AnyRef, AnyRef] for compatibility with older code

    Definition Classes
    Job
  16. implicit def dateParser: DateParser

    Permalink

    Override this to control how dates are parsed

    Override this to control how dates are parsed

    Definition Classes
    Job
  17. implicit lazy val dateRange: DateRange

    Permalink
    Definition Classes
    DefaultDateRangeJob
  18. def defaultComparator: Option[Class[_ <: Comparator[_]]]

    Permalink

    Override this if you want to customize comparisons/hashing for your job the config method overwrites using this before sending to cascading The one we use by default is needed used to make Joins in the Fields-API more robust to Long vs Int differences.

    Override this if you want to customize comparisons/hashing for your job the config method overwrites using this before sending to cascading The one we use by default is needed used to make Joins in the Fields-API more robust to Long vs Int differences. If you only use the Typed-API, consider changing this to return None

    Definition Classes
    Job
  19. def defaultMode(fromFields: Fields, toFields: Fields): Fields

    Permalink

    Rather than give the full power of cascading's selectors, we have a simpler set of rules encoded below: 1) if the input is non-definite (ALL, GROUP, ARGS, etc...) ALL is the output.

    Rather than give the full power of cascading's selectors, we have a simpler set of rules encoded below: 1) if the input is non-definite (ALL, GROUP, ARGS, etc...) ALL is the output. Perhaps only fromFields=ALL will make sense 2) If one of from or to is a strict super set of the other, SWAP is used. 3) If they are equal, REPLACE is used. 4) Otherwise, ALL is used.

    Definition Classes
    FieldConversions
  20. def defaultSpillThreshold: Int

    Permalink

    Keep 100k tuples in memory by default before spilling Turn this up as high as you can without getting OOM.

    Keep 100k tuples in memory by default before spilling Turn this up as high as you can without getting OOM.

    This is ignored if there is a value set in the incoming jobConf on Hadoop

    Definition Classes
    Job
  21. def defaultTimeZone: TimeZone

    Permalink
    Definition Classes
    UtcDateRangeJobDefaultDateRangeJob
  22. lazy val endDate: RichDate

    Permalink
    Definition Classes
    DefaultDateRangeJob
  23. final def ensureUniqueFields(left: Fields, right: Fields, rightPipe: Pipe): (Fields, Pipe)

    Permalink
    Definition Classes
    FieldConversions
  24. implicit def enumValueToFields(x: Value): Fields

    Permalink
    Definition Classes
    FieldConversions
  25. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  26. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  27. implicit def fieldFields[T <: TraversableOnce[Field[_]]](f: T): RichFields

    Permalink
    Definition Classes
    FieldConversions
  28. implicit def fieldToFields(f: Field[_]): RichFields

    Permalink
    Definition Classes
    FieldConversions
  29. implicit def fields[T <: TraversableOnce[Symbol]](f: T): Fields

    Permalink
    Definition Classes
    FieldConversions
  30. implicit def fieldsToRichFields(fields: Fields): RichFields

    Permalink

    We can't set the field Manifests because cascading doesn't (yet) expose field type information in the Fields API.

    We can't set the field Manifests because cascading doesn't (yet) expose field type information in the Fields API.

    Definition Classes
    FieldConversions
  31. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  32. implicit val flowDef: FlowDef

    Permalink
    Attributes
    protected
    Definition Classes
    Job
  33. implicit def fromEnum[T <: Enumeration](enumeration: T): Fields

    Permalink

    Multi-entry fields.

    Multi-entry fields. This are higher priority than Product conversions so that List will not conflict with Product.

    Definition Classes
    FieldConversions
  34. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  35. def getField(f: Fields, idx: Int): Fields

    Permalink
    Definition Classes
    FieldConversions
  36. def handleStats(statsData: CascadingStats): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Job
  37. def hasInts(f: Fields): Boolean

    Permalink
    Definition Classes
    FieldConversions
  38. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  39. implicit def intFields[T <: TraversableOnce[Int]](f: T): Fields

    Permalink
    Definition Classes
    FieldConversions
  40. implicit def intToFields(x: Int): Fields

    Permalink
    Definition Classes
    FieldConversions
  41. implicit def integerToFields(x: Integer): Fields

    Permalink
    Definition Classes
    FieldConversions
  42. def ioSerializations: List[Class[_ <: Serialization[_]]]

    Permalink

    These are user-defined serializations IN-ADDITION to (but deduped) with the required serializations

    These are user-defined serializations IN-ADDITION to (but deduped) with the required serializations

    Definition Classes
    Job
  43. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  44. implicit def iterableToRichPipe[T](iter: Iterable[T])(implicit set: TupleSetter[T], conv: TupleConverter[T]): RichPipe

    Permalink
    Definition Classes
    Job
  45. def keepAlive(): Unit

    Permalink

    Use this if a map or reduce phase takes a while before emitting tuples.

    Use this if a map or reduce phase takes a while before emitting tuples.

    Definition Classes
    Job
  46. def listeners: List[FlowListener]

    Permalink
    Definition Classes
    Job
  47. implicit def mode: Mode

    Permalink
    Definition Classes
    Job
  48. def name: String

    Permalink
    Definition Classes
    Job
  49. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  50. final def newSymbol(avoid: Set[Symbol], guess: Symbol, trial: Int = 0): Symbol

    Permalink
    Definition Classes
    FieldConversions
    Annotations
    @tailrec()
  51. def next: Option[Job]

    Permalink

    Implement this method if you want some other jobs to run after the current job.

    Implement this method if you want some other jobs to run after the current job. These will not execute until the current job has run successfully.

    Definition Classes
    DefaultDateRangeJobJob
  52. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  53. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  54. implicit def parseAnySeqToFields[T <: TraversableOnce[Any]](anyf: T): Fields

    Permalink

    Useful to convert f : Any* to Fields.

    Useful to convert f : Any* to Fields. This handles mixed cases ("hey", 'you). Not sure we should be this flexible, but given that Cascading will throw an exception before scheduling the job, I guess this is okay.

    Definition Classes
    FieldConversions
  55. val period: Int

    Permalink
    Definition Classes
    DefaultDateRangeJob
  56. implicit def pipeToRichPipe(pipe: Pipe): RichPipe

    Permalink

    you should never call this directly, it is here to make the DSL work.

    you should never call this directly, it is here to make the DSL work. Just know, you can treat a Pipe as a RichPipe within a Job

    Definition Classes
    Job
  57. implicit def productToFields(f: Product): Fields

    Permalink

    Handles treating any TupleN as a Fields object.

    Handles treating any TupleN as a Fields object. This is low priority because List is also a Product, but this method will not work for List (because List is Product2(head, tail) and so productIterator won't work as expected. Lists are handled by an implicit in FieldConversions, which have higher priority.

    Definition Classes
    LowPriorityFieldConversions
  58. implicit def read(src: Source): Pipe

    Permalink

    This is implicit so that a Source can be used as the argument to a join or other method that accepts Pipe.

    This is implicit so that a Source can be used as the argument to a join or other method that accepts Pipe.

    Definition Classes
    Job
  59. def run(): Boolean

    Permalink
    Definition Classes
    Job
  60. implicit def scaldingConfig: Config

    Permalink

    This is here so that Mappable.toIterator can find an implicit config

    This is here so that Mappable.toIterator can find an implicit config

    Attributes
    protected
    Definition Classes
    Job
  61. def skipStrategy: Option[FlowSkipStrategy]

    Permalink
    Definition Classes
    Job
  62. implicit def sourceToRichPipe(src: Source): RichPipe

    Permalink

    This implicit is to enable RichPipe methods directly on Source objects, such as map/flatMap, etc...

    This implicit is to enable RichPipe methods directly on Source objects, such as map/flatMap, etc...

    Note that Mappable is a subclass of Source, and Mappable already has mapTo and flatMapTo BUT WITHOUT incoming fields used (see the Mappable trait). This creates some confusion when using these methods (this is an unfortunate mistake in our design that was not noticed until later). To remove ambiguity, explicitly call .read on any Source that you begin operating with a mapTo/flatMapTo.

    Definition Classes
    Job
  63. lazy val startDate: RichDate

    Permalink
    Definition Classes
    DefaultDateRangeJob
  64. def stepListeners: List[FlowStepListener]

    Permalink
    Definition Classes
    Job
  65. def stepStrategy: Option[FlowStepStrategy[_]]

    Permalink

    Specify a callback to run before the start of each flow step.

    Specify a callback to run before the start of each flow step.

    Defaults to what Config.getReducerEstimator specifies.

    Definition Classes
    Job
    See also

    ExecutionContext.buildFlow

  66. implicit def strFields[T <: TraversableOnce[String]](f: T): Fields

    Permalink
    Definition Classes
    FieldConversions
  67. implicit def stringToFields(x: String): Fields

    Permalink
    Definition Classes
    FieldConversions
  68. implicit def symbolToFields(x: Symbol): Fields

    Permalink

    '* means Fields.ALL, otherwise we take the .name

    '* means Fields.ALL, otherwise we take the .name

    Definition Classes
    FieldConversions
  69. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  70. def timeout[T](timeout: AbsoluteDuration)(t: ⇒ T): Option[T]

    Permalink
    Definition Classes
    Job
  71. implicit def toPipe[T](iter: Iterable[T])(implicit set: TupleSetter[T], conv: TupleConverter[T]): Pipe

    Permalink
    Definition Classes
    Job
  72. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  73. implicit def tuple2ToFieldsPair[T, U](pair: (T, U))(implicit tf: (T) ⇒ Fields, uf: (U) ⇒ Fields): (Fields, Fields)

    Permalink
    Definition Classes
    FieldConversions
  74. implicit lazy val tz: TimeZone

    Permalink
    Definition Classes
    DefaultDateRangeJob
  75. implicit def unitToFields(u: Unit): Fields

    Permalink
    Definition Classes
    FieldConversions
  76. def validate(): Unit

    Permalink
    Definition Classes
    Job
  77. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  78. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  79. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  80. def write(pipe: Pipe, src: Source): Unit

    Permalink

    This is only here for Java jobs which cannot automatically access the implicit Pipe => RichPipe which makes: pipe.write( ) convenient

    This is only here for Java jobs which cannot automatically access the implicit Pipe => RichPipe which makes: pipe.write( ) convenient

    Definition Classes
    Job

Inherited from DefaultDateRangeJob

Inherited from Job

Inherited from Serializable

Inherited from FieldConversions

Inherited from AnyRef

Inherited from Any

Ungrouped