Class

com.twitter.scalding

PartitionSource

Related Doc: package scalding

Permalink

abstract class PartitionSource extends SchemedSource with HfsTapProvider

This is a base class for partition-based output sources

Source
PartitionSource.scala
Linear Supertypes
HfsTapProvider, SchemedSource, Source, Serializable, AnyRef, Any
Known Subclasses
Type Hierarchy
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. PartitionSource
  2. HfsTapProvider
  3. SchemedSource
  4. Source
  5. Serializable
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new PartitionSource(openWritesThreshold: Option[Int] = None)

    Permalink

Abstract Value Members

  1. abstract def basePath: String

    Permalink

Concrete Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  5. def checkFlowDefNotNull()(implicit flowDef: FlowDef, mode: Mode): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Source
  6. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  7. def createHfsTap(scheme: Scheme[JobConf, RecordReader[_, _], OutputCollector[_, _], _, _], path: String, sinkMode: SinkMode): Hfs

    Permalink
    Definition Classes
    HfsTapProvider
  8. def createTap(readOrWrite: AccessMode)(implicit mode: Mode): Tap[_, _, _]

    Permalink

    Creates the partition tap.

    Creates the partition tap.

    readOrWrite

    Describes if this source is being read from or written to.

    mode

    The mode of the job. (implicit)

    returns

    A cascading PartitionTap.

    Definition Classes
    PartitionSourceSource
  9. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  10. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  11. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  12. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  13. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  14. def hdfsScheme: Scheme[JobConf, RecordReader[_, _], OutputCollector[_, _], _, _]

    Permalink

    The scheme to use if the source is on hdfs.

    The scheme to use if the source is on hdfs.

    Definition Classes
    SchemedSource
  15. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  16. def localScheme: Scheme[Properties, InputStream, OutputStream, _, _]

    Permalink

    The scheme to use if the source is local.

    The scheme to use if the source is local.

    Definition Classes
    SchemedSource
  17. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  18. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  19. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  20. val openWritesThreshold: Option[Int]

    Permalink
  21. def partition: Partition

    Permalink
  22. def read(implicit flowDef: FlowDef, mode: Mode): Pipe

    Permalink
    Definition Classes
    Source
  23. val sinkMode: SinkMode

    Permalink
    Definition Classes
    SchemedSource
  24. def sourceId: String

    Permalink

    This is a name the refers to this exact instance of the source (put another way, if s1.sourceId == s2.sourceId, the job should work the same if one is replaced with the other

    This is a name the refers to this exact instance of the source (put another way, if s1.sourceId == s2.sourceId, the job should work the same if one is replaced with the other

    Definition Classes
    Source
  25. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  26. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  27. def transformForRead(pipe: Pipe): Pipe

    Permalink
    Attributes
    protected
    Definition Classes
    Source
  28. def transformForWrite(pipe: Pipe): Pipe

    Permalink
    Attributes
    protected
    Definition Classes
    Source
  29. def transformInTest: Boolean

    Permalink

    The mock passed in to scalding.JobTest may be considered as a mock of the Tap or the Source.

    The mock passed in to scalding.JobTest may be considered as a mock of the Tap or the Source. By default, as of 0.9.0, it is considered as a Mock of the Source. If you set this to true, the mock in TestMode will be considered to be a mock of the Tap (which must be transformed) and not the Source.

    Definition Classes
    Source
  30. def validateTaps(mode: Mode): Unit

    Permalink

    Validates the taps, makes sure there are no nulls in the path.

    Validates the taps, makes sure there are no nulls in the path.

    mode

    The mode of the job.

    Definition Classes
    PartitionSourceSource
  31. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  32. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  33. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  34. def writeFrom(pipe: Pipe)(implicit flowDef: FlowDef, mode: Mode): Pipe

    Permalink

    write the pipe but return the input so it can be chained into the next operation

    write the pipe but return the input so it can be chained into the next operation

    Definition Classes
    Source

Deprecated Value Members

  1. def readAtSubmitter[T](implicit mode: Mode, conv: TupleConverter[T]): Stream[T]

    Permalink
    Definition Classes
    Source
    Annotations
    @deprecated
    Deprecated

    (Since version 0.9.0) replace with Mappable.toIterator

Inherited from HfsTapProvider

Inherited from SchemedSource

Inherited from Source

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped