Class/Object

com.twitter.scalding

PartitionedSequenceFile

Related Docs: object PartitionedSequenceFile | package scalding

Permalink

case class PartitionedSequenceFile(basePath: String, partition: Partition, sequenceFields: Fields, sinkMode: SinkMode) extends PartitionSource with SequenceFileScheme with Product with Serializable

An implementation of SequenceFile output, split over a partition tap.

basePath

The root path for the output.

partition

The partitioning strategy to use.

sequenceFields

The set of fields to use for the sequence file.

sinkMode

How to handle conflicts with existing output.

Source
PartitionSource.scala
Linear Supertypes
Serializable, Product, Equals, SequenceFileScheme, PartitionSource, HfsTapProvider, SchemedSource, Source, Serializable, AnyRef, Any
Type Hierarchy
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. PartitionedSequenceFile
  2. Serializable
  3. Product
  4. Equals
  5. SequenceFileScheme
  6. PartitionSource
  7. HfsTapProvider
  8. SchemedSource
  9. Source
  10. Serializable
  11. AnyRef
  12. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new PartitionedSequenceFile(basePath: String, partition: Partition, sequenceFields: Fields, sinkMode: SinkMode)

    Permalink

    basePath

    The root path for the output.

    partition

    The partitioning strategy to use.

    sequenceFields

    The set of fields to use for the sequence file.

    sinkMode

    How to handle conflicts with existing output.

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  5. val basePath: String

    Permalink

    The root path for the output.

    The root path for the output.

    Definition Classes
    PartitionedSequenceFilePartitionSource
  6. def checkFlowDefNotNull()(implicit flowDef: FlowDef, mode: Mode): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Source
  7. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  8. def createHfsTap(scheme: Scheme[JobConf, RecordReader[_, _], OutputCollector[_, _], _, _], path: String, sinkMode: SinkMode): Hfs

    Permalink
    Definition Classes
    HfsTapProvider
  9. def createTap(readOrWrite: AccessMode)(implicit mode: Mode): Tap[_, _, _]

    Permalink

    Creates the partition tap.

    Creates the partition tap.

    readOrWrite

    Describes if this source is being read from or written to.

    mode

    The mode of the job. (implicit)

    returns

    A cascading PartitionTap.

    Definition Classes
    PartitionSourceSource
  10. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  11. val fields: Fields

    Permalink
  12. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  13. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  14. def hdfsScheme: Scheme[JobConf, RecordReader[_, _], OutputCollector[_, _], _, _]

    Permalink

    The scheme to use if the source is on hdfs.

    The scheme to use if the source is on hdfs.

    Definition Classes
    SequenceFileSchemeSchemedSource
  15. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  16. def localScheme: Scheme[Properties, InputStream, OutputStream, _, _]

    Permalink

    The scheme to use if the source is local.

    The scheme to use if the source is local.

    Definition Classes
    SchemedSource
  17. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  18. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  19. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  20. val openWritesThreshold: Option[Int]

    Permalink
    Definition Classes
    PartitionSource
  21. val partition: Partition

    Permalink

    The partitioning strategy to use.

    The partitioning strategy to use.

    Definition Classes
    PartitionedSequenceFilePartitionSource
  22. def read(implicit flowDef: FlowDef, mode: Mode): Pipe

    Permalink
    Definition Classes
    Source
  23. val sequenceFields: Fields

    Permalink

    The set of fields to use for the sequence file.

  24. val sinkMode: SinkMode

    Permalink

    How to handle conflicts with existing output.

    How to handle conflicts with existing output.

    Definition Classes
    PartitionedSequenceFileSchemedSource
  25. def sourceId: String

    Permalink

    This is a name the refers to this exact instance of the source (put another way, if s1.sourceId == s2.sourceId, the job should work the same if one is replaced with the other

    This is a name the refers to this exact instance of the source (put another way, if s1.sourceId == s2.sourceId, the job should work the same if one is replaced with the other

    Definition Classes
    Source
  26. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  27. def transformForRead(pipe: Pipe): Pipe

    Permalink
    Attributes
    protected
    Definition Classes
    Source
  28. def transformForWrite(pipe: Pipe): Pipe

    Permalink
    Attributes
    protected
    Definition Classes
    Source
  29. def transformInTest: Boolean

    Permalink

    The mock passed in to scalding.JobTest may be considered as a mock of the Tap or the Source.

    The mock passed in to scalding.JobTest may be considered as a mock of the Tap or the Source. By default, as of 0.9.0, it is considered as a Mock of the Source. If you set this to true, the mock in TestMode will be considered to be a mock of the Tap (which must be transformed) and not the Source.

    Definition Classes
    Source
  30. def validateTaps(mode: Mode): Unit

    Permalink

    Validates the taps, makes sure there are no nulls in the path.

    Validates the taps, makes sure there are no nulls in the path.

    mode

    The mode of the job.

    Definition Classes
    PartitionSourceSource
  31. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  32. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  33. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  34. def writeFrom(pipe: Pipe)(implicit flowDef: FlowDef, mode: Mode): Pipe

    Permalink

    write the pipe but return the input so it can be chained into the next operation

    write the pipe but return the input so it can be chained into the next operation

    Definition Classes
    Source

Deprecated Value Members

  1. def readAtSubmitter[T](implicit mode: Mode, conv: TupleConverter[T]): Stream[T]

    Permalink
    Definition Classes
    Source
    Annotations
    @deprecated
    Deprecated

    (Since version 0.9.0) replace with Mappable.toIterator

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from SequenceFileScheme

Inherited from PartitionSource

Inherited from HfsTapProvider

Inherited from SchemedSource

Inherited from Source

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped