Trait/Object

com.twitter.scalding

Config

Related Docs: object Config | package scalding

Permalink

trait Config extends Serializable

This is a wrapper class on top of Map[String, String]

Source
Config.scala
Linear Supertypes
Serializable, Serializable, AnyRef, Any
Type Hierarchy
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Config
  2. Serializable
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def toMap: Map[String, String]

    Permalink

Concrete Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. def +(kv: (String, String)): Config

    Permalink
  4. def ++(that: Config): Config

    Permalink
  5. def -(k: String): Config

    Permalink
  6. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  7. def addCascadingClassSerializationTokens(clazzes: Set[Class[_]]): Config

    Permalink

    The serialization of your data will be smaller if any classes passed between tasks in your job are listed here.

    The serialization of your data will be smaller if any classes passed between tasks in your job are listed here. Without this, strings are used to write the types IN EACH RECORD, which compression probably takes care of, but compression acts AFTER the data is serialized into buffers and spilling has been triggered.

  8. def addFlowListener(flowListenerProvider: (Mode, Config) ⇒ FlowListener): Config

    Permalink

    configure flow listeneres for observability

  9. def addFlowStepListener(flowListenerProvider: (Mode, Config) ⇒ FlowStepListener): Config

    Permalink
  10. def addFlowStepStrategy(flowStrategyProvider: (Mode, Config) ⇒ FlowStepStrategy[JobConf]): Config

    Permalink
  11. def addReducerEstimator(clsName: String): Config

    Permalink

    Prepend an estimator so it will be tried first.

    Prepend an estimator so it will be tried first. If it returns None, the previously-set estimators will be tried in order.

  12. def addReducerEstimator[T](cls: Class[T]): Config

    Permalink

    Prepend an estimator so it will be tried first.

    Prepend an estimator so it will be tried first. If it returns None, the previously-set estimators will be tried in order.

  13. def addUniqueId(u: UniqueID): Config

    Permalink
  14. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  15. def clearFlowStepStrategies: Config

    Permalink
  16. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  17. def ensureUniqueId: (UniqueID, Config)

    Permalink

    Allocate a new UniqueID if there is not one present

  18. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  19. def equals(that: Any): Boolean

    Permalink
    Definition Classes
    Config → AnyRef → Any
  20. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  21. def get(key: String): Option[String]

    Permalink
  22. def getArgs: Args

    Permalink
  23. def getCascadingAppJar: Option[Try[Class[_]]]

    Permalink

    Returns None if not set, otherwise reflection is used to create the Class.forName

  24. def getCascadingAppName: Option[String]

    Permalink

    This is a name that if present is passed to flow.setName, which should appear in the job tracker.

  25. def getCascadingSerializationTokens: Map[Int, String]

    Permalink
  26. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  27. def getFlowListeners: List[Try[(Mode, Config) ⇒ FlowListener]]

    Permalink
  28. def getFlowStepListeners: List[Try[(Mode, Config) ⇒ FlowStepListener]]

    Permalink
  29. def getFlowStepStrategies: List[Try[(Mode, Config) ⇒ FlowStepStrategy[JobConf]]]

    Permalink
  30. def getHashJoinAutoForceRight: Boolean

    Permalink
  31. def getKryo: Option[KryoInstantiator]

    Permalink
  32. def getKryoRegisteredClasses: Set[Class[_]]

    Permalink

    This function gets the set of classes that have been registered to Kryo.

    This function gets the set of classes that have been registered to Kryo. They may or may not be used in this job, but Cascading might want to be made aware that these classes exist

  33. def getNumReducers: Option[Int]

    Permalink

    Get the number of reducers (this is the parameter Hadoop will use)

  34. def getRequireOrderedSerialization: Boolean

    Permalink
  35. def getScaldingExecutionId: Option[String]

    Permalink
  36. def getScaldingVersion: Option[String]

    Permalink
  37. def getSubmittedTimestamp: Option[RichDate]

    Permalink
  38. def getUniqueIds: Set[UniqueID]

    Permalink
  39. def hashCode(): Int

    Permalink
    Definition Classes
    Config → AnyRef → Any
  40. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  41. def maybeSetSubmittedTimestamp(date: RichDate = RichDate.now): (Option[RichDate], Config)

    Permalink
  42. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  43. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  44. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  45. def setArgs(args: Args): Config

    Permalink
  46. def setCascadingAppId(id: String): Config

    Permalink
  47. def setCascadingAppJar(clazz: Class[_]): Config

    Permalink

    Non-fat-jar use cases require this, BUT using it with fat jars can cause problems.

    Non-fat-jar use cases require this, BUT using it with fat jars can cause problems. It is not set by default, but if you have problems you might need to set the Job class here Consider also setting this same class here: setScaldingFlowClass

  48. def setCascadingAppName(name: String): Config

    Permalink
  49. def setDefaultComparator(clazz: Class[_ <: Comparator[_]]): Config

    Permalink
  50. def setHRavenHistoryUserName: Config

    Permalink

    Set username from System.used for querying hRaven.

  51. def setHashJoinAutoForceRight(b: Boolean): Config

    Permalink
  52. def setListSpillThreshold(count: Int): Config

    Permalink
  53. def setMapSideAggregationThreshold(count: Int): Config

    Permalink
  54. def setMapSpillThreshold(count: Int): Config

    Permalink
  55. def setNumReducers(n: Int): Config

    Permalink
  56. def setReducerEstimators(clsList: String): Config

    Permalink

    Set the entire list of reducer estimators (overriding the existing list)

  57. def setRequireOrderedSerialization(b: Boolean): Config

    Permalink

    Set this configuration option to require all grouping/cogrouping to use OrderedSerialization

  58. def setScaldingExecutionId(id: String): Config

    Permalink

    Set an ID to be shared across this usage of run for Execution

  59. def setScaldingFlowClass(clazz: Class[_]): Config

    Permalink
  60. def setScaldingVersion: Config

    Permalink
  61. def setSerialization(kryo: Either[(Class[_ <: KryoInstantiator], KryoInstantiator), Class[_ <: KryoInstantiator]], userHadoop: Seq[Class[_ <: Serialization[_]]] = Nil): Config

    Permalink
  62. def setVerboseFileSourceLogging(b: Boolean): Config

    Permalink

    Set to true to enable very verbose logging during FileSource's validation and planning.

    Set to true to enable very verbose logging during FileSource's validation and planning. This can help record what files were present / missing at runtime. Should only be enabled for debugging.

  63. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  64. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  65. def update[R](k: String)(fn: (Option[String]) ⇒ (Option[String], R)): (R, Config)

    Permalink
  66. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  67. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  68. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped