com.twitter.summingbird.example

StormRunner

object StormRunner

The following object contains code to execute the Summingbird WordCount job defined in ExampleJob.scala on a storm cluster.

Source
StormRunner.scala
Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. StormRunner
  2. AnyRef
  3. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. def apply(args: Args): StormExecutionConfig

    This function will be called by the storm runner to request the info of what to run.

    This function will be called by the storm runner to request the info of what to run. In local mode it will start up as a separate thread on the local machine, pulling tweets off of the TwitterSpout, generating and aggregating key-value pairs and merging the incremental counts in the memcache store.

    Before running this code, make sure to start a local memcached instance with "memcached". ("brew install memcached" will get you all set up if you don't already have memcache installed locally.)

  7. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  8. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  9. lazy val config: Configuration

    Configuration for Twitter4j.

    Configuration for Twitter4j. Configuration can also be managed via a properties file, as described here:

    http://tugdualgrall.blogspot.com/2012/11/couchbase-create-large-dataset-using.html

  10. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  11. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  12. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  13. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  14. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  15. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  16. def lookup(word: String): Option[Long]

    Once you've got this running in the background, fire up another repl and query memcached for some counts.

    Once you've got this running in the background, fire up another repl and query memcached for some counts.

    The following commands will look up words. Hitting a word twice will show that Storm is updating Memcache behind the scenes:

    scala>     lookup("i") // Or any other common word
    res7: Option[Long] = Some(1774)
    
    scala>     lookup("i")
    res8: Option[Long] = Some(1779)
  17. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  18. final def notify(): Unit

    Definition Classes
    AnyRef
  19. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  20. val spout: TwitterSpout[Status]

    "spout" is a concrete Storm source for Status data.

    "spout" is a concrete Storm source for Status data. This will act as the initial producer of Status instances in the Summingbird word count job.

  21. val storeSupplier: MergeableStoreFactory[(String, BatchID), Long]

    the param to store is by name, so this is still not created created yet

  22. lazy val stringLongStore: MergeableStore[(String, BatchID), Long]

    And here's our MergeableStore supplier.

    And here's our MergeableStore supplier.

    A supplier is required (vs a bare store) because Storm serializes every constructor parameter to its "bolts". Serializing a live memcache store is a no-no, so the Storm platform accepts a "supplier", essentially a function0 that when called will pop out a new instance of the required store. This instance is cached when the bolt starts up and starts merging tuples.

    A MergeableStore is a store that's aware of aggregation and knows how to merge in new (K, V) pairs using a Monoid[V]. The Monoid[Long] used by this particular store is being pulled in from the Monoid companion object in Algebird. (Most trivial Monoid instances will resolve this way.)

    First, the backing store:

  23. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  24. def toString(): String

    Definition Classes
    AnyRef → Any
  25. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  26. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  27. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped