com.twitter.summingbird

example

package example

The "example" package contains all of the code and configuration necessary to run a basic Summingbird job locally that consumes Tweets from the public streaming API and generates counts of the number of times each word appears per tweet.

Clients can use the code exposed in this example to build realtime versions of word-count dashboards like Google's N-Gram product:

http://books.google.com/ngrams

# Code Structure

## Serialization.scala

defines a number of serialization Injections needed by the Storm and Scalding platforms to ensure that data can move across network boundaries without corruption.

## Storage.scala

Defines a few helper methods that make it easy to instantiate instances of MergeableStore backed by Memcache.

## ExampleJob.scala

The actual Summingbird job, plus a couple of helper implicits (a batcher and a time extractor) necessary for running jobs in combined batch/realtime mode across Scalding and Storm.

## StormRunner.scala

Configuration and Execution of the summingbird word count job in Storm's local mode, plus some advice on how to test that Storm is populating the Memcache store with good counts.

# Have Fun!

Please send any questions on the code you see here to sritchie@twitter.com, or send me a tweet at @sritchie. Once you get this code compiling and running, you'll be cooking with gas!

Source
package.scala
Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. example
  2. AnyRef
  3. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Value Members

  1. object ExeStorm

  2. object Memcache

    TODO: Delete when https://github.

  3. object Serialization

    Serialization is often the most important (and hairy) configuration issue for any system that needs to store its data over the long term.

  4. object StatusStreamer

  5. object StormRunner

    The following object contains code to execute the Summingbird WordCount job defined in ExampleJob.

  6. package javaapi

Inherited from AnyRef

Inherited from Any

Ungrouped