Quick Start

This tutorial assumes you are starting from fresh and have no existing BookKeeper or ZooKeeper data.

Step 1: Download the binary

Download the stable version of DistributedLog and un-zip it.

// Download the binary `distributedlog-all-${gitsha}.zip`
> unzip distributedlog-all-${gitsha}.zip

Step 2: Start ZooKeeper & BookKeeper

DistributedLog uses ZooKeeper as the metadata store and BookKeeper as the log segment store. So you need to first start a zookeeper server and a few bookies if you don't already have one. You can use the dlog script in distributedlog-service package to get a standalone bookkeeper sandbox. It starts a zookeeper server and N bookies (N is 3 by default).

// Start the local sandbox instance at port `7000`
> ./distributedlog-service/bin/dlog local 7000
DistributedLog Sandbox is running now. You could access distributedlog://

Step 3: Create a DistributedLog namespace

Before using distributedlog, you need to create a distributedlog namespace to store your own list of streams. The zkServer for the local sandbox is and the bookkeeper's ledgers path is /ledgers. You could create a namespace pointing to the corresponding bookkeeper cluster.

> ./distributedlog-service/bin/dlog admin bind -l /ledgers -s -c distributedlog://
No bookkeeper is bound to distributedlog://
Created binding on distributedlog://

If you don't want to create a separated namespace, you could use the default namespace distributedlog://

Step 4: Create some log streams

Let's create 5 log streams, prefixed with messaging-test-.

> ./distributedlog-service/bin/dlog tool create -u distributedlog:// -r messaging-stream- -e 1-5

We can now see the streams if we run the list command from the tool.

> ./distributedlog-service/bin/dlog tool list -u distributedlog://
Streams under distributedlog:// :

Step 5: Start a write proxy

Now, lets start a write proxy server that serves writes to distributedlog namespace distributedlog:// The server listens on 8000 to accept fan-in write requests.

> ./distributedlog-service/bin/dlog-daemon.sh start writeproxy -p 8000 --shard-id 1 -sp 8001 -u distributedlog:// -mx -c `pwd`/distributedlog-service/conf/distributedlog_proxy.conf

Step 6: Tail reading records

The distributedlog tutorial has a multi-streams reader that will dump out received records to standard output.

> ./distributedlog-tutorials/distributedlog-basic/bin/runner run com.twitter.distributedlog.basic.MultiReader distributedlog:// messaging-stream-1,messaging-stream-2,messaging-stream-3,messaging-stream-4,messaging-stream-5

Step 7: Write some records

The distributedlog tutorial also has a multi-streams writer that will take input from a console and write it out as records to the distributedlog write proxy. Each line will be sent as a separate record.

Run the writer and type a few lines into the console to send to the server.

> ./distributedlog-tutorials/distributedlog-basic/bin/runner run com.twitter.distributedlog.basic.ConsoleProxyMultiWriter 'inet!' messaging-stream-1,messaging-stream-2,messaging-stream-3,messaging-stream-4,messaging-stream-5

If you have each of the above commands running in a different terminal then you should now be able to type messages into the writer terminal and see them appear in the reader terminal.