Heron supports deployment on Apache Aurora out of the box. A step by step guide on how to setup Heron with Apache Aurora locally can be found in Setting up Heron with Aurora Cluster Locally on Linux. You can also run Heron on a local scheduler.
How Heron on Aurora Works
Aurora doesn’t have a Heron scheduler per se. Instead, when a topology is
submitted to Heron,
heron cli interacts with Aurora to automatically deploy
all the components necessary to manage
To run Heron on Aurora, you’ll need to set up a ZooKeeper cluster and configure Heron to communicate with it. Instructions can be found in Setting up ZooKeeper.
To deploy Heron, the Aurora cluster needs access to the Heron core binary, which can be hosted wherever you’d like, so long as it’s accessible to Aurora (for example in Amazon S3 or using a local blob storage solution). You can download the core binary from github or build it using the instructions in Creating a New Heron Release.
Command for fetching the binary is in the
heron.aurora config file. By default it is
curl command to fetch the binary. For example, if the binary is hosted in
HDFS, you need to change the fetch user package command in
heron.aurora to use the
hdfs command instead of
heron.aurora example binary fetch using HDFS
fetch_heron_system = Process( name = 'fetch_heron_system', cmdline = 'hdfs dfs -get %s %s && tar zxf %s' % (heron_core_release_uri, core_release_file, core_release_file) )
Once your Heron binaries are hosted somewhere that is accessible to Aurora, you should run tests to ensure that Aurora can successfully fetch them.
Uploading the Topologies
Heron uses an uploader to upload the topology to a shared location so that a worker can fetch
the topology to its sandbox. The configuration for an uploader is in the
config file. For distributed Aurora deployments, Heron can use
Details on configuring the uploaders can be found in the documentation for the
HDFS and S3 uploaders.
After configuring an uploader, the
heron.aurora config file needs to be modified accordingly to
fetch the topology.
heron.aurora example topology fetch using HDFS
fetch_user_package = Process( name = 'fetch_user_package', cmdline = 'hdfs dfs -get %s %s && tar zxf %s' % (heron_topology_jar_uri, topology_package_file, topology_package_file) )
Aurora Scheduler Configuration
To configure Heron to use Aurora scheduler, modify the
config file specific for the Heron cluster. The following must be specified
for each cluster:
heron.class.scheduler— Indicates the class to be loaded for Aurora scheduler. You should set this to
heron.class.launcher— Specifies the class to be loaded for launching and submitting topologies. To configure the Aurora launcher, set this to
heron.package.core.uri— Indicates the location of the heron core binary package. The local scheduler uses this URI to download the core package to the working directory.
heron.directory.sandbox.java.home— Specifies the java home to be used when running topologies in the containers.
heron.scheduler.is.service— This config indicates whether the scheduler is a service. In the case of Aurora, it should be set to
Example Aurora Scheduler Configuration
# scheduler class for distributing the topology for execution heron.class.scheduler: com.twitter.heron.scheduler.aurora.AuroraScheduler # launcher class for submitting and launching the topology heron.class.launcher: com.twitter.heron.scheduler.aurora.AuroraLauncher # location of the core package heron.package.core.uri: file:///vagrant/.herondata/dist/heron-core-release.tar.gz # location of java - pick it up from shell environment heron.directory.sandbox.java.home: /usr/lib/jvm/java-1.8.0-openjdk-amd64/ # Invoke the IScheduler as a library directly heron.scheduler.is.service: False
Working with Topologies
After setting up ZooKeeper and generating an Aurora-accessible Heron core binary
release, any machine that has the
heron cli tool can be used to manage Heron
topologies (i.e. can submit topologies, activate and deactivate them, etc.).
The most important thing at this stage is to ensure that
heron cli is available
across all machines. Once the cli is available, Aurora as a scheduler
can be enabled by specifying the proper configuration when managing topologies.