How to implement a custom scheduler in Heron?

I have read the Heron Documents about Implementing a Custom Scheduler. And I have known that I should implement some interfaces to implement a Custom Scheduler, such as ILauncher, IPacking, IScheduler and IUploader.

I have realized my CustomScheduler that implemented IScheduler interface, and I want to using LocalLauncher, LocalUploader and default Packing algorithm with my custom scheduler.

What's more, I modified the heron configuration file named scheduler.yaml that located at conf/local/ to use the custom scheduler. At the same time, I added the CustomScheduler.jar to heron-core/lib/scheduler/. However, there is something wrong as logs shows:

[2018-04-15 20:44:27 -0700] [STDERR] stderr: Exception in thread "main"   
[2018-04-15 20:44:27 -0700] [STDERR] stderr: com.twitter.heron.spi.scheduler.SchedulerException: Failed to instantiate scheduler using class 'com.zyt.heron.custom.scheduler.CustomScheduler'  
[2018-04-15 20:44:27 -0700] [STDERR] stderr:    at com.twitter.heron.scheduler.utils.LauncherUtils.getSchedulerInstance(LauncherUtils.java:120)  
[2018-04-15 20:44:27 -0700] [STDERR] stderr:    at com.twitter.heron.scheduler.SchedulerMain.runScheduler(SchedulerMain.java:382)  
[2018-04-15 20:44:27 -0700] [STDERR] stderr:    at com.twitter.heron.scheduler.SchedulerMain.main(SchedulerMain.java:218)  

what should I do to fix this problem? Thanks for your help!

The key is to provide your compiled .jar files to heron by copying your jar files into the .heron/lib/ folder.

I want to implement a custom scheduler and packing algorithm without recompiling Heron.

I have a project called "packing" that has the maven dependencies heron.api and heron.spi as requested.

My jar is called "packing-0.0.1-SNAPSHOT-jar-with-dendencies.jar built with maven. What works for me are the following steps summarized in a small shell-script. I run it from within my packing-project folder to the assembly-command works:

#!/bin/bash
mvn assembly:assembly
cp target/packing-0.0.1-SNAPSHOT-jar-with-dependencies.jar $HOME/.heron/lib/packing/
cp target/packing-0.0.1-SNAPSHOT-jar-with-dependencies.jar $HOME/.heron/lib/scheduler/
echo "updated packing plan and scheduler"

Then, simply add your custom packing or scheduler to your config:

Example from the packing.yaml:

heron.class.packing.algorithm: com.hcep.packing.CustomPacking

with com.hcep.packing being my package structure in my packing-project and CustomPacking my packing class.

incubator-heron/extending-heron-scheduler.md at master · apache , If you'd like to run Heron on a not-yet-supported system, such as Amazon ECS, you can create your own scheduler using Heron's spi, as detailed in the sections​  We design and implement L-Heron 1 based on Heron as shown in Fig. 5. To enable the load-aware online scheduling in L-Heron, several new components need to be deployed, including a custom scheduler, a schedule trigger, a schedule controller and load monitors.

The direct reason for your Exception is your custom scheduler is not packaged with the heron release.

To make your custom scheduler work, you can follow the steps below:

  1. add your implementation into the directory heron/schedulers/src/java/com/twitter/heron/scheduler
  2. update the BUILD file at directory heron/schedulers/src/java
  3. add your custom scheduler building target at tools/rules/heron_client.bzl and tools/rules/heron_core.bzl

Here is an example of how local scheduler is packaged: https://github.com/apache/incubator-heron/search?utf8=%E2%9C%93&q=heron-local-scheduler&type=

[incubator-heron] branch master updated: Joshfischer/update docs , Cluster Scheduling Heron supports two cluster schedulers out of the box: Running the Scheduler -To run the a custom scheduler, the implementation of the​  How to implement a custom scheduler in Heron? 0. Heron Failed to set packing plan for topology 'WordCountTopology' 0.

You can develop custom scheduler for Heron - please follow any of the implementations for a scheduler in the code base.

[incubator-heron] branch master updated: Joshfischer/newsite (#3280), custom-scheduler); info on the currently available schedulers +can be found in + +If you'd like to implement your own custom metrics handler  In its simplest form, a scheduler implementation is very easy: A loop to watch the unbound pods in the cluster through querying the apiserver. Some custom logic that finds the best node for a pod. A request to the bind endpoint on the apiserver.

Heron Documentation, Implementing a Custom Metrics Sink. Each Heron topology has its own centralized Metrics Manager (MM), which collects metrics from all instances in the​  It is possible to use YARN as Kubernetes custom scheduler. A few years back, Hortonworks tried to create a fork of Kubernetes with YARN scheduler but at that time support for multiple schedulers was not available in Kubernetes. Twitter Heron is another good example of this approach.

Config Java Source Code, heron. packing. src. java. com. twitter. heron. packing. utils. PackingUtils.java custom-scheduler.md; license.md; governance.md; codebase.md This class provides a * convenient way to create a topology config map by providing setter  Creating a custom task scheduler. To build your own custom task scheduler you would need to create a class that extends the System.Threading.Tasks.TaskScheduler class. So, to build a custom task scheduler, you would need to extend the TaskScheduler abstract class and override the following methods.

Heron: Is it Apache Storm 2.0?, The Heron system provides some enhanced features comparing with Storm: Easy to deploy scheduler and supports Mesos, YARN, or a custom  Implement Custom Editors To see the example, refer to the project on how to implement a custom editor similar to the build-in editor of a Scheduler in ASP.NET MVC applications. Fields that do not come from the ISchedulerEvent interface preserve their exact names.

Comments
  • I also have solved this question. Thanks.
  • What should I do to deploy my custom scheduler to Heron? Do I should upload my customScheduler.jar to heron-core.jar/lib/scheduler/? Is that right?
  • I have edited this question with more details. Thanks for you help.