Kafka Connect can't find connector

kafka connect kubernetes
kafka connect example
kafka connect environment variables
kafka connect docker
kafka connect key converter
kafka connect use cases
kafka-connect github
kafka connect ppt

I'm trying to use the Kafka Connect Elasticsearch connector, and am unsuccessful. It is crashing with the following error:

[2018-11-21 14:48:29,096] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:108)
java.util.concurrent.ExecutionException: org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches io.confluent.connect.elasticsearch.ElasticsearchSinkConnector , available connectors are: PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSinkConnector, name='org.apache.kafka.connect.file.FileStreamSinkConnector', version='1.0.1', encodedVersion=1.0.1, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSourceConnector, name='org.apache.kafka.connect.file.FileStreamSourceConnector', version='1.0.1', encodedVersion=1.0.1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockConnector, name='org.apache.kafka.connect.tools.MockConnector', version='1.0.1', encodedVersion=1.0.1, type=connector, typeName='connector', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSinkConnector, name='org.apache.kafka.connect.tools.MockSinkConnector', version='1.0.1', encodedVersion=1.0.1, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSourceConnector, name='org.apache.kafka.connect.tools.MockSourceConnector', version='1.0.1', encodedVersion=1.0.1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.SchemaSourceConnector, name='org.apache.kafka.connect.tools.SchemaSourceConnector', version='1.0.1', encodedVersion=1.0.1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSinkConnector, name='org.apache.kafka.connect.tools.VerifiableSinkConnector', version='1.0.1', encodedVersion=1.0.1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSourceConnector, name='org.apache.kafka.connect.tools.VerifiableSourceConnector', version='1.0.1', encodedVersion=1.0.1, type=source, typeName='source', location='classpath'}

I've got a build for the plugin unzipped in a kafka subfolder, and have the following line in connect-standalone.properties:

plugin.path=/opt/kafka/plugins/kafka-connect-elasticsearch-5.0.1/src/main/java/io/confluent/connect/elasticsearch

I can see the various connectors inside that folder, but Kafka Connect does not load them; but it does load the standard connectors, like this:

[2018-11-21 14:56:28,258] INFO Added plugin 'org.apache.kafka.connect.transforms.Cast$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:136)
[2018-11-21 14:56:28,259] INFO Added aliases 'FileStreamSinkConnector' and 'FileStreamSink' to plugin 'org.apache.kafka.connect.file.FileStreamSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:335)
[2018-11-21 14:56:28,260] INFO Added aliases 'FileStreamSourceConnector' and 'FileStreamSource' to plugin 'org.apache.kafka.connect.file.FileStreamSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:335)

How can I properly register the connectors?

Confluent Connectors, We Have Connectors for the Most Popular Data Sources and Sinks. Kafka Connect is an integral component of an ETL pipeline, when combined with Kafka and a stream processing framework. Kafka Connect can be deployed either as a standalone process that runs jobs on a single machine (for example, log collection), or as a distributed, scalable, fault-tolerant service supporting an entire organization.

Getting Started with Kafka Connect, No, it is not required, but it is recommended if you plan to use Avro for a data format. This is because it can help you with serialization and schema evolution as​  The Kafka connector can run in any Kafka Connect cluster, and can send data to a Snowflake account on any supported cloud platform.

The plugin path must load JAR files, containing compiled code, not raw Java classes of the source code (src/main/java).

It also needs to be the parent directory of other directories which are containing those plug-ins.

plugin.path=/opt/kafka-connect/plugins/

Where

$ ls - lR /opt/kafka-connect/plugins/
kafka-connect-elasticsearch-x.y.z/
    file1.jar
    file2.jar 
    etc

Ref - Manually installing Community Connectors

The Kafka Connect startup scripts in the Confluent Platform automatically (used to?) read all folders that match share/java/kafka-connect-*, too, so that's one way to go. At least, it will continue doing so, if you include the path to the share/java folder of the Confluent package installation in the plugin path as well

If you are not very familiar with Maven, or even if you are, then you actually cannot just clone the Elasticsearch connector repo and build the master branch; it has prerequisites of first Kafka, then the common Confluent repo first. Otherwise, you must checkout a Git tag like 5.0.1-post that matches a Confluent release.

An even simpler option would be to grab the package using Confluent Hub CLI

And if none of that works, just downloading the Confluent Platform and using the Kafka Connect scripts would be the most easiest. This does not imply you need to use the Kafka or Zookeeper configurations from that

Kafka Connect FAQ, Kafka Connect is a framework to stream data into and out of Apache Kafka®. The Confluent Platform ships with several built-in connectors that can be used to  Kafka Connect can automatically create the internal topics when it starts up, using the Connect distributed worker configuration properties to specify the topic names, replication factor, and number of partitions for these topics.

Kafka Connect Concepts, Kafka Connect can be deployed either as a standalone process that runs jobs on a single machine (for example, log collection), or as a distributed, scalable,  Kafka Connect serves as a robust framework for hosting connectors. An ecosystem of pre-built connectors makes it easy for you to get data to where you need it, in the form that you require.

Kafka Connect, It makes it simple to quickly define connectors that move large data sets into and out of Kafka. Kafka Connect can ingest entire databases or collect metrics from all​  We use Apache Kafka Connect for streaming data between Apache Kafka and other systems, scalably as well as reliably. Moreover, connect makes it very simple to quickly define Kafka connectors that move large collections of data into and out of Kafka.

Kafka Connect Architecture, The worker model allows Kafka Connect to scale to the application. It can run scaled down to a single worker process that also acts as its own coordinator, or in​  Kafka Connect – an open source component of the Apache Kafka project – facilitates integrations between Kafka clusters and external data sources and sinks. The technology uses reusable open source

Comments
  • For the JDBC Connect, I've noticed drivers can even be placed in a subfolder of that. E.g lib/drivers
  • Building it myself results in this: [ERROR] [ERROR] Some problems were encountered while processing the POMs: [FATAL] Non-resolvable parent POM for io.confluent:kafka-connect-elasticsearch:[unknown-version]: Could not transfer artifact io.confluent:common:pom:5.0.1 from/to confluent (${confluent.maven.repo}):
  • ¯_(ツ)_/¯ that's a good reason to use the pre-built version in Confluent Platform ;) It's open source, it's free to use. If you really don't want to, you can d/l it and simply extract the JAR and deploy it in your existing installation.
  • OK, I've saved the pre-built version in /var/confluentinc-kafka-connect-elasticsearch-5.0.0/. In my config, I have this line: plugin.path=/var/silverbolt/confluentinc-kafka-connect-elasticsearch-5.0.0/ . I'm still getting the same error about no matching connector class.
  • I built the JAR successfully, moved it into the a folder under /plugins/, added the path to the config, and am still getting the same "failed to find any class" error.
  • I believe the Elastic connector actually makes a tar.gz file that you need to extract. It doesn't create just one JAR with all the needed classes
  • I'm looking at the target folder, and it created a kafka-connect-elasticsearch-5.0.1.jar file. No tar.gz that I can see.
  • For me, adding a comma at the end solved the issue plugin.path=/opt/bitnami/kafka/connectors,. Without the comma, kafka keep complaining failed to find the class.