**[KIP-145](https://cwiki.apache.org/confluence/display/KAFKA/KIP-145+-+Expose+Record+Headers+in+Kafka+Connect) has been accepted, and this PR implements KIP-145 except without the SMTs.**

Changed the Connect API and runtime to support message headers as described in [KIP-145](https://cwiki.apache.org/confluence/display/KAFKA/KIP-145+-+Expose+Record+Headers+in+Kafka+Connect).

The new `Header` interface defines an immutable representation of a Kafka header (key-value pair) with support for the Connect value types and schemas. This interface provides methods for easily converting between many of the built-in primitive, structured, and logical data types.

The new `Headers` interface defines an ordered collection of headers and is used to track all headers associated with a `ConnectRecord` (and thus `SourceRecord` and `SinkRecord`). This does allow multiple headers with the same key. The `Headers` contains methods for adding, removing, finding, and modifying headers. Convenience methods allow connectors and transforms to easily use and modify the headers for a record.

A new `HeaderConverter` interface is also defined to enable the Connect runtime framework to be able to serialize and deserialize headers between the in-memory representation and Kafka’s byte[] representation. A new `SimpleHeaderConverter` implementation has been added, and this serializes to strings and deserializes by inferring the schemas (`Struct` header values are serialized without the schemas, so they can only be deserialized as `Map` instances without a schema.) The `StringConverter`, `JsonConverter`, and `ByteArrayConverter` have all been extended to also be `HeaderConverter` implementations. Each connector can be configured with a different header converter, although by default the `SimpleHeaderConverter` is used to serialize header values as strings without schemas.

Unit and integration tests are added for `ConnectHeader` and `ConnectHeaders`, the two implementation classes for headers. Additional test methods are added for the methods added to the `Converter` implementations. Finally, the `ConnectRecord` object is already used heavily, so only limited tests need to be added while quite a few of the existing tests already cover the changes.

Author: Randall Hauch <rhauch@gmail.com>

Reviewers: Arjun Satish <arjun@confluent.io>, Ted Yu <yuzhihong@gmail.com>, Magesh Nandakumar <magesh.n.kumar@gmail.com>, Konstantine Karantasis <konstantine@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #4319 from rhauch/kafka-5142-b

This PR implements the JIRA issue [KAFKA-4029: SSL support for Connect REST API](https://issues.apache.org/jira/browse/KAFKA-4029) / [KIP-208](https://cwiki.apache.org/confluence/display/KAFKA/KIP-208%3A+Add+SSL+support+to+Kafka+Connect+REST+interface).

Summary of the main changes:
- Jetty `HttpClient` is used as HTTP client instead of the one shipped with Java. That allows to keep the SSL configuration for Server and Client be in single place (both use the Jetty `SslContextFactory`). It also has much richer configuration than the JDK client (it is easier to configure things such as supported cipher suites etc.).
- The `RestServer` class has been broker into 3 parts. `RestServer` contains the server it self. `RestClient` contains the HTTP client used for forwarding requests etc. and `SSLUtils` contain some helper classes for configuring SSL. One of the reasons for this was Findbugs complaining about the class complexity.
- A new method `valuesWithPrefixAllOrNothing` has been added to `AbstractConfig` to make it easier to handle the situation that we want to use either only the prefixed SSL options or only the non-prefixed. But not mixed them.

Author: Jakub Scholz <www@scholzj.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #4429 from scholzj/kip-208

There are more methods that had to be touched than I anticipated when writing [the KIP](https://cwiki.apache.org/confluence/display/KAFKA/KIP-215%3A+Add+topic+regex+support+for+Connect+sinks).

The implementation here is now complete and includes a test that verifies that there's a call to `consumer.subscribe(Pattern, RebalanceHandler)` when `topics.regex` is provided.

Author: Jeff Klukas <jeff@klukas.net>

Reviewers: Randall Hauch <rhauch@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #4151 from jklukas/connect-topics.regex

Author: Adem Efe Gencer <agencer@linkedin.com>

Reviewers: Jiangjie Qin <becket.qin@gmail.com>

Closes #4170 from efeg/bug/typoFix

This just adds `<p>`/`</p>` to paragraphs in Kafak Connect docs.

ewencp are you a good person to review this?

Author: Tom Bentley <tbentley@redhat.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #4008 from tombentley/MINOR-connect-docs-paragraph

Author: Robin Moffatt <robin@rmoff.net>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #3576 from rmoff/patch-1

Author: ppatierno <ppatierno@live.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #3269 from ppatierno/connect-doc

Author: Guozhang Wang <wangguoz@gmail.com>

Reviewers: Derrick Or <derrickor@gmail.com>, Ismael Juma <ismael@juma.me.uk>

Closes #3214 from guozhangwang/KMinor-doc-java-brush

Author: Gwen Shapira <cshapi@gmail.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #2721 from gwenshap/improve_smt_docs

KAFKA-4567 - Connect Producer and Consumer ignore ssl parameters configured for worker

Added brief explanation to the docs about parameter inheritance of Kafka consumers and producers from the worker config.

Author: Sönke Liebau <soenke.liebau@opencore.com>

Reviewers: Gwen Shapira

Closes #2511 from soenkeliebau/KAFKA-4567

Renames `HoistToStruct` SMT to `HoistField`.

Adds the following SMTs:
`ExtractField`
`MaskField`
`RegexRouter`
`ReplaceField`
`SetSchemaMetadata`
`ValueToKey`

Adds HTML doc generation and updates to `connect.html`.

Author: Shikhar Bhushan <shikhar@confluent.io>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #2374 from shikhar/more-smt

- Seperate Streams documentation out to a standalone page.
- Setup templates to use handlebars.js
- Create template variables to swap in frequently updated values like version number from a single file templateData.js

Author: Derrick Or <derrickor@gmail.com>

Reviewers: Guozhang Wang <wangguoz@gmail.com>

Closes #2245 from derrickdoo/docTemplates

And improve readability by adding proper punctuations.

Author: Vahid Hashemian <vahidhashemian@us.ibm.com>

Reviewers: Jason Gustafson <jason@confluent.io>

Closes #2002 from vahidhashemian/doc/fix_typos

Update documentation for Kafka Connect distributed’s config.storage.topic, offset.storage.topic, and status.storage.topic configuration values to indicate that all three should refer to compacted topics.

Author: Mathieu Fenniak <mathieu.fenniak@replicon.com>

Reviewers: Jason Gustafson

Closes #1832 from mfenniak/kafka-connect-topic-docs

Author: Ewen Cheslack-Postava <me@ewencp.org>

Reviewers: Shikhar Bhushan, Gwen Shapira

Closes #1721 from ewencp/kafka-3845-per-connector-converters

Fix some trivial misses.
ewencp

Author: uchan-nos <uchan0@gmail.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1496 from uchan-nos/docfix-connect

This is a follow up of KAKFA-3421 to update the connect developer guide to include the configuration validation. Also includes a couple of minor fixes.

Author: Liquan Pei <liquanpei@gmail.com>

Reviewers: Jason Gustafson <jason@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1366 from Ishiihara/connect-dev-doc

Author: Jason Gustafson <jason@confluent.io>

Reviewers: Roger Hoover <roger.hoover@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1358 from hachikuji/KAFKA-3583

KAFKA-3421; Follow up to fix name of SourceTask method and add documentation of connector status REST API

Author: Ewen Cheslack-Postava <me@ewencp.org>

Reviewers: Ismael Juma <ismael@juma.me.uk>

Closes #1355 from ewencp/kafka-3421-follow-up

Author: Kaufman Ng <kaufman@confluent.io>

Reviewers: Ismael Juma <ismael@juma.me.uk>

Closes #1351 from coughman/KAFKA-3681-connect-doc-formatting

ewencp gwenshap Docs. I also tried to clean up some typos. However, it seems that the we don't have two words without space in between in the source yet they showed up as no space in between in the generated doc.

Author: Liquan Pei <liquanpei@gmail.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1227 from Ishiihara/config-doc

This PR fixes 8 typos in HTML files of `docs` module. I wrote explicitly here since Github sometimes does not highlight the corrections on long lines correctly.
- docs/api.html: compatability => compatibility
- docs/connect.html: simultaneoulsy => simultaneously
- docs/implementation.html: LATIEST_TIME => LATEST_TIME, nPartions => nPartitions
- docs/migration.html: Decomission => Decommission
- docs/ops.html: stoping => stopping, ConumserGroupCommand => ConsumerGroupCommand, youre => you're

Author: Dongjoon Hyun <dongjoon@apache.org>

Reviewers: Ismael Juma

Closes #1138 from dongjoon-hyun/KAFKA-3461

Author: Sasaki Toru <sasakitoa@nttdata.co.jp>

Reviewers: Gwen Shapira

Closes #1003 from sasakitoa/Fix_typo_in_docs

Often it is useful to link to a specific header within the documentation. Especially when referencing docs in the mailing lists.

This adds anchors and links for all headers in the docs.

Author: Grant Henke <granthenke@gmail.com>

Reviewers: Jun Rao <junrao@gmail.com>

Closes #498 from granthenke/doc-links

Author: Ewen Cheslack-Postava <me@ewencp.org>

Reviewers: Gwen Shapira

Closes #475 from ewencp/kafka-2379-connect-docs