Skip to content
Snippets Groups Projects
  1. Jan 12, 2019
    • Colin Patrick McCabe's avatar
      KAFKA-7609; Add Protocol Generator for Kafka (#5893) · 71e85f5e
      Colin Patrick McCabe authored
      This patch adds a framework to automatically generate the request/response classes for Kafka's protocol. The code will be updated to use the generated classes in follow-up patches. Below is a brief summary of the included components:
      
      **buildSrc/src**
      The message generator code is here.  This code is automatically re-run by gradle when one of the schema files changes.  The entire directory is processed at once to minimize the number of times we have to start a new JVM.  We use Jackson to translate the JSON files into Java objects.
      
      **clients/src/main/java/org/apache/kafka/common/protocol/Message.java**
      This is the interface implemented by all automatically generated messages.
      
      **clients/src/main/java/org/apache/kafka/common/protocol/MessageUtil.java**
      Some utility functions used by the generated message code.
      
      **clients/src/main/java/org/apache/kafka/common/protocol/Readable.java, Writable.java, ByteBufferAccessor.java**
      The generated message code uses these classes for writing to a buffer.
      
      **clients/src/main/message/README.md**
      This README file explains how the JSON schemas work.
      
      **clients/src/main/message/\*.json**
      The JSON files in this directory implement every supported version of every Kafka API.  The unit tests automatically validate that the generated schemas match the hand-written schemas in our code.  Additionally, there are some things like request and response headers that have schemas here.
      
      **clients/src/main/java/org/apache/kafka/common/utils/ImplicitLinkedHashSet.java**
      I added an optimization here for empty sets.  This is useful here because I want all messages to start with empty sets by default prior to being loaded with data.  This is similar to the "empty list" optimizations in the `java.util.ArrayList` class.
      
      Reviewers: Stanislav Kozlovski <stanislav_kozlovski@outlook.com>, Ismael Juma <ismael@juma.me.uk>, Bob Barrett <bob.barrett@outlook.com>, Jason Gustafson <jason@confluent.io>
      71e85f5e
  2. Jun 12, 2018
  3. Apr 23, 2018
    • Debasish Ghosh's avatar
      KAFKA-6670: Implement a Scala wrapper library for Kafka Streams · b2e4db01
      Debasish Ghosh authored
      This PR implements a Scala wrapper library for Kafka Streams. The library is implemented as a project under streams, namely `:streams:streams-scala`. The PR contains the following:
      
      * the library implementation of the wrapper abstractions
      * the test suite
      * the changes in `build.gradle` to build the library jar
      
      The library has been tested running the tests as follows:
      
      ```
      $ ./gradlew -Dtest.single=StreamToTableJoinScalaIntegrationTestImplicitSerdes streams:streams-scala:test
      $ ./gradlew -Dtest.single=StreamToTableJoinScalaIntegrationTestImplicitSerdesWithAvro streams:streams-scala:test
      $ ./gradlew -Dtest.single=WordCountTest streams:streams-scala:test
      ```
      
      Author: Debasish Ghosh <ghosh.debasish@gmail.com>
      Author: Sean Glover <seglo@randonom.com>
      
      Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Ismael Juma <ismael@juma.me.uk>, John Roesler <john@confluent.io>, Damian Guy <damian@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
      
      Closes #4756 from debasishg/scala-streams
      b2e4db01
  4. Mar 15, 2018
    • John Roesler's avatar
      MINOR: Streams system tests fixes/updates (#4689) · 7006d0f5
      John Roesler authored
      Some changes required to get the Streams system tests working via Docker
      
      To test:
      
      TC_PATHS="tests/kafkatest/tests/streams" bash tests/docker/run_tests.sh
      
      That command will take about 3.5 hours, and should pass. Note there are a couple of ignored tests.
      
      Reviewers: Guozhang Wang <wangguoz@gmail.com>, Bill Bejeck <bill@confluent.io>
      7006d0f5
  5. Aug 04, 2017
  6. Jul 20, 2017
    • Ismael Juma's avatar
      MINOR: Simplify SensorAccess usage · 84d2b6a0
      Ismael Juma authored
      I was investigating an exception in this code and found a few
      opportunities for making it clearer.
      
      I also added the `out` folder to `.gitignore` as IntelliJ sometimes
      uses that as the build folder.
      
      Author: Ismael Juma <ismael@juma.me.uk>
      
      Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
      
      Closes #3552 from ijuma/minor-quota-improvements
      84d2b6a0
  7. May 10, 2017
    • Ewen Cheslack-Postava's avatar
      MINOR: Add a release script that helps generate release candidates. · 0eedd6d8
      Ewen Cheslack-Postava authored
      Author: Ewen Cheslack-Postava <me@ewencp.org>
      
      Reviewers: Gwen Shapira
      
      Closes #2795 from ewencp/release-script and squashes the following commits:
      
      f1d0590 [Ewen Cheslack-Postava] Don't expose promotion to the user since it is not implemented yet.
      1a6947a [Ewen Cheslack-Postava] Handle cleanup if there's a failure during generation of release notes.
      fa58401 [Ewen Cheslack-Postava] Fix hard-coded uses of trunk
      639bcca [Ewen Cheslack-Postava] Try to cleanup after most failures.
      a3a7245 [Ewen Cheslack-Postava] Fix SCRIPT_DIR to be an absolute path so git clones against the REPO work when it is also your cwd
      de54c97 [Ewen Cheslack-Postava] Load/save preferences in a .release-settings.json file so you don't have to keep entering the same info repeatedly
      b559a61 [Ewen Cheslack-Postava] Check that the user doesn't have any oustanding diffs before starting the rest of the script
      ff0b330 [Ewen Cheslack-Postava] Store original starting branch to switch back to instead of using a default.
      b793562 [Ewen Cheslack-Postava] Use 2.12 instead of specific Scala version so we use the default 2.12 version.
      382b7f9 [Ewen Cheslack-Postava] MINOR: Add a release script that helps generate release candidates.
      0eedd6d8
  8. Feb 21, 2017
    • Christopher L. Shannon's avatar
      MINOR: Add build_eclipse to .gitignore · f48e8c1a
      Christopher L. Shannon authored
      build_eclipse is the configured output directory for eclipse when using
      the gradle eclipse plugin and should be ignored
      
      Author: Christopher L. Shannon <christopher.l.shannon@gmail.com>
      
      Reviewers: Ismael Juma <ismael@juma.me.uk>
      
      Closes #2569 from cshannon/eclipse-gitignore
      f48e8c1a
  9. Dec 20, 2016
    • Kamil Szymanski's avatar
      MINOR: Remove incomplete gradle wrapper infrastructure · 13586022
      Kamil Szymanski authored
      Since jar files (including gradle-wrapper.jar) have to
      be excluded from source releases, current gradle wrapper
      infrastructure is incomplete rendering it unusable and
      confusing (I expected gradlew script to work without
      having to run default gradle task to download
      gradle-wrapper.jar).
      
      Author: Kamil Szymanski <kamil.szymanski.dev@gmail.com>
      
      Reviewers: Gwen Shapira <cshapi@gmail.com>, Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>
      
      Closes #1722 from kamilszymanski/gradlew-cleanup
      13586022
  10. Aug 26, 2016
    • Shikhar Bhushan's avatar
      KAFKA-4042: Contain connector & task start/stop failures within the Worker · 71f7e7c3
      Shikhar Bhushan authored
      Invoke the statusListener.onFailure() callback on start failures so that the statusBackingStore is updated. This involved a fix to the putSafe() functionality which prevented any update that was not preceded by a (non-safe) put() from completing, so here when a connector or task is transitioning directly to FAILED.
      
      Worker start methods can still throw if the same connector name or task ID is already registered with the worker, as this condition should not happen.
      
      Author: Shikhar Bhushan <shikhar@confluent.io>
      
      Reviewers: Jason Gustafson <jason@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
      
      Closes #1778 from shikhar/distherder-stayup-take4
      71f7e7c3
  11. Aug 07, 2016
    • Kaufman Ng's avatar
      KAFKA-3479: Add new consumer metrics documentation · 6b256481
      Kaufman Ng authored
      added new consumer metrics section
      refactored common metrics into new section
      updated TOC
      
      Author: Kaufman Ng <kaufman@confluent.io>
      
      Reviewers: Jason Gustafson <jason@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
      
      Closes #1361 from coughman/KAFKA-3479-consumer-metrics-doc
      6b256481
  12. May 06, 2016
    • Geoff Anderson's avatar
      KAFKA-3592: System test - configurable paths · 54092c12
      Geoff Anderson authored
      This patch adds logic for the following:
      - remove hard-coded paths to various scripts and jars in kafkatest service classes
      - provide a mechanism for overriding path resolution logic with a "pluggable" path resolver class
      
      Author: Geoff Anderson <geoff@confluent.io>
      
      Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
      
      Closes #1245 from granders/configurable-install-path
      54092c12
  13. Mar 10, 2016
  14. Nov 10, 2015
  15. Nov 09, 2015
  16. Oct 02, 2015
  17. Aug 03, 2015
    • Stevo Slavić's avatar
      MINOR: Added to .gitignore Kafka server logs directory · cd3dc7a5
      Stevo Slavić authored
      When running Kafka server from sources, logs directory gets created in root of repository, and kafka server logs end up there. Currently that directory is not ignored by git.
      
      This change adds root logs directory to .gitignore so that Kafka server logs are ignored and do not get tracked by git.
      
      Author: Stevo Slavić <sslavic@gmail.com>
      
      Reviewers: Ismael Juma
      
      Closes #94 from sslavic/patch-7 and squashes the following commits:
      
      c7b62a7 [Stevo Slavić] MINOR: Added to .gitignore Kafka server logs
      cd3dc7a5
  18. Jul 29, 2015
    • Geoff Anderson's avatar
      KAFKA-2276; KIP-25 initial patch · e43c9aff
      Geoff Anderson authored
      Initial patch for KIP-25
      
      Note that to install ducktape, do *not* use pip to install ducktape. Instead:
      
      ```
      $ git clone gitgithub.com:confluentinc/ducktape.git
      $ cd ducktape
      $ python setup.py install
      ```
      
      Author: Geoff Anderson <geoff@confluent.io>
      Author: Geoff <granders@gmail.com>
      Author: Liquan Pei <liquanpei@gmail.com>
      
      Reviewers: Ewen, Gwen, Jun, Guozhang
      
      Closes #70 from granders/KAFKA-2276 and squashes the following commits:
      
      a62fb6c [Geoff Anderson] fixed checkstyle errors
      a70f0f8 [Geoff Anderson] Merged in upstream trunk.
      8b62019 [Geoff Anderson] Merged in upstream trunk.
      47b7b64 [Geoff Anderson] Created separate tools jar so that the clients package does not pull in dependencies on the Jackson JSON tools or argparse4j.
      a9e6a14 [Geoff Anderson] Merged in upstream changes
      d18db7b [Geoff Anderson] fixed :rat errors (needed to add licenses)
      321fdf8 [Geoff Anderson] Ignore tests/ and vagrant/ directories when running rat build task
      795fc75 [Geoff Anderson] Merged in changes from upstream trunk.
      1d93f06 [Geoff Anderson] Updated provisioning to use java 7 in light of KAFKA-2316
      2ea4e29 [Geoff Anderson] Tweaked README, changed default log collection behavior on VerifiableProducer
      0eb6fdc [Geoff Anderson] Merged in system-tests
      69dd7be [Geoff Anderson] Merged in trunk
      4034dd6 [Geoff Anderson] Merged in upstream trunk
      ede6450 [Geoff] Merge pull request #4 from confluentinc/move_muckrake
      7751545 [Geoff Anderson] Corrected license headers
      e6d532f [Geoff Anderson] java 7 -> java 6
      8c61e2d [Geoff Anderson] Reverted jdk back to 6
      f14c507 [Geoff Anderson] Removed mode = "test" from Vagrantfile and Vagrantfile.local examples. Updated testing README to clarify aws setup.
      98b7253 [Geoff Anderson] Updated consumer tests to pre-populate kafka logs
      e6a41f1 [Geoff Anderson] removed stray println
      b15b24f [Geoff Anderson] leftover KafkaBenchmark in super call
      0f75187 [Geoff Anderson] Rmoved stray allow_fail. kafka_benchmark_test -> benchmark_test
      f469f84 [Geoff Anderson] Tweaked readme, added example Vagrantfile.local
      3d73857 [Geoff Anderson] Merged downstream changes
      42dcdb1 [Geoff Anderson] Tweaked behavior of stop_node, clean_node to generally fail fast
      7f7c3e0 [Geoff Anderson] Updated setup.py for kafkatest
      c60125c [Geoff Anderson] TestEndToEndLatency -> EndToEndLatency
      4f476fe [Geoff Anderson] Moved aws scripts to vagrant directory
      5af88fc [Geoff Anderson] Updated README to include aws quickstart
      e5edf03 [Geoff Anderson] Updated example aws Vagrantfile.local
      96533c3 [Geoff] Update aws-access-keys-commands
      25a413d [Geoff] Update aws-example-Vagrantfile.local
      884b20e [Geoff Anderson] Moved a bunch of files to kafkatest directory
      fc7c81c [Geoff Anderson] added setup.py
      632be12 [Geoff] Merge pull request #3 from confluentinc/verbose-client
      51a94fd [Geoff Anderson] Use argparse4j instead of joptsimple. ThroughputThrottler now has more intuitive behavior when targetThroughput is 0.
      a80a428 [Geoff Anderson] Added shell program for VerifiableProducer.
      d586fb0 [Geoff Anderson] Updated comments to reflect that throttler is not message-specific
      6842ed1 [Geoff Anderson] left out a file from last commit
      1228eef [Geoff Anderson] Renamed throttler
      9100417 [Geoff Anderson] Updated command-line options for VerifiableProducer. Extracted throughput logic to make it reusable.
      0a5de8e [Geoff Anderson] Fixed checkstyle errors. Changed name to VerifiableProducer. Added synchronization for thread safety on println statements.
      475423b [Geoff Anderson] Convert class to string before adding to json object.
      bc009f2 [Geoff Anderson] Got rid of VerboseProducer in core (moved to clients)
      c0526fe [Geoff Anderson] Updates per review comments.
      8b4b1f2 [Geoff Anderson] Minor updates to VerboseProducer
      2777712 [Geoff Anderson] Added some metadata to producer output.
      da94b8c [Geoff Anderson] Added number of messages option.
      07cd1c6 [Geoff Anderson] Added simple producer which prints status of produced messages to stdout.
      a278988 [Geoff Anderson] fixed typos
      f1914c3 [Liquan Pei] Merge pull request #2 from confluentinc/system_tests
      81e4156 [Liquan Pei] Bootstrap Kafka system tests
      e43c9aff
  19. Mar 05, 2015
  20. Feb 18, 2015
  21. Dec 05, 2014
  22. Apr 09, 2014
  23. Mar 19, 2014
  24. Dec 15, 2012
  25. Oct 10, 2011
  26. Oct 07, 2011
  27. Aug 03, 2011
Loading