• Create
    • Ask a question
    • Create an article
    • Topics
    • Questions
    • Articles
    • Users
    • Badges
  • Sign in
  • Home /
avatar image
0

Kafka Streams: Implement a simple KeyValueStore where I can put and get data

kafkajavakafka streaming
Question by Irshad P I · Oct 11, 2018 at 06:53 AM ·

I have a Kafka streams application which operates on the incoming state and need to store the state before writing to the next topic. The write should occur only after the state is updated in local store.

Something like this.

stream.map(this::getAndUpdateState)
          .map(this::processStateAndEvent)
          .to("topicname");

So that in getAndUpdateState() I can do like

state = store.get(key); // or new if null
state = updateState(state, event);  // update changes to state
store.put(key, state);  // write back the state
return state;

How do I implement that simple get() and put() operation on a kafka store? I already tried using KeyValueStore but it had issues as I had to add it a source and sink processor and all.

Alternatively, a way to get and put in kafka using a KTable or some other concept is also fine.

Add comment
Comment
10 |600 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Sort

  • Votes
  • Created
  • Oldest

Your answer

Hint: You can notify a user about this post by typing @username

Up to 2 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

Follow this Question

12 People are following this question.

avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image

Related Questions

How to use Structured Streaming with Kafka Direct Stream? 1 Answer

spark & kafka connecion issue 1 Answer

Kafka Consumer Error 2 Answers

Wondering if there is a scheduled upgrade to Java 1.8? 1 Answer

org.apache.spark.SparkException:Failed to get broadcast_25_piece0 of broadcast_25 when try to read broadcast variable updated at regular interval 0 Answers

  • Product
    • Databricks Cloud
    • FAQ
  • Spark
    • About Spark
    • Developer Resources
    • Community + Events
  • Services
    • Certification
    • Spark Support
    • Spark Training
  • Company
    • About Us
    • Team
    • News
    • Contact
  • Careers
  • Blog

Databricks Inc.
160 Spear Street, 13th Floor
San Francisco, CA 94105

info@databricks.com
1-866-330-0121

  • Twitter
  • LinkedIn
  • Facebook
  • Facebook

© Databricks 2015. All rights reserved. Apache Spark and the Apache Spark Logo are trademarks of the Apache Software Foundation.

  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Create an article
  • Explore
  • Topics
  • Questions
  • Articles
  • Users
  • Badges