I'm using Spark 2.3.0, Scala 2.11.8 and Kafka and I'm trying to write into parquet files all the messages from Kafka with Structured Streaming but for each query that my implementation does the total time for each one increase a lot Spark Stages Image. I would like to know why this happens, I tried with different possibles triggers (Continues,0 seconds, 1 seconds, 10 seconds,10 minutes, etc) and always I get the same behavior. I wrote the code and images related to the problem on https://stackoverflow.com/questions/51019555/each-query-takes-more-time-using-structured-streaming-with-spark
30 People are following this question.
