We started using structured streaming. When I submit the structured streaming job, I see the job is running on Spark UI. I go to running application, i see the tabs Stages, Storage, Environment, Executors, SQL. But i don't see the Streaming tab anymore which used to be available on 1.4, 1.6 version. Is Streaming tab not available for Structured Streaming?
Answer by Burak Yavuz · 2 days ago
Hi @pritpalm. For monitoring on open source Spark you have a couple of programmatic options:
1. Once you start a streaming query, you may call
recentProgress (Scala, Python) to see the amount of data processed by the stream. You may call
status to learn what the stream is doing at that exact moment.
2. You may write a StreamingQueryListener that can push the progress updates to your monitoring service.
Hope that helps!
NPE: guava.reflect.TypeToken.method 0 Answers
Graphite Spark monitoring 0 Answers
How can we enable JmxSink in Spark ? 0 Answers