Hi Team,
What is the best way to monitor the CPU and memory of the databricks cluster ? Is it supporting any tools ( paid or free) ?
Answer by jules · Mar 14, 2018 at 01:36 AM
We internally use Prometheus, but you can use Ganglia too and other commercial tools such as DataDog.
We did a meetup on monitoring at Databricks, and the talks are here:
https://www.slideshare.net/julesdamji/monitoring-largescale-apache-spark-clusters-at-databricks
https://www.slideshare.net/julesdamji/monitoring-error-logs-at-databricks
https://www.meetup.com/MonitorSF/events/238074373/
Cheers
Jules
Databricks Inc.
160 Spear Street, 13th Floor
San Francisco, CA 94105
info@databricks.com
1-866-330-0121