• Create
    • Ask a question
    • Create an article
    • Topics
    • Questions
    • Articles
    • Users
    • Badges
  • Sign in
  • Home /
avatar image
0

What is the best way to monitor memory and cpu usage of worker nodes in a cluster

cluster monitoring
Question by Manoj · Feb 08, 2017 at 03:26 PM ·

Hi Team,

What is the best way to monitor the CPU and memory of the databricks cluster ? Is it supporting any tools ( paid or free) ?

Add comment · Show 1
Comment
10 |600 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image rsmith54 · Mar 09, 2018 at 02:29 PM 0
Share

Hi, was there any answer to this?

2 Answers

Sort

  • Votes
  • Created
  • Oldest
avatar image
0

Answer by rsmith54 · Mar 09, 2018 at 02:28 PM

Hi, was there any answer to this?

Comment
Add comment · Share
10 |600 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image
0

Answer by jules · Mar 14, 2018 at 01:36 AM

We internally use Prometheus, but you can use Ganglia too and other commercial tools such as DataDog.

https://databricks.com/blog/2017/06/01/apache-spark-cluster-monitoring-with-databricks-and-datadog.html

We did a meetup on monitoring at Databricks, and the talks are here:

https://www.slideshare.net/julesdamji/monitoring-largescale-apache-spark-clusters-at-databricks

https://www.slideshare.net/julesdamji/monitoring-error-logs-at-databricks

https://www.meetup.com/MonitorSF/events/238074373/

Cheers
Jules

Comment
Add comment · Share
10 |600 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Your answer

Hint: You can notify a user about this post by typing @username

Up to 2 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

Follow this Question

9 People are following this question.

avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image

Related Questions

Is there any way to monitor the CPU, disk and memory usage of a cluster while a job is running? 1 Answer

  • Product
    • Databricks Cloud
    • FAQ
  • Spark
    • About Spark
    • Developer Resources
    • Community + Events
  • Services
    • Certification
    • Spark Support
    • Spark Training
  • Company
    • About Us
    • Team
    • News
    • Contact
  • Careers
  • Blog

Databricks Inc.
160 Spear Street, 13th Floor
San Francisco, CA 94105

info@databricks.com
1-866-330-0121

  • Twitter
  • LinkedIn
  • Facebook
  • Facebook

© Databricks 2015. All rights reserved. Apache Spark and the Apache Spark Logo are trademarks of the Apache Software Foundation.

  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Create an article
  • Explore
  • Topics
  • Questions
  • Articles
  • Users
  • Badges