• Create
    • Ask a question
    • Create an article
    • Topics
    • Questions
    • Articles
    • Users
    • Badges
  • Sign in
  • Home /
avatar image
1

New JAR conflicts with old JAR

jar
Question by Kyle · Jul 05, 2015 at 06:53 PM ·

We updated one of our jars with a new version using the databricks UI.

I detached and deleted the old jar. Created a new cluster and uploaded and attached the new jar.

However I am getting versions conflict since the old versions of the jar is still added to the classpath, and idea how to solve this?

Add comment · Show 1
Comment
10 |600 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image Mathieu Boespflug · May 30, 2016 at 05:55 PM 0
Share

Still experiencing the same issue. @Kyle is this supposed to have been fixed in previous releases?

3 Answers

Sort

  • Votes
  • Created
  • Oldest
avatar image
2
Best Answer

Answer by Kyle · Jul 05, 2015 at 11:51 PM

Try deleting all old jars in "dbfs://FileStore/jars/

Comment
Add comment · Share
10 |600 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image
1

Answer by sujitpal · Jul 15, 2015 at 04:09 AM

I had the same issue. Based on what I learned from my colleague, this is an open issue with Databricks. Here is what I did (partly based on his advise) to get around it. Not pretty but seems to work:

1) delete the old JAR file.

2) remove the cached JAR per Kyle's advice above - snippet from Scala notebook to do this.

val jarfiles = dbutils.fs.ls("dbfs:/FileStore/jars")

.map(_.path)

.filter(_.indexOf("your pattern") > -1)

jarfiles.foreach(dbutils.fs.rm(_))

3) detach notebook from cluster.

4) restart cluster.

5) add the jar back and attach the jar to the cluster.

6) re-attach notebook to cluster.

Its a painful process, and leads me to think that the pattern of depending on a JAR file under active development is discouraged in this workflow. I should probably have set up my tests on a local Spark cluster instead of debugging it here, would have gone much faster.

Comment
Add comment · Share
10 |600 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image
0

Answer by Kyle · Jul 16, 2015 at 12:45 AM

@sujitpal- thanks for the update. This actually helped us recreate the issue. We will be fixing this in one of the next couple releases.

Comment
Add comment · Share
10 |600 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Your answer

Hint: You can notify a user about this post by typing @username

Up to 2 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

Follow this Question

6 People are following this question.

avatar image avatar image avatar image avatar image avatar image avatar image

Related Questions

The jar versions embedded in my application uber jar are conflicting with the jar versions in Spark and Databricks Cloud. How do I overcome this? 1 Answer

How to secure secrets in an uploaded JAR file? 0 Answers

How do I overcome MethodNotFoundException, NoSuchMethodError, and NoClassDefNotFound? 1 Answer

problem with running sparkR 1 Answer

Can I use dbutils in a jar-based job ? 1 Answer

  • Product
    • Databricks Cloud
    • FAQ
  • Spark
    • About Spark
    • Developer Resources
    • Community + Events
  • Services
    • Certification
    • Spark Support
    • Spark Training
  • Company
    • About Us
    • Team
    • News
    • Contact
  • Careers
  • Blog

Databricks Inc.
160 Spear Street, 13th Floor
San Francisco, CA 94105

info@databricks.com
1-866-330-0121

  • Twitter
  • LinkedIn
  • Facebook
  • Facebook

© Databricks 2015. All rights reserved. Apache Spark and the Apache Spark Logo are trademarks of the Apache Software Foundation.

  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Create an article
  • Explore
  • Topics
  • Questions
  • Articles
  • Users
  • Badges