Need help with stanford CoreNLP library download

Here's a notebook that shows how you can workaround this:

https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/2531312965265039/1640910664551931/3261354835245810/latest.html

Hi @raela, thank you for this lead.

I tried out the code on the Databricks CE with a Version 2.0 preview cluster, but ran into the following problem: The following line fails:

ucl.addURL(new java.net.URL("file:////tmp/stanford-corenlp-3.6.0-models-english.jar"))

The error message is:

java.lang.NoSuchMethodException: com.databricks.backend.daemon.driver.ClassLoaders$ReplWrappingClassLoader.addURL(java.net.URL)
    at java.lang.Class.getMethod(Class.java:1786)
    at Notebook.reflMethod$Method1(<console>:47)

I did try to print out the method names using the following line:

ucl.getClass.getMethods.map(_.getName).foreach(println)

But the "addURL" method is not listed. None of the other methods had similar names.

What am I missing?

Thanks again for your help.

Hi @raela, here is an update to my last post. Using the parent classloader -- instead of the grandparent -- seems to be working for some of the notebook (upto the NERDemo.java cell):

val cl = java.lang.Thread.currentThread.getContextClassLoader.getParent//.getParent

The imports at the top of the NERDemo are still failing though. I am now using version 1.6.1/Hadoop2 of Databricks CE.

Thanks again for your lead.

Ok, here is what seems to be working (apache/branch-2.0 preview on Databricks CE) :

type CL = ClassLoader with AnyRef {def addURL(url: java.net.URL): Unit}
val cl = java.lang.Thread.currentThread.getContextClassLoader.getParent
val cl2 = java.lang.Thread.currentThread.getContextClassLoader.getParent.getParent.getParent
// Using Scala's structural types to load the file.
val ucl: CL = cl.asInstanceOf[CL]
val ucl2: CL = cl2.asInstanceOf[CL]
ucl.addURL(new java.net.URL("file:////tmp/stanford-corenlp-3.6.0-models-english.jar"))
ucl2.addURL(new java.net.URL("file:////tmp/stanford-corenlp-3.6.0-models-english.jar"))
sc.addJar("/tmp/stanford-corenlp-3.6.0-models-english.jar")

Hi @raela

Thank you for the heads up.

Somehow, the code that wgets the model jar and calls "sc.addJar(...)" does not work for me. However, when I replace that block of code with the following block (derived from your hint above), everything works fine.

import java.io.File
val file = new File("file:/tmp/stanford-corenlp-3.6.0-models-english.jar")
if (file.exists) {
  println("File exists, ...")
} else {
  println("File NOT FOUND, making copy ...")
  val copyStatus = dbutils.fs.cp("/FileStore/tables/stanford-corenlp-3.6.0-models-english.jar", "file:/tmp/")  
  println(s"copyStatus: $copyStatus")
}
type CL = ClassLoader with AnyRef {def addURL(url: java.net.URL): Unit}
val cl = java.lang.Thread.currentThread.getContextClassLoader.getParent
val cl2 = java.lang.Thread.currentThread.getContextClassLoader.getParent.getParent.getParent
// Using Scala's structural types to load the file.
val ucl: CL = cl.asInstanceOf[CL]
val ucl2: CL = cl2.asInstanceOf[CL]
ucl.addURL(new java.net.URL("file:////tmp/stanford-corenlp-3.6.0-models-english.jar"))
ucl2.addURL(new java.net.URL("file:////tmp/stanford-corenlp-3.6.0-models-english.jar"))
sc.addJar("/tmp/stanford-corenlp-3.6.0-models-english.jar")

(I had stashed the model jar inside a FileStore directory to avoid having to download it every time)

It would be great if Databricks could also address the need for the user to download the JAR, and of not allowing a JAR of this size to be uploaded directly using the normal library creation option. Is it not so much simpler to just loosen that JAR size limit a bit? I must be missing something :-)

Thanks for your help.

- Sanjay

Hi @raela, this does not seem to be working for me anymore. I'm using Spark-2.10 (with Scala 2.10), but have also tried falling back to 1.6.2. Code that worked earlier does not compile anymore. the "addURL()" calls and the one "addJar()" all appear to succeed, but imports in later application code still fail:

import edu.stanford.nlp.ling.CoreAnnotations.{LemmaAnnotation, PartOfSpeechAnnotation, SentencesAnnotation, TokensAnnotation} import edu.stanford.nlp.pipeline.{Annotation, StanfordCoreNLP}

<console>:30: error: object ling is not a member of package edu.stanford.nlp import edu.stanford.nlp.ling.CoreAnnotations.{LemmaAnnotation, PartOfSpeechAnnotation, SentencesAnnotation, TokensAnnotation} ^ <console>:31: error: object Annotation is not a member of package edu.stanford.nlp.pipeline import edu.stanford.nlp.pipeline.{Annotation, StanfordCoreNLP}

Can you give me any insight into what I need to check?

Thanks for any hints.

Hi all,

I was successful in getting this to run in mid-July. Just tried it again (8/15/2016), and it still works (using ver. 1.6.2, with Hadoop 2). You can see an abridged version at

https://www.dropbox.com/s/em9bs0fr4lm2oz7/https___databricks-prod-cloudfront.cloud.databricks.pdf?dl=0

Here's what I did.

(1) Loaded and attached spark-corenlp-0.1

(2) Loaded stanford-english-corenlp-2016-01-10-models.jar into my AWS account, and created a "mnt" link in Databricks.

Feel free to reply if you have any questions.

Best,
Tom

Hi Tom,

Thanks for your response.

I see no material difference between your code and mine, and am really baffled by this sudden change. I tweaked and tinkered all day yesterday, this is so disappointing.

I am using the Databricks community edition, what about you?

Hi Sanjay,

Yes, I'm in CE.

My code does NOT run on Spark 2.0 (Scala 2.10). It first throws a warning:

Warning: there were 2 feature warning(s); re-run with -feature for details
defined type alias CL
cl: ClassLoader = org.apache.spark.repl.SparkIMain$$anon$2@56507b9a
cl2: ClassLoader = com.databricks.backend.daemon.driver.ClassLoaders$LibraryClassLoader@6eafb10e
ucl: CL = org.apache.spark.repl.SparkIMain$$anon$2@56507b9a
ucl2: CL = com.databricks.backend.daemon.driver.ClassLoaders$LibraryClassLoader@6eafb10e

Then an error following this code.

val output = input
  .select(cleanxml('text).as('doc))
  .select(explode(ssplit('doc)).as('sen))
  .select('sen, tokenize('sen).as('words), ner('sen).as('nerTags), sentiment('sen).as('sentiment))
-----------------
error: bad symbolic reference. A signature in functions.class refers to type UserDefinedFunction in package org.apache.spark.sql which is not available.
It may be completely missing from the current classpath, or the version on the classpath might be incompatible with the version used when compiling functions.class.
<console>:40: error: org.apache.spark.sql.UserDefinedFunction does not take parameters
         .select(cleanxml('text).as('doc))

I suspect "library" compiles are out of synch with Spark 2.0, but don't have time to track it down. Did you use a different library from the last time it worked in 1.6.2? Sorry can't be more help right now.

Best,
Tom