• Create
    • Ask a question
    • Create an article
    • Topics
    • Questions
    • Articles
    • Users
    • Badges
  • Sign in
  • Home /
avatar image
1

Pass SQL query as parameter to sqlContext

sparkspark sqlhivecontext
Question by Balavigneshnv · Feb 03, 2018 at 08:23 AM ·

Hi,

I have created a dataframe which contains a sql query. I wanted to use the query in sqlContext. My code is as below:

query = auto.select('column1')

sqlContext.sql(query)

But It throws an error as follows

Could someone guide on it. Thanks in advance!

error.jpg (81.1 kB)
Add comment
Comment
10 |600 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

3 Answers

Sort

  • Votes
  • Created
  • Oldest
avatar image
0
Best Answer

Answer by Balavigneshnv · Feb 05, 2018 at 01:30 PM

This works fine

qvar = ''.join(query)

sqlContext.sql("%s" % qvar)

Comment
Add comment · Share
10 |600 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image
0

Answer by careenjoseph · Feb 06, 2018 at 09:44 AM

Spark supports SQL queries on top of RDDs/ DataFrames. For an application that requires complex data processing, SQLs may very well be the best way to process data. But writing queries that span multiple lines may make the spark code less readable and difficult to debug (had a tough time doing it in our project). Or we can as well do the following:

  • Save the well formatted SQL into a file on local file system
  • Read it into a variable as string
  • Use the variable to execute the query
  • hadoop-training-in-bangalore

Comment
Add comment · Share
10 |600 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image
0

Answer by toxic wapour · Feb 06, 2018 at 10:58 AM

Just use the below code . First you need to create the view from the dataframe , then you can use SparkSession object to query from the view . This will return a dataframe .

dataset.createOrReplaceTempView("dataset") val result:Dataset[Row]=spark.sql("select * from dataset")

Comment
Add comment · Share
10 |600 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Your answer

Hint: You can notify a user about this post by typing @username

Up to 2 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

Follow this Question

34 People are following this question.

avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image

Related Questions

How to read a existing Hive Table which is in ORC format in Spark 1.6.2?? 0 Answers

Spark from lab to production ? 0 Answers

Escape option is not working while writing dataframe. 0 Answers

Save mongoDB data to parquet file format usign Apache spark 1 Answer

How can I create CassandraSQlContext in a Notebook using SparKContext.getOrCreate() ? 1 Answer

  • Product
    • Databricks Cloud
    • FAQ
  • Spark
    • About Spark
    • Developer Resources
    • Community + Events
  • Services
    • Certification
    • Spark Support
    • Spark Training
  • Company
    • About Us
    • Team
    • News
    • Contact
  • Careers
  • Blog

Databricks Inc.
160 Spear Street, 13th Floor
San Francisco, CA 94105

info@databricks.com
1-866-330-0121

  • Twitter
  • LinkedIn
  • Facebook
  • Facebook

© Databricks 2015. All rights reserved. Apache Spark and the Apache Spark Logo are trademarks of the Apache Software Foundation.

  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Create an article
  • Explore
  • Topics
  • Questions
  • Articles
  • Users
  • Badges