Pass SQL query as parameter to sqlContext
Hi,
I have created a dataframe which contains a sql query. I wanted to use the query in sqlContext. My code is as below:
query = auto.select('column1')
sqlContext.sql(query)
But It throws an error as follows
Could someone guide on it. Thanks in advance!
Best Answer
This works fine
qvar = ''.join(query)
sqlContext.sql("%s" % qvar)
Spark supports SQL queries on top of RDDs/ DataFrames. For an application that requires complex data processing, SQLs may very well be the best way to process data. But writing queries that span multiple lines may make the spark code less readable and difficult to debug (had a tough time doing it in our project). Or we can as well do the following:
- Save the well formatted SQL into a file on local file system
- Read it into a variable as string
- Use the variable to execute the query
- hadoop-training-in-bangalore
Just use the below code . First you need to create the view from the dataframe , then you can use SparkSession object to query from the view . This will return a dataframe .
dataset.createOrReplaceTempView("dataset") val result:Dataset[Row]=spark.sql("select * from dataset")
Firstly, you have to type https://robloxmusicscode.com/ the in the address bar of your browser and that will lead you to the site...
37 People are following this question.
