Hi,
I have created a dataframe which contains a sql query. I wanted to use the query in sqlContext. My code is as below:
query = auto.select('column1')
sqlContext.sql(query)
But It throws an error as follows
Could someone guide on it. Thanks in advance!
This works fine
qvar = ''.join(query)
sqlContext.sql("%s" % qvar)
Spark supports SQL queries on top of RDDs/ DataFrames. For an application that requires complex data processing, SQLs may very well be the best way to process data. But writing queries that span multiple lines may make the spark code less readable and difficult to debug (had a tough time doing it in our project). Or we can as well do the following:
Just use the below code . First you need to create the view from the dataframe , then you can use SparkSession object to query from the view . This will return a dataframe .
dataset.createOrReplaceTempView("dataset") val result:Dataset[Row]=spark.sql("select * from dataset")
34 People are following this question.
