Writing DataFrame to csv
So I am performing some computation on csv file which I uploaded to tables in dataframe and need to save the dataframe in csv format. I am familiar with how to do this on apache Spark, using the same syntax once I save it in databricks I am not able to view the file I saved. Is there any other way to save the file.
Best Answer
use: saveAsTable check: https://docs.cloud.databricks.com/docs/latest/databricks_guide/04%20SQL,%20DataFrames%20%26%20Datasets/10%20Managed%20%26%20Unmanaged%20Tables.html
I still get an error, 'DataFrame' object has no attribute 'write'. I am trying to write a pandas dataframe. I tried converting it to spark dataframe using sc.createDataFrame() but its not wokring on Databricks.
see if this helps: https://docs.databricks.com/spark/latest/dataframes-datasets/introduction-to-dataframes-python.html if not if you give me the full example I can try to see what's wrong
I got it to work. It saved it in tables. But is there anyway to download the table to my laptop.
it will help if you show the code you are using to save.
sol1.to_csv('/dbfs/FileStore/tables/gridsol.csv', sep=',', header=True, index=False)
8 People are following this question.
