So I am performing some computation on csv file which I uploaded to tables in dataframe and need to save the dataframe in csv format. I am familiar with how to do this on apache Spark, using the same syntax once I save it in databricks I am not able to view the file I saved. Is there any other way to save the file.
Answer by peyman · Nov 30, 2016 at 04:45 PM
use: saveAsTable check: https://docs.cloud.databricks.com/docs/latest/databricks_guide/04%20SQL,%20DataFrames%20%26%20Datasets/10%20Managed%20%26%20Unmanaged%20Tables.html
I still get an error, 'DataFrame' object has no attribute 'write'. I am trying to write a pandas dataframe. I tried converting it to spark dataframe using sc.createDataFrame() but its not wokring on Databricks.
see if this helps: https://docs.databricks.com/spark/latest/dataframes-datasets/introduction-to-dataframes-python.html if not if you give me the full example I can try to see what's wrong
I got it to work. It saved it in tables. But is there anyway to download the table to my laptop.
Answer by peyman · Nov 30, 2016 at 04:08 AM
it will help if you show the code you are using to save.
sol1.to_csv('/dbfs/FileStore/tables/gridsol.csv', sep=',', header=True, index=False)
Error in getting the DataFrame from simple sequence 0 Answers
dataframe negative numbers saved with quotes in csv format 2 Answers
Which compression formats are supported by Databricks Cloud? 1 Answer
Load csv to databricks cluster using Spark 0 Answers
Modify data frame name when writing (as .csv) to a Blob Storage using Azure Databricks 0 Answers
Databricks Inc.
160 Spear Street, 13th Floor
San Francisco, CA 94105
info@databricks.com
1-866-330-0121