• Create
    • Ask a question
    • Create an article
    • Topics
    • Questions
    • Articles
    • Users
    • Badges
  • Sign in
  • Home /
avatar image
0

Writing DataFrame to csv

csvspark dataframesaveapache spark dataframefile formats
Question by ankit biradar · Nov 30, 2016 at 03:48 AM ·

So I am performing some computation on csv file which I uploaded to tables in dataframe and need to save the dataframe in csv format. I am familiar with how to do this on apache Spark, using the same syntax once I save it in databricks I am not able to view the file I saved. Is there any other way to save the file.

Add comment
Comment
10 |600 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

2 Answers

Sort

  • Votes
  • Created
  • Oldest
avatar image
1
Best Answer

Answer by peyman · Nov 30, 2016 at 04:45 PM

use: saveAsTable check: https://docs.cloud.databricks.com/docs/latest/databricks_guide/04%20SQL,%20DataFrames%20%26%20Datasets/10%20Managed%20%26%20Unmanaged%20Tables.html

Comment
Add comment · Show 3 · Share
10 |600 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image ankit biradar · Nov 30, 2016 at 05:15 PM 0
Share

I still get an error, 'DataFrame' object has no attribute 'write'. I am trying to write a pandas dataframe. I tried converting it to spark dataframe using sc.createDataFrame() but its not wokring on Databricks.

avatar image peyman ankit biradar · Nov 30, 2016 at 05:32 PM 0
Share

see if this helps: https://docs.databricks.com/spark/latest/dataframes-datasets/introduction-to-dataframes-python.html if not if you give me the full example I can try to see what's wrong

avatar image ankit biradar peyman · Nov 30, 2016 at 06:28 PM 0
Share

I got it to work. It saved it in tables. But is there anyway to download the table to my laptop.

avatar image
0

Answer by peyman · Nov 30, 2016 at 04:08 AM

it will help if you show the code you are using to save.

Comment
Add comment · Show 1 · Share
10 |600 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image ankit biradar · Nov 30, 2016 at 04:50 AM 0
Share

sol1.to_csv('/dbfs/FileStore/tables/gridsol.csv', sep=',', header=True, index=False)

Your answer

Hint: You can notify a user about this post by typing @username

Up to 2 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

Follow this Question

8 People are following this question.

avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image

Related Questions

Error in getting the DataFrame from simple sequence 0 Answers

dataframe negative numbers saved with quotes in csv format 2 Answers

Which compression formats are supported by Databricks Cloud? 1 Answer

Load csv to databricks cluster using Spark 0 Answers

Modify data frame name when writing (as .csv) to a Blob Storage using Azure Databricks 0 Answers

  • Product
    • Databricks Cloud
    • FAQ
  • Spark
    • About Spark
    • Developer Resources
    • Community + Events
  • Services
    • Certification
    • Spark Support
    • Spark Training
  • Company
    • About Us
    • Team
    • News
    • Contact
  • Careers
  • Blog

Databricks Inc.
160 Spear Street, 13th Floor
San Francisco, CA 94105

info@databricks.com
1-866-330-0121

  • Twitter
  • LinkedIn
  • Facebook
  • Facebook

© Databricks 2015. All rights reserved. Apache Spark and the Apache Spark Logo are trademarks of the Apache Software Foundation.

  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Create an article
  • Explore
  • Topics
  • Questions
  • Articles
  • Users
  • Badges