• Create
    • Ask a question
    • Create an article
    • Topics
    • Questions
    • Articles
    • Users
    • Badges
  • Sign in
  • Home /
avatar image
2

How to rename nested json fields in Dataframe

sparkspark sqldataframejsonnested
Question by Nishant · Dec 04, 2015 at 12:19 PM ·

I have json events of the following structure:

{ 'atlas_ts': '', 'ev': { '_a': { 'build': '', 'name': '', 'version': ''

     },
     '_abtest': '',
     '_c': {  
           'referrer': '',
           'utm_campaign': '',
           'utm_content': '',
           'utm_medium': '',
           'utm_source': '',
           'utm_term': ''
     }
 }

}

I want to rename the columns of the dataframe created. I want to replace ev._a to ev.app; ev._c to ev.campaign; ev._a.name to ev.app_name. How can I do this.

Add comment
Comment
10 |600 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

1 Answer

Sort

  • Votes
  • Created
  • Oldest
avatar image
-1

Answer by Miklos · Dec 04, 2015 at 05:50 PM

You specify a schema when created the DataFrame if you want to retain the structure of the json records. If you're planning to flatten the records, you can use the .withColumnRenamed() api.

Comment
Add comment · Show 1 · Share
10 |600 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image Nishant · Dec 11, 2015 at 09:10 AM 1
Share

@Miklos:

I am aware of the .withColumnRenamed() api. But it does not work for nested fields.

Your answer

Hint: You can notify a user about this post by typing @username

Up to 2 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

Follow this Question

10 People are following this question.

avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image

Related Questions

How can I create a DataFrame from a nested array struct elements? 1 Answer

Conversion of a StructType column to MapType column inside a DataFrame? 1 Answer

org.apache.spark.SparkException: Task not serializable : Case class serialization issue may be? 1 Answer

How to calculate Percentile of column in a DataFrame in spark? 2 Answers

Apply a logic for a particular column in dataframe in spark 0 Answers

  • Product
    • Databricks Cloud
    • FAQ
  • Spark
    • About Spark
    • Developer Resources
    • Community + Events
  • Services
    • Certification
    • Spark Support
    • Spark Training
  • Company
    • About Us
    • Team
    • News
    • Contact
  • Careers
  • Blog

Databricks Inc.
160 Spear Street, 13th Floor
San Francisco, CA 94105

info@databricks.com
1-866-330-0121

  • Twitter
  • LinkedIn
  • Facebook
  • Facebook

© Databricks 2015. All rights reserved. Apache Spark and the Apache Spark Logo are trademarks of the Apache Software Foundation.

  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Create an article
  • Explore
  • Topics
  • Questions
  • Articles
  • Users
  • Badges