• Create
    • Ask a question
    • Create an article
    • Topics
    • Questions
    • Articles
    • Users
    • Badges
  • Sign in
  • Home /
avatar image
0

dbutils.fs.ls not showing lowest level of directory

dbfs
Question by macwanj · Jan 16 at 06:02 PM ·

For some reason, I am unable to list the lowest level of the directory structure in DBFS. For example:

dbutils.fs.ls("dbfs:/user/jason/pumps/models/")

Only yields:

Out[205]: [FileInfo(path='dbfs:/user/jason/pumps/models', name='models', size=162367703)]

and not the actual (in this case) list of pickled estimators. Any idea why I am observing this behaviour? I feel like I'm missing something really simple here...

Add comment · Show 3
Comment
10 |600 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image fish_databricks · Jan 16 at 06:43 PM 1
Share

hi @macwanj, i believe you are looking at the bottom most directory. dbutils only returns the size of an object if it is a file, if it's a directory it says size=0. i'm guessing you may have accidentally overwritten your /models directory with a with 'models' file with no extension. if this is not the case, there could be a bug in play, in which case please raise a support ticket!

keep in mind too, dbfs:/user/jason is not a mounted bucket and will probably end up in your root bucket, so maybe look at your s3 or adls to verify your directory is setup the way you think it is.

avatar image macwanj fish_databricks · Jan 16 at 08:11 PM 0
Share

Thanks you were right...for some reason models was some kind of file. After redoing, models as a folder (with size=0 - this was a good tip) it all works now.

Best,

Jason

avatar image fish_databricks macwanj · Jan 16 at 08:22 PM 0
Share

awesome, glad to help!

Sort

  • Votes
  • Created
  • Oldest

Your answer

Hint: You can notify a user about this post by typing @username

Up to 2 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

Follow this Question

9 People are following this question.

avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image

Related Questions

How do I use my own JavaScript, images, or styles in my displayHTML() code? 1 Answer

sharing notebooks in community edition 0 Answers

What cluster does dbutils.fs.cacheFiles cache to? 1 Answer

Error trying to execute a job 1 Answer

Best Practice for Writing to DBFS Locally 1 Answer

  • Product
    • Databricks Cloud
    • FAQ
  • Spark
    • About Spark
    • Developer Resources
    • Community + Events
  • Services
    • Certification
    • Spark Support
    • Spark Training
  • Company
    • About Us
    • Team
    • News
    • Contact
  • Careers
  • Blog

Databricks Inc.
160 Spear Street, 13th Floor
San Francisco, CA 94105

info@databricks.com
1-866-330-0121

  • Twitter
  • LinkedIn
  • Facebook
  • Facebook

© Databricks 2015. All rights reserved. Apache Spark and the Apache Spark Logo are trademarks of the Apache Software Foundation.

  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Create an article
  • Explore
  • Topics
  • Questions
  • Articles
  • Users
  • Badges