For some reason, I am unable to list the lowest level of the directory structure in DBFS. For example:
dbutils.fs.ls("dbfs:/user/jason/pumps/models/")
Only yields:
Out[205]: [FileInfo(path='dbfs:/user/jason/pumps/models', name='models', size=162367703)]
and not the actual (in this case) list of pickled estimators. Any idea why I am observing this behaviour? I feel like I'm missing something really simple here...
hi @macwanj, i believe you are looking at the bottom most directory. dbutils only returns the size of an object if it is a file, if it's a directory it says size=0. i'm guessing you may have accidentally overwritten your /models directory with a with 'models' file with no extension. if this is not the case, there could be a bug in play, in which case please raise a support ticket!
keep in mind too, dbfs:/user/jason is not a mounted bucket and will probably end up in your root bucket, so maybe look at your s3 or adls to verify your directory is setup the way you think it is.
Thanks you were right...for some reason models was some kind of file. After redoing, models as a folder (with size=0 - this was a good tip) it all works now.
Best,
Jason
9 People are following this question.
