Persisting to MEMORY_AND_DISK significantly slows read from s3 stage
I have a DataFrame I'm trying to cache shortly after reading from s3:
datasets["nav"]["dataFrame"] = df \
.withColumn('dpkey', concat_ws("-", *data_partitions)) \
.persist(StorageLevel.MEMORY_AND_DISK)
Running my notebook without caching runs as expected, but for some reason when I try to persist that dataframe the task takes several multiples longer.
I believe the dataset is larger my available memory, but the task is slow from the get-go even while there's still plenty available.
Any ideas?
Hey Christian - persisting/caching isn't free because you're going to take some serialization overhead. There's also potentially some storage issues, if the data isn't distributed evenly across the cluster than certain machines are going to have to work much harder and do much more work than other ones.
Data is evenly distributed as the delay immediately follows reading from mounted s3 and memory distribution across workers looks even in UI. Is the serialization tax really so high that it takes multiples as long as reading from s3?
10 People are following this question.
