Why am I seeing "FetchFailedException: Adjusted frame length" and large shuffle block sizes as reported by the error?

This usually means you're exceeding the 2GB shuffle block size limit which could be caused by the following:

• not enough partitions (1 in the case of creating an RDD from a non-splittable, compressed file like Gzip). You need to repartition() after the data is loaded in order to partition the data (via shuffle) to other nodes in the cluster. This will give you the parallelism that you need for faster processing.
• skewed data due to a poor partition key choice. Note: the average block size for an unskewed data source is (total data size) / (# mappers) / (# reducers), which is usually a divisor of around 100*100 to 1000*1000, so we typically see on the order of KB or MB for single block sizes.