15 min read
Spark’s small-file crossover is 4 MB a file, not 40,000 files
Spark prices every file open at 4 MB of scanning, which puts the small-file crossover at 4 MB of mean size, about 40x before listing becomes the bottleneck.
- Spark
- Lakehouse
- Performance