Using Apache Spark to Tune Spark Adrian Popescu (Unravel Data Systems) and Shivnath Babu (Unravel Data Systems Duke University) from executor Watch Video

⏲ Duration: 15 min 42 sec
✓ Published: 11-Jun-2018

Description:
We have developed a workload-aware performance tuning framework for Spark that collects and analyzes telemetry information about all the Spark applications in a cluster. Based on this analysis—which uses batch processing, real-time streaming, and ML analysis that Spark excels at—the framework can identify many ways to improve the overall performance of the Spark workload: (i) by identifying datasets with skewed distributions that are causing significant performance degradation, (ii) by ident