A method to accelerate the joining of distributed datasets by a given criterion
Introduction: Rapidly growing volumes of information pose new challenges to modern data analysis technologies. One of the most common related operations in analytics is the joins of datasets. A join is a highly resource-intensive operation which is difficult to scale and improve in terms of resource efficiency in distributed databases or systems based on the MapReduce paradigm. Purpose: To develop a method to accelerate the integration of datasets in distributed systems. Results: The architecture of Apache Spark, features of distributed computing based on MapReduce are examined, typical methods of joining datasets are analyzed, and the main ways to speed up the operation of joining data are considered. We propose a method to speed up a special case of joins implemented in Apache Spark with the use of partitioning and partial transfer of datasets to cluster computing nodes. The method is realized to simultaneously harness the benefits of merge and broadcast joins. The presented experimental data demonstrate that the larger the volume of input data the more efficiently the method speeds up the operation of joining data in comparison with standard methods. Thus, for 2 TB compressed data, acceleration up to ~37% was obtained in comparison with standard Spark SQL.