Spark workflow task scheduling with deadline and privacy constraints in hybrid cloud networks

The increasing adoption of hybrid clouds in organizations stems from their ability to bolster private cloud resources with additional public cloud capacity when required. However, scheduling distributed applications, such as workflow tasks, on hybrid cloud resources presents new and intricate challenges. A significant concern revolves around the potential exposure of private data and tasks within third-party, public cloud infrastructures, especially within sensitive domains like healthcare applications. The complexity escalates when considering the selection of resources from multiple cloud providers due to the fluctuating resource computation prices and data transmission costs. This paper presents the Spark Workflow Task Scheduling to Hybrid Cloud (SWSHC) framework, designed to schedule Spark workflows precisely while adhering to deadline and task privacy constraints within a hybrid cloud setting. Our innovative approach encompasses developing and implementing three pivotal components: deadline division, stage order optimization, and task scheduling mechanisms. We segregate the workflow deadline for each stage to bridge the gaps between stages effectively. Additionally, job prioritization is achieved using the maximum rank rule. The proposed solution considers diverse factors, including interval pricing variations, utilization of heterogeneous VM instances, intra- and inter-bandwidth considerations, and the efficient utilization of private cloud resources. Through meticulous calibration of our algorithm and comprehensive experimentation with various realistic workflows, our findings unequivocally demonstrate that SWSHC surpasses existing solutions in the current literature the cost by up to 40–70% in terms of cost efficiency and resource utilization.