H2C-Scheduler: An Efficient and Scalable Job Scheduling Method for Hybrid Heterogeneous AI Computing Clusters

With the rapid development of deep neural networks, the demand for computational power for model training is growing exponentially. To meet this demand, many supercomputing centers deploy hybrid heterogeneous AI computing clusters. However, the existing Kubernetes scheduling algorithm for such clusters suffers from inefficient node pre-selection, low AI chip resource utilization, and uneven cluster load. Therefore, realizing efficient job scheduling in hybrid heterogeneous AI computing clusters has become a key challenge to be solved. In this paper, we propose an efficient and scalable job scheduling method for hybrid heterogeneous AI computing power clusters: the H2C-Scheduler. In the node pre-selection phase, we design an algorithm based on multi-dimensional resource dynamic labeling with a three-level cache filtering mechanism (DTRC) to significantly reduce the pre-selection time. In the node preference stage, for the cluster load situation, we design a smooth weighted polling algorithm (HWRR) based on the performance difference of heterogeneous computing resources, which improves the utilization of AI chips. An algorithm based on heterogeneous resource weight normalization (HWN) is designed to improve the load balancing of the cluster. The experimental results show that this method significantly improves the node pre-selection time, AI chip utilization and cluster load balancing in job scheduling of hybrid heterogeneous AI computing power clusters, in which the AI chip utilization is improved by 13.5% on average.