In the era of data transformation, the timeliness needed to be ensured and avoid the system downtime caused by task congestion as well, which puts forward higher requirements for the system stability and the intelligence of task scheduling. However, the manual maintenance which used to be adopted for IT batch or scheduling in data center and jobs platform domain cannot meet the requirements. For instance, existing methods of task scheduling lack some valid scheduling standards in face of thousands of tasks or relatively complicate dependencies. Therefore, task scheduling ought to do automated assignments instead of manual configuration in order to solve the problem of task congestion and achieve an appropriate allocation of resources.
Firstly, we establish a mapping model between real-time resources pool and fluctuating hardware resources, and we conduct a prediction function for resources by algorithm. Then we quantify the resource consumption of each task and intelligently assign each of tasks to an appropriate segment of resource units. In addition, we propose a model that is composed of function of task dependencies and function of intelligent allocation to ensure the order of task execution to avoid deadlock and recursion. Finally, we develop a program to automatically allocate task sequences to resource clusters. This program can intelligently schedule tasks to improve system stability and solve the problem of task congestion.
In conclusion, our algorithm has a significant effect in the following aspects:
1) Our algorithm reduces the resources consumed by the task queue,saves the 50 daily computing hours for average task and improves the operational efficiency by 400%.
2) The algorithm significantly reduced the fluctuation range. The CPUtime of a task daily fluctuates between approximately + 5% and − 5% now, compared with about 300% before.
3) The model makes the task number curve smoother and more uniform after the algorithm intervention.