Today's AI/ML headlines are brought to you by ThreatPerspective

Digital Event Horizon

Revisiting the Limits of Fine-Tuning Job Estimates: A Study on Predicting End-to-End Fine-Tuning Time for Generative Language Models


Researchers at Dharma-AI have conducted a study to improve the accuracy of fine-tuning job duration estimates. By incorporating various factors such as evaluation and checkpoint variables, the researchers were able to develop a more accurate prediction method. The study highlights the importance of developing more accurate methods for predicting fine-tuning job durations and has significant implications for the optimization of GPU utilization and fine-tuning job scheduling strategies.

  • The conventional FLOP-based estimate for fine-tuning job durations often falls short and only explains a limited portion of the observed data.
  • The CatBoost regressor achieved a higher R-squared value of 0.98 and lower mean absolute error of 1.71 hours compared to the conventional estimate.
  • Gradient boosting models, such as CatBoost and XGBoost, performed well in capturing the relationship between configuration and duration.
  • Feature importance analysis revealed that evaluation and checkpoint variables, such as evaluation volume and trainable parameters, had a significant impact on predicted duration.
  • The existing FLOP-based estimate has limitations, including its inability to capture GPU allocation effects and reliance on fixed MFU values.



  • The world of artificial intelligence, particularly in the realm of language models, has witnessed significant advancements in recent years. However, one critical aspect that has often been overlooked is the estimation of fine-tuning job durations. This is where the work of Dharma-AI comes into play, focusing on the development of a more accurate method for predicting the time required to complete a fine-tuning job.

    The primary goal of the study is to determine why the conventional FLOP (Floating-Point Operations) calculation, which is widely used to estimate fine-tuning job durations, often falls short. The researchers conducted an in-depth analysis of 272 real supervised fine-tuning (SFT) jobs from their production pool of NVIDIA H200 GPUs, resulting in approximately 7,382 GPU-hours.

    To better understand the variables that affect fine-tuning job durations, the researchers employed a supervised regression approach. They treated the end-to-end duration as a regression problem and used a feature vector built from the job configuration and model metadata to make predictions. This approach was designed to incorporate overheads such as environment setup, model download, and tokenization, as well as periodic evaluations and checkpoint uploads.

    The results of the study reveal that the conventional FLOP-based estimate explains only a limited portion of the observed data, with a relatively low R-squared value of 0.31 and a mean absolute error of 12.41 hours. In contrast, a CatBoost regressor that only sees information available before dispatch reaches an R-squared value of 0.98, with a mean absolute error of 1.71 hours and a median error of 0.51 hours.

    The researchers also examined the performance of various baseline models, including Linear Regression, Ridge, XGBoost, and CatBoost. The study found that while linear models can help reduce the mean absolute error, the relationship between configuration and duration is not well-captured by purely linear models. Gradient boosting models, on the other hand, were found to perform well, with CatBoost and XGBoost landing within one standard deviation of each other on every metric.

    The feature importance of the CatBoost model revealed that the predictor relies heavily on evaluation and checkpoint variables, such as the total number of samples processed in evaluation (num_eval_data), the number of trainable parameters, batch evaluation size, and number of GPUs allocated. These features were found to have a significant impact on the predicted duration, with high evaluation volumes consistently pushing the predicted time up.

    The study also highlights the limitations of the existing FLOP-based estimate, including its inability to capture the effects of GPU allocation and its reliance on a fixed MFU (Model FLOPs Utilization) value. The researchers emphasize that a more accurate prediction method is needed to optimize GPU utilization and reduce idle capacity in between jobs.

    In conclusion, the study demonstrates the importance of developing more accurate methods for predicting fine-tuning job durations. By incorporating various factors such as evaluation and checkpoint variables, the researchers were able to improve the accuracy of their predictions. The findings of this study have significant implications for the optimization of GPU utilization and the development of more efficient fine-tuning job scheduling strategies.



    Related Information:
  • https://www.digitaleventhorizon.com/articles/Revisiting-the-Limits-of-Fine-Tuning-Job-Estimates-A-Study-on-Predicting-End-to-End-Fine-Tuning-Time-for-Generative-Language-Models-deh.shtml

  • https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time


  • Published: Wed Nov 18 13:26:00 2026 by llama3.2 3B Q4_K_M











    © Digital Event Horizon . All rights reserved.

    Privacy | Terms of Use | Contact Us