Extreme Gradient Boosting (XGBoost)

This chapter investigates extreme gradient boosting (XGBoost). In addition to the Summary, there are five sections in Chapter 12 . Section 12.1 introduces the XGBoost algorithmic optimizations, including (1) regularized learning objective, (2) gradient tree boosting in XGBoost, (3) weighted quantile sketch, and (4) sparsity-aware split finding. Section 12.2 describes the XGBoost algorithm. Section 12.3 briefly introduces some important system improvements in XGBoost, including parallelization, cache-aware access, and blocks for out-of-core computation. Section 12.4 illustrates the implementation of XGBoost in R with the caret package ( Section 12.4.1 ) and the xgboost package ( Section 12.4.2 ), respectively. Section 12.5 provides some remarks on XGBoost regarding (1) XGBoost&s;s major characteristics; (2) a comparison of XGBoost (GBDT) and other models, including XGBoost (GBDT) versus single tree-based models, XGBoost versus MART, GBDT, and its developments, and XGBoost (GBDT) versus random forest (RF) and support vector machines (SVMs); (3) the advantages of XGBoost; and (4) the disadvantages of XGBoost.