GPU acceleration for XGBOOST learner

[phisanti]

I would like to know if it is possible to train a model using GPU acceleration. I have seen that is possible to do it with LightGBM here, but the build is falling and I am not sure it translate to XGBOOST.

Thanks

[pat-s]

If the underlying learner supports it, it should be possible. We don't add or strip features from algorithms in mlr3.

If you're facing problems with particular learners, you can post them here.

[phisanti]

According to the example in the XGBootst R package (ref), GPU acceleration can be achieved by adding one parameter:

param <- list(objective = 'reg:logistic', eval_metric = 'auc', subsample = 0.5, nthread = 4,
              max_bin = 64, tree_method = 'gpu_hist')

However, to change the computing strategy in MLR3, I have read that we have to use the future package such as future::plan("multisession"). Thus, I was wondering if I have to add the GPU in the "parameter set" or with the future package

[pat-s]

These are two different things:

• future the internal R package used by mlr3 for CPU parallelization
• tree_method = 'gpu_hist' is the option of the xgboost package to enable GPU processing
• nthread should be for CPU processing and in fact handled by mlr3 via the future package (and might possibly have no effect)

There is no relation between both, even though you might be able to combine them when resampling/tuning the learner.

Hope that helps.

[phisanti]

``But I was considering this previous issue. I have tried to add tree_method = 'gpu_hist' as parameter but I got the following error:

Error in xgb.iter.update(bst$handle, dtrain, iteration - 1, obj) : 
  [13:44:15] amalgamation/../src/metric/../common/common.h:157: XGBoost version not compiled with GPU support.
Stack trace:
  [bt] (0) /usr/local/lib/R/site-library/xgboost/libs/xgboost.so(+0xded58) [0x7fc97fcb6d58]
  [bt] (1) /usr/local/lib/R/site-library/xgboost/libs/xgboost.so(xgboost::common::AssertGPUSupport()+0x41) [0x7fc97fd42de1]
  [bt] (2) /usr/local/lib/R/site-library/xgboost/libs/xgboost.so(xgboost::gbm::GBTree::ConfigureUpdaters()+0xad) [0x7fc97fcb7bad]
  [bt] (3) /usr/local/lib/R/site-library/xgboost/libs/xgboost.so(xgboost::gbm::GBTree::Configure(std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&)+0x2b0) [0x7fc97fcdac20]
  [bt] (4) /usr

Here is how it looks my full parameter set:

XGB_parameters <- ps(

  eta = p_dbl(default = 0.05, lower = 0.001, upper = 0.5),
  max_depth = p_int(default = 6L, lower = 2L, upper = 18L),
  nrounds = p_int(default = 50L, lower = 5L, upper = 500L),
  gamma = p_dbl(default = 7, lower = 2, upper = 20),
  tree_method = p_fct(default = "gpu_hist", levels = c('gpu_hist')),
  colsample_bytree = p_dbl(
    default = 0.15, lower = 0.05, upper = 0.5),
    subsample = p_dbl(default = 0.15, lower = 0.01, upper = 0.5),
    min_child_weight = p_dbl(default = 1, lower = 0, upper = 3),
    booster = p_fct(levels = c("dart")),
  scale_pos_weight = p_dbl(default = 1, lower = 0.75, upper = 1.1),
  # Parameters specific for DART

  rate_drop = p_dbl(default = 0.1, lower = 0.1, upper = 1, tags = "train"),
  skip_drop = p_dbl(default = 0.1, lower = 0.1, upper = 1, tags = "train")
  )


XGB_parameters$add_dep("skip_drop", "booster", CondEqual$new("dart"))
XGB_parameters$add_dep("rate_drop", "booster", CondEqual$new("dart"))
XGB_parameters

Any help will be appreciated

[pat-s]

XGBoost version not compiled with GPU support.

This means you need to install {xgboost} with GPU support, which seems to be off by default. This is not within the capabilities of mlr3 and should be documented somewhere in the {xgboost} package itself.

[mllg]

To compute on GPUs, you need to compile xgboost yourself and link against CUDA.

https://xgboost.readthedocs.io/en/stable/build.html#building-with-gpu-support

Nothing we can do on our side, but I'll mention this in the docs.

[phisanti]

Sorry for the late answer.

I have now compiled the GPU version of XGBOOST. I then loaded the parameters as before. However, when I run a tuning instance, the process seems to remain in the CPU (see image):

[phisanti]

Here is a proper reproducible code:

library(data.table)
library(mlr3)

# Generate dummy data
n <- 10e4
d <- data.table(target = sample(2, n, replace = T))
d[, paste0("x", 1:10) := rnorm(n)]

# Set learner
xgb_learner <- lrn("classif.xgboost",
                  eval_metric = "error",
                  predict_type = "prob")

traintask <- TaskClassif$new(id = "training_data",
                                backend = d,
                                target = "target")

# Set parameters for training
tuner <- tnr("random_search")
rc = rsmp("subsampling")

mult_measure <- list(msr("classif.specificity", na_value = 0),
                     msr("classif.sensitivity", na_value = 0),
                     msr("classif.fbeta", na_value = 0),
                     msr("classif.bacc")
                     )

XGB_parameters <- ps(

  # GPU
  tree_method = p_fct(default = "gpu_hist", levels = c('gpu_hist')),
  eta = p_dbl(default = 0.05, lower = 0.001, upper = 0.5),
  max_depth = p_int(default = 6L, lower = 2L, upper = 18L),
  nrounds = p_int(default = 150L, lower = 5L, upper = 1000L),
  gamma = p_dbl(default = 7, lower = 2, upper = 20),
  colsample_bytree = p_dbl(
    default = 0.15, lower = 0.05, upper = 0.5),
    subsample = p_dbl(default = 0.15, lower = 0.01, upper = 0.5),
    min_child_weight = p_dbl(default = 1, lower = 0, upper = 3),
    booster = p_fct(levels = c("dart")),
  scale_pos_weight = p_dbl(default = 1, lower = 0.75, upper = 1.1),
  # Parameters specific for DART

  rate_drop = p_dbl(default = 0.1, lower = 0.1, upper = 1, tags = "train"),
  skip_drop = p_dbl(default = 0.1, lower = 0.1, upper = 1, tags = "train")
  )


XGB_parameters$add_dep("skip_drop", "booster", CondEqual$new("dart"))
XGB_parameters$add_dep("rate_drop", "booster", CondEqual$new("dart"))
XGB_parameters


stop_time = now()
stop_time <- stop_time + hours(4)

term_combo <- trm("combo",
  list(
  trm("evals", n_evals = 1000),
  trm("clock_time", stop_time = stop_time))
  )

# Setup instance
instance <- TuningInstanceMultiCrit$new(
  task = traintask,
  learner = xgb_learner,
  resampling = rc,
  measure = mult_measure,
  search_space = XGB_parameters,
  terminator = term_combo)

tuner$optimize(instance)

[mllg]

Does it compute on the GPU if you call xgboost yourself?

[phisanti]

When I try to run the following code, it seems to run normally. Here is the sample code that I tried:

library('xgboost')

# Simulate N x p random matrix with some binomial response dependent on pp columns
set.seed(111)
N <- 1000000
p <- 50
pp <- 25
X <- matrix(runif(N * p), ncol = p)
betas <- 2 * runif(pp) - 1
sel <- sort(sample(p, pp))
m <- X[, sel] %*% betas - 1 + rnorm(N)
y <- rbinom(N, 1, plogis(m))

tr <- sample.int(N, N * 0.75)
dtrain <- xgb.DMatrix(X[tr, ], label = y[tr])
dtest <- xgb.DMatrix(X[-tr, ], label = y[-tr])
wl <- list(train = dtrain, test = dtest)

# An example of running 'gpu_hist' algorithm
# which is
# - similar to the 'hist'
# - the fastest option for moderately large datasets
# - current limitations: max_depth < 16, does not implement guided loss
# You can use tree_method = 'gpu_hist' for another GPU accelerated algorithm,
# which is slower, more memory-hungry, but does not use binning.
param <- list(objective = 'reg:logistic', eval_metric = 'auc', subsample = 0.5, nthread = 32,
              max_bin = 64, tree_method = 'gpu_hist')
pt <- proc.time()
bst_gpu <- xgb.train(param, dtrain, watchlist = wl, nrounds = 1000)
proc.time() - pt

I have checked a bit more in depth and I think it is related with loading the object directly on memory via xgb.DMatrix(). I am not sure how does the data.table passes form the mlr3 instance to the xgboost algo, but that seems to make a difference.

[phisanti]

Here is a screenshot of the GPU usage: the peak between 6.28 to 6.30 is when I ran the little sample script from the previous comment.