how to enforce Monotonic Constraints in XGBoost with ScikitLearn?

monotonic constraints xgboost
xgboost python
xgboost documentation
xgboost regression
xgboost num_round
xgboost gamma
xgboost loss function
xgboost classifier

I build up a XGBoost model using scikit-learn and I am pretty happy with it. As fine tuning to avoid overfitting, I'd like to ensure monotonicity of some features but there I start facing some difficulties...

As far as I understood, there is no documentation in scikit-learn about xgboost (which I confess I am really surprised about - knowing that this situation is lasting for several months). The only documentation I found is directly on http://xgboost.readthedocs.io

On this website, I found out that monotonicity can be enforced using "monotone_constraints" option. I tried to use it in Scikit-Learn but I got an error message "TypeError: init() got an unexpected keyword argument 'monotone_constraints'"

Do you know a way to do it ?

Here is the code I wrote in python (using spyder):

grid = {'learning_rate' : 0.01, 'subsample' : 0.5, 'colsample_bytree' : 0.5,
    'max_depth' : 6, 'min_child_weight' : 10, 'gamma' : 1, 
    'monotone_constraints' : monotonic_indexes}
#'monotone_constraints' ~ = "(1,-1)"
m07_xgm06 = xgb.XGBClassifier(n_estimators=2000, **grid)
m07_xgm06.fit(X_train_v01_oe, Label_train, early_stopping_rounds=10, eval_metric="logloss", 
    eval_set=[(X_test1_v01_oe, Label_test1)])

In order to do this using the xgboost sklearn API, you need to upgrade to xgboost 0.81. They fixed the ability to set parameters controlled via kwargs as part of this PR: https://github.com/dmlc/xgboost/pull/3791

Monotonic Constraints, whenever x≤x′ is a decreasing constraint. XGBoost has the ability to enforce monotonicity constraints on any features used in a boosted model. A Simple  Assume for every pair of values (α,β) of feature x1, where α≤β, we want to enforce a positive monotonic relationship, that is w(α)≤w(β). At the root node, only one weight w₀ is produced. w₀(α)=w₀(β) for α,β in feature x₁ in root node 0. This conforms to the monotonic constraint.

XGBoost Scikit-Learn API currently (0.6a2) doesn't support monotone_constraints. You can use Python API instead. Take a look into example.

This code in the example can be removed:

params_constr['updater'] = "grow_monotone_colmaker,prune"

A Tutorial of Model Monotonicity Constraint Using Xgboost , This monotonicity constraint has been implemented in the R gbm model. import fetch_california_housing from sklearn.model_selection import We can also enforce monotonicity constraints while fitting multiple features. Enforcing Monotonic Constraints in XGBoost¶ It is very simple to enforce monotonicity constraints in XGBoost. Here we will give an example using Python, but the same general idea generalizes to other platforms. Suppose the following code fits your model without monotonicity constraints

How would you expect monotone constraints to work for a general classification problem where the response might have more than 2 levels? All the examples I've seen relating to this functionality are for regression problems. If your classification response only has 2 levels, try switching to regression on an indicator variable and then choose an appropriate score threshold for classification.

This feature appears to work as of the latest xgboost / scikit-learn, provided that you use an XGBregressor rather than an XGBclassifier and set monotone_constraints via kwargs.

The syntax is like this:

params = {
    'monotone_constraints':'(-1,0,1)'
}

normalised_weighted_poisson_model = XGBRegressor(**params)

In this example, there is a negative constraint on column 1 in the training data, no constraint on column 2, and a positive constraint on column 3. It is up to you to keep track of which is which - you cannot refer to columns by name, only by position, and you must specify an entry in the constraint tuple for every column in your training data.

Monotonicity constraints in machine learning, Monotonicity constraints in LighGBM and XGBoost. For tree based methods from sklearn.metrics import mean_squared_error as mse. size = 1000000 Other methods for enforcing monotonicity. Tree based methods are  Fitting a model and having a high accuracy is great, but is usually not enough. Quite often, we also want a model to be simple and interpretable. An example of such an interpretable model is a linear regression, for which the fitted coefficient of a variable means holding other variables as fixed, how the response variable changes with respect to the predictor. For a linear regression, this

[New Feature] Monotonic Constraints in Tree Construction · Issue , I got a few requests on supporting monotonic constraints on certain feature with respect to the output, i.e. Could you clarify how XGBoost enforce monotonic constrains? Second test: California hHousing data from sklearn. Monotonic Constraints¶. This example illustrates the effect of monotonic constraints on a gradient boosting estimator. We build an artificial dataset where the target value is in general positively correlated with the first feature (with some random and non-random variations), and in general negatively correlated with the second feature.

[MRG] Monotonic constraints for GBDT · Issue #15582 · scikit-learn , This PR adds support for monotonic constraints for the histogram (see https://​xgboost.readthedocs.io/en/latest/tutorials/monotonic.html). scikit-learn.scikit-learn (macOS pylatest_conda_mkl_no_openmp) macOS pylatest_conda_mkl_no_openmp succeeded Details Meeting Issues automation moved this from Reviewer approved to Done Mar 24, 2020

How does the popular XGBoost and LightGBM algorithm enforce , I build up a XGBoost model using scikit-learn and I am pretty happy with it. As fine tuning to avoid overfitting, I'd like to ensure monotonicity of some features but  Monotonicity constraints in machine learning Posted September 16, 2018 In practical machine learning and data science tasks, an ML model is often used to quantify a global, semantically meaningful relationship between two or more values.

Comments
  • However seems like monotone constraints does not work for now. I asked question about this here. @chapelon does it work for you?
  • This answer is now deprecated. See my response. stackoverflow.com/a/53147154/339588
  • Is the statement about XGBClassifier not being supported for monotonicity still true?
  • Yes, and I doubt it ever will be, for the reason stated in my answer.