Gradient Boosting Explained

Gradient boosting builds an ensemble of small decision trees sequentially, where each new tree is trained to predict the errors of the trees before it. Summing the trees' outputs turns many weak learners into one strong model, and it remains the go-to method for tabular data.

How Does Gradient Boosting Actually Work?

Start with a naive prediction (like the mean), compute the residual errors, fit a small tree to those residuals, add a scaled-down version of it to the model, and repeat — each round nudges predictions toward the truth.

# Pseudocode for regression with squared error
prediction = mean(y)
for m in range(n_estimators):
    residuals = y - prediction          # what the model still gets wrong
    tree = fit_small_tree(X, residuals) # learn the leftover error
    prediction += learning_rate * tree.predict(X)
# Final model = mean + lr * (tree_1 + tree_2 + ... + tree_m)

Boosting vs Bagging: How Is This Different From Random Forests?

Random forests use bagging: they train deep trees independently in parallel on bootstrap samples and average them to cut variance. Boosting trains shallow trees sequentially, each one correcting the last, which cuts bias — so boosting usually wins on accuracy but is easier to overfit and harder to parallelize across trees.

XGBoost vs LightGBM vs CatBoost: Which Library Should You Use?

All three are optimized gradient boosting implementations, and the practical differences fit in one line each.

• XGBoost: the battle-tested default with regularized objectives and the largest ecosystem of docs and integrations.
• LightGBM: histogram-based and leaf-wise growth, typically the fastest and most memory-efficient on large datasets.
• CatBoost: handles categorical features natively (no one-hot encoding) and works well with minimal tuning.

Which Hyperparameters Matter Most?

Three knobs dominate: learning rate, number of trees, and tree depth — lower the learning rate, raise the tree count, and keep trees shallow.

from lightgbm import LGBMClassifier

model = LGBMClassifier(
    n_estimators=1000,     # many trees...
    learning_rate=0.05,    # ...each contributing a little
    max_depth=4,           # shallow trees generalize better
    subsample=0.8,         # row sampling per tree fights overfitting
    colsample_bytree=0.8,  # feature sampling per tree
)
# Use early stopping on a validation set to pick n_estimators automatically

Common Pitfalls

• High learning rate with few trees: fast to train, quick to overfit. Prefer lr around 0.01-0.1 with early stopping.
• Skipping the validation set: boosting will happily memorize training data; monitor validation loss every round.
• Deep trees: depth 3-6 is usually enough; boosting gets its power from many trees, not big ones.
• Ignoring class imbalance: set scale_pos_weight (XGBoost/LightGBM) or class weights instead of hoping the loss sorts it out.