What is a Decision Tree?

A decision tree is a supervised-learning model that makes a prediction by applying a sequence of if-then rules. Each internal node tests a feature, each branch represents an outcome of that test, and each leaf produces a class prediction, probability, or numerical value.

Decision trees are used for classification and regression. Their appeal is practical: they can represent nonlinear interactions, require relatively little preprocessing, and produce a path that a person can inspect. Their weakness is instability—small changes in the training data can create a different tree.

Key takeaways

• A tree recursively partitions the feature space; it does not have to isolate every training observation.
• Classification splits commonly use Gini impurity or entropy, while regression splits reduce prediction error or variance.
• Depth, minimum leaf size, and pruning control complexity and overfitting.
• Random forests and gradient-boosted trees improve predictive power by combining many trees.

How a decision tree makes a prediction

Suppose a model predicts whether a machine is likely to fail. The root node might ask whether vibration exceeds a learned threshold. A branch could then test operating temperature. The observation reaches a leaf containing the estimated failure probability among training examples that followed the same path.

For regression, the leaf may return the mean target value of the observations in that region. For classification, it may return the majority class or a distribution of class frequencies. A leaf can contain many observations; fully separating the training data is usually undesirable because it can produce an overfit tree.

How a tree chooses a split

Training considers candidate features and thresholds, then selects the split that most improves a defined objective. The improvement must be weighted by how many observations go to each child node.

Gini impurity

For classification, Gini impurity measures how mixed the classes are in a node:

Gini = 1 - Σ p(k)²

A node containing only one class has impurity zero. A candidate split is useful when the weighted impurity of its children is lower than the impurity of the parent.

Entropy and information gain

Entropy is another measure of class uncertainty:

Entropy = -Σ p(k) log₂ p(k)

Information gain is the parent entropy minus the weighted child entropy. Gini and entropy often produce similar trees, though not always identical ones.

Regression loss

Regression trees commonly choose splits that reduce squared error, absolute error, or another regression criterion. Each leaf then predicts a value based on the training targets inside that region.

CART and other tree algorithms

CART, or Classification and Regression Trees, uses binary splits and underlies common implementations such as scikit-learn’s decision trees. Other algorithms include ID3, C4.5, and C5.0. Implementations differ in their supported split types, handling of missing values, pruning, and objectives.

Categorical variables may require encoding, direct subset splits, or implementation-specific handling. Missing values can be imputed or handled through learned default directions or surrogate splits. It is important to understand the behavior of the specific library rather than assume every tree implementation works the same way.

Controlling tree complexity

A deep tree can memorize noise. Common controls include:

• Maximum depth: limits the length of a prediction path.
• Minimum samples per split or leaf: prevents tiny regions.
• Minimum impurity decrease: requires a split to provide enough benefit.
• Maximum number of leaves: caps total complexity.
• Cost-complexity pruning: removes branches whose improvement does not justify added complexity.

Pruning is a structured optimization process, not random deletion. Hyperparameters should be chosen with validation data or cross-validation, while the final test set remains untouched.

Strengths and limitations

Decision trees can model interactions and threshold effects without feature scaling. They accept numerical and, depending on the implementation, categorical inputs. Prediction is fast, and a small tree is easy to visualize.

However, a single tree can have high variance, create abrupt prediction changes near a split, and favor features with many possible split points. Trees also extrapolate poorly in regression: outside the observed regions, a leaf still returns a value learned from its training samples. A large tree may be no more understandable than another complex model.

From one tree to ensembles

Ensemble learning combines multiple models. A random forest trains many trees on resampled observations and subsets of features, then averages their predictions. Gradient boosting builds trees sequentially so each new tree addresses remaining error. These approaches usually outperform one tree, but they trade away some interpretability and add computational cost.

Feature importance from a tree or ensemble should be interpreted carefully. Impurity-based importance can be biased, and a feature’s importance does not prove causation. Permutation importance, partial-dependence tools, and domain review provide additional context.

How a tree learns splits and predictions

A decision tree recursively partitions feature space. At each node, a training algorithm evaluates candidate feature thresholds or category partitions and selects a split that most reduces impurity, such as Gini impurity or entropy for classification and squared error for regression. Leaves store a class distribution or numeric prediction based on training observations that reach them. Greedy splitting is computationally practical but does not guarantee the globally best tree, and different samples or tie-breaking can produce different structures.

Continuous, ordinal, categorical, and missing features need explicit handling. One-hot encoding can create many candidate splits; native categorical methods may use ordered statistics but need leakage-safe implementation. Trees do not require scaling, yet they can favor high-cardinality variables and isolate small groups. Depth, minimum leaf size, minimum impurity decrease, and cost-complexity pruning control variance. Choose them with validation data and evaluate calibration, because a leaf probability based on few cases may be extreme and unstable.

Interpretation, failure modes, and production use

A path from root to leaf is an exact rule for one model prediction, but it is not automatically a causal explanation. Correlated variables can substitute for one another, small data changes can alter upper splits, and a simple-looking path can depend on biased labels. Global feature importance based on impurity can be misleading; permutation importance, partial dependence, and counterfactual checks add context but also have assumptions. Report uncertainty and test whether a purported rule holds on independent data and relevant subgroups.

Single trees are useful when transparency, low latency, and modest nonlinear structure matter, but ensembles usually provide stronger predictive performance. Validate boundary behavior, rare categories, missingness, and inputs outside the training range. Exported rules must reproduce training preprocessing and numeric comparison exactly. Monitor leaf occupancy, output distribution, error, and emerging categories. A tree that routes many new cases into a tiny or previously empty region should trigger review even if aggregate drift remains small. Keep a fallback for invalid schemas and document every pruning or threshold decision.

Worked example: an interpretable loan triage tree

A lender uses a tree only to prioritize incomplete applications for manual review, not to approve or deny credit. The target is a documented completeness outcome, and features available at intake exclude later decisions. Grouped temporal validation compares a shallow pruned tree with rules and logistic regression. Minimum leaf size prevents rules based on a handful of applicants, while calibration and class-specific errors are reported across channels and relevant protected groups.

Reviewers see the exact path and source values but can correct erroneous data and override routing. The organization tests correlated proxies and counterfactual changes, monitors leaf occupancy and missingness, and treats sudden traffic into a small leaf as a data-quality incident. Policy changes create a new model version and validation, not an undocumented split edit. Because the use affects access and burden, applicants receive a human channel and the tree is never presented as a causal explanation of creditworthiness.

Implementation evidence and operational readiness

A production decision needs more than a successful demonstration. Define the intended users, operating environment, inputs, outputs, dependencies, owner, and the consequence of each important failure. Establish a reproducible baseline and a versioned evaluation set before tuning. Test ordinary cases, boundary conditions, malformed or missing input, distribution shift, dependency outage, misuse, and the groups or environments most likely to be underserved. Measure task quality together with calibration or uncertainty, latency, throughput, resource cost, accessibility, privacy, and security. Record every transformation and threshold so an independent reviewer can reproduce the result and distinguish evidence from an attractive prototype.

Before launch, assign authority for release, exceptions, changes, rollback, and retirement. Use a staged rollout, preserve a safe fallback, and verify monitoring with deliberately injected failures. Operational telemetry should reveal input quality, output behavior, model or rule version, dependency health, human overrides, and confirmed outcomes without collecting unnecessary sensitive data. Define alert thresholds and a response owner, then review real-world evidence after deployment rather than assuming offline performance will persist. Reevaluate whenever data sources, users, models, vendors, policies, hardware, or objectives change. A maintained system also needs documented recovery, incident learning, deletion and retention procedures, and a clear point at which it should be disabled or replaced.

Primary references

• scikit-learn: Decision Trees
• scikit-learn: Ensemble methods