Feature Transformation

Feature Selection

Feature Selection Techniques in Machine Learning

Wrapper methods

use specific predictive model to evaluate feature subset

Filter methods

use measures that are fast to compute

more general than wrapper, so better for preprocessing

• Fisher’s Score

• Variance Threshold

  It removes all features which variance doesn’t meet some threshold. We assume that features with a higher variance may contain more useful information, but note that we are not taking the relationship between feature variables or feature and target variables into account, which is one of the drawbacks of filter methods.

• Correlation Coefficient

  If two variables are correlated, we can predict one from the other. Therefore, if two features are correlated, the model only really needs one of them, as the second one does not add additional information.

• Chi-square Test

  conditions have to be met: the variables have to be categorical, sampled independently and values should have an expected frequency greater than 5.

• Mean Absolute Difference (MAD)

• Dispersion ratio

Intrinsic

Algorithms that perform automatic feature selection during training.

• Decision Trees
  • Decision Tree is an excellent base learner for ensemble methods because it can perform bias-variance tradeoff easily by simply tuning max_depth.
  • The reason is that Decision Tree is very good at capturing interactions among different features, and the order of interactions captured by a tree is controlled by its max_depth.
  • Shallow trees can only capture low order interactions, hence they have high bias and low variance. Deep trees can model complex interactions and is prone to overfitting. They have low bias and high variance.

Embedded methods

Embedded methods are iterative in the sense that takes care of each iteration of the model training process and carefully extracts those features which contribute the most to the training for a particular iteration.

• Linear Regression
  • LASSO Regularization (L1)
  • Ridge Regression (L2)