step-select

A SciKit-Learn style feature selector using best subsets and stepwise regression.

Install

Create a virtual environment with Python 3.8 and install from PyPi:

pip install step-select

Use

Preliminaries

Note: this example requires two additional packages: pandas and statsmodels.

In this example we'll show how the ForwardSelector and SubsetSelector classes can be used on their own or in conjuction with a Scikit-Learn Pipeline object.

import pandas as pd
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LinearRegression
import statsmodels.datasets
from statsmodels.api import OLS
from statsmodels.tools import add_constant

from steps.forward import ForwardSelector
from steps.subset import SubsetSelector

We'll download the auto dataset via Statsmodels; we'll use mpg as the endogenous variable and the remaining variables as exongenous. We won't use make, as that will create several dummies and increase the number of paramters to 12+, which is too many for the SubsetSelector class; we'll also drop price.

data = statsmodels.datasets.webuse('auto')
data['foreign'] = pd.Series([x == 'Foreign' for x in data['foreign']]).astype(int)
data.fillna(0, inplace=True)
data.head()

X = data.iloc[:, 3:]
y = data['mpg']

Forward Stepwise Selection

The ForwardSelector follows the standard stepwise regression algorithm: begin with a null model, iteratively test each variable and select the one that gives the most statistically significant improvement of the fit, and repeat. This greedy algorithm continues until the fit no longer improves.

The ForwardSelector is instantiated with two parameters: normalize and metric. Normalize defaults to False, assuming that this class is part of a larger pipeline; metric defaults to AIC.

The ForwardSelector class follows the Scikit-Learn API. After fitting the selector using the .fit() method, the selected features can be accessed using the boolean mask under the .best_support_ attribute.

selector = ForwardSelector(normalize=True, metric='aic')
selector.fit(X, y)

ForwardSelector(normalize=True)

X.loc[:, selector.best_support_]

Best Subset Selection

The SubsetSelector follows a very simple algorithm: compare all possible models with $k$ predictors, and select the model that minimizes our selection criteria. This algorithm is only appropriate for $k<=12$ features, as it becomes computationally expensive: there are $\frac{k!}{(p-k)!}$possible models, where $p$ is the total number of paramters and $k$ is the number of features included in the model.

The SubsetSelector is instantiated with two parameters: normalize and metric. Normalize defaults to False, assuming that this class is part of a larger pipeline; metric defaults to AIC.

The SubsetSelector class follows the Scikit-Learn API. After fitting the selector using the .fit() method, the selected features can be accessed using the boolean mask under the .best_support_ attribute.

selector = SubsetSelector(normalize=True, metric='aic')
selector.fit(X, y)

SubsetSelector(normalize=True)

X.loc[:, selector.get_support()]

Comparing the full model

Using the SubsetSelector selected features yields a model with 4 fewer parameters and slightly improved AIC and BIC metrics. The summaries indicate possible multicollinearity in both models, likely caused by weight, length, displacement and other features that are all related to the weight of a vehicle.

Note: Selection using BIC as the optimization metric yields a model where weight is the only selected feature. Bayesian information criteria penalizes additional parameters more then AIC.

mod = OLS(endog=y, exog=add_constant(X)).fit()
mod.summary()

mod = OLS(endog=y, exog=add_constant(X.loc[:, selector.best_support_])).fit()
mod.summary()

Use in Scikit-Learn Pipeline

Both ForwardSelector and SubsetSelector objects are compatible with Scikit-Learn Pipeline objects, and can be used as feature selection steps:

pl = Pipeline([
    ('feature_selection', SubsetSelector(normalize=True)),
    ('regression', LinearRegression())
])
pl.fit(X, y)

Pipeline(steps=[('feature_selection', SubsetSelector(normalize=True)),
                ('regression', LinearRegression())])

pl.score(X, y)

0.7097132531085899