1. Application Development
2. Libraries
3. Machine Learning
4. Cross Validation

Cross Validation

• Description
• Cross Validation Strategies
  • Train-Test Splits
  • Train-Test-Holdout Splits
  • K-Fold Splits
  • Multi-Random Splits
• Parameters
• Examples

Description

A prevalent problem when utilizing machine learning algorithms is overfitting, or when an algorithm “memorizes” the training data but does a poor job extrapolating to out of sample cases. A common method for dealing with the overfitting problem is to hold back some subset of data from the original training algorithm and then measure the fit algorithm’s performance on this hold-out set. This is commonly known as cross validation. A model is trained on one subset of data and then validated on another set of data.

Cross Validation Strategies

There are several strategies for holding out data. FlinkML has convenience methods for

• Train-Test Splits
• Train-Test-Holdout Splits
• K-Fold Splits
• Multi-Random Splits

Train-Test Splits

The simplest method of splitting is the trainTestSplit. This split takes a DataSet and a parameter fraction. The fraction indicates the portion of the DataSet that should be allocated to the training set. This split also takes two additional optional parameters, precise and seed.

By default, the Split is done by randomly deciding whether or not an observation is assigned to the training DataSet with probability = fraction. When precise is true however, additional steps are taken to ensure the training set is as close as possible to the length of the DataSet $\cdot$ fraction.

The method returns a new TrainTestDataSet object which has a .training attribute containing the training DataSet and a .testing attribute containing the testing DataSet.

Train-Test-Holdout Splits

In some cases, algorithms have been known to ‘learn’ the testing set. To combat this issue, a train-test-hold out strategy introduces a secondary holdout set, aptly called the holdout set.

Traditionally, training and testing would be done to train an algorithms as normal and then a final test of the algorithm on the holdout set would be done. Ideally, prediction errors/model scores in the holdout set would not be significantly different than those observed in the testing set.

In a train-test-holdout strategy we sacrifice the sample size of the initial fitting algorithm for increased confidence that our model is not over-fit.

When using trainTestHoldout splitter, the fraction Double is replaced by a fraction array of length three. The first element coresponds to the portion to be used for training, second for testing, and third for holdout. The weights of this array are relative, e.g. an array Array(3.0, 2.0, 1.0) would results in approximately 50% of the observations being in the training set, 33% of the observations in the testing set, and 17% of the observations in holdout set.

K-Fold Splits

In a k-fold strategy, the DataSet is split into k equal subsets. Then for each of the k subsets, a TrainTestDataSet is created where the subset is the .training DataSet, and the remaining subsets are the .testing set.

For each training set, an algorithm is trained and then is evaluated based on the predictions based on the associated testing set. When an algorithm that has consistent grades (e.g. prediction errors) across held out datasets we can have some confidence that our approach (e.g. choice of algorithm / algorithm parameters / number of iterations) is robust against overfitting.

K-Fold Cross Validatation

Multi-Random Splits

The multi-random strategy can be thought of as a more general form of the train-test-holdout strategy. In fact, .trainTestHoldoutSplit is a simple wrapper for multiRandomSplit which also packages the datasets into a trainTestHoldoutDataSet object.

The first major difference, is that multiRandomSplit takes an array of fractions of any length. E.g. one can create multiple holdout sets. Alternatively, one could think of kFoldSplit as a wrapper for multiRandomSplit (which it is), the difference being kFoldSplit creates subsets of approximately equal size, where multiRandomSplit will create subsets of any size.

The second major difference is that multiRandomSplit returns an array of DataSets, equal in size and proportion to the fraction array that it was passed as an argument.

Parameters

The various Splitter methods share many parameters.

Examples

// An input dataset- does not have to be of type LabeledVector
val data: DataSet[LabeledVector] = ...

// A Simple Train-Test-Split
val dataTrainTest: TrainTestDataSet = Splitter.trainTestSplit(data, 0.6, true)

// Create a train test holdout DataSet
val dataTrainTestHO: trainTestHoldoutDataSet = Splitter.trainTestHoldoutSplit(data, Array(6.0, 3.0, 1.0))

// Create an Array of K TrainTestDataSets
val dataKFolded: Array[TrainTestDataSet] =  Splitter.kFoldSplit(data, 10)

// create an array of 5 datasets
val dataMultiRandom: Array[DataSet[T]] = Splitter.multiRandomSplit(data, Array(0.5, 0.1, 0.1, 0.1, 0.1))