Machine Learning in Biomechanics – Challenges and Opportunities

As written above, an ML algorithm needs training data from which it can learn and adapt its model. The rule of thumb is that about 60% of the work in dealing with ML is spent on pure training data acquisition. The quality of the training data is ultimately one of the decisive points as to whether the algorithm will do a good job or not.

In general, ML can be divided into two different learning strategies, which have an impact on the requirements for the training data: Supervised learning and non-supervised learning. Supervised learning is used to determine functions, such as in classification and regression problems. Here it is important that both the predictors (i.e., the data we hand over to the algorithm to receive a result) and the target values (i.e. the data the algorithm should return) are recorded as well. In the training process, the ML algorithm is thus able to learn which results it should put out for which input values and can later apply this "knowledge" to unseen data. For example, if one wants to predict ground reaction forces using MoCap data, as Mundt et al. (2019) did, one needs to record both 3D kinematics (predictor) and ground reaction forces (target values) for the training data. After the training process, the algorithm should then be able to estimate the ground reaction forces from the kinematics data on its own.

However, if the ML algorithm is to be used for the determination of unknown structures, such as clustering or dimensionality reduction, the so-called non-supervised learning is applied. For this purpose, no target values need to be recorded.

One of the decisive factors for the success of the training process is the pure amount of training data. The more complicated the relationships to be modelled are and the more precise the desired results need to be, the more training data is needed. And this is exactly where one of the great challenges in biomechanics lies. If you have ever conducted a study with volunteers, you know how difficult it is to recruit enough participants. Most studies are conducted with 10-20 subjects, which is not enough for ML applications in most cases. Biomechanical studies training an AI have an average of around 40 subjects, but even this does not always seem to be sufficient (Halilaj et al., 2018). It is difficult, if not impossible, to define a number that is needed as the minimum training data for an ML algorithm. But one can say unequivocally: you can never collect too much high-quality data! The more you have, the more accurate your model becomes. Of course, there is a kind of saturation somewhere, where the improvements will be marginal, yet there is no general maximum value for this either.

In addition to a more accurate model, a large training data set also ensures high generalisability and prevents so-called overfitting, which we will discuss later.

In addition to the sheer amount of training data, the variability in the targeted range of values is of crucial importance. The training data collected should be representative for your field of application in order to ensure a high degree of generalisability. In other words, the data you use for training should cover all cases that have occurred and will occur. An example might shed some light on this: Let's say you have programmed an ML algorithm that reads in images and distinguishes between dogs and cats. You train the algorithm with three images per animal: a dog with white fur, one with brown fur and one with black fur; a cat with white fur, one with brown fur and one with mackerel fur. Then you test the trained model with a picture of a cat that has a black fur. The probability is now relatively high that the ML algorithm incorrectly identifies a dog in this image, because there was no black cat in the training data set, only a black dog. If you add an image with a black cat to the training data, the algorithm is probably able to distinguish a black dog from a black cat, because it determines the differences between the two animals on the basis of other features, such as stature. A high variability in the training data thus also goes hand in hand with a high amount of data. A high amount of data, on the other hand, does not necessarily mean a high variability, which would be fatal. 10 pictures of brown cats do not help to identify a black cat. Here we would speak of a sampling bias.

It is equally important that the training data set does not contain cases that cannot occur at all. Say, for example, a mistake was made in the naming of the image files and a picture of a cat is thereby assigned to the category "dog". The picture of the cat would obviously not be representative of a dog. If you use a non-representative training data set, the trained model will also not be able to make accurate predictions.

When training the model, care should be taken to use only features as predictors that have actual utility. If the training data are characterised by a high number of irrelevant features, your model will produce significantly worse results. The motto is: if you put rubbish in, rubbish will come out. To stay with our dog-cat example, you could, for example, look at the metadata of the image files. Here, among other things, the exact time and date when the picture was taken is saved for each picture. This information is clearly of no use in distinguishing between dogs and cats and could even lead to a reduction in the performance of the algorithm. It is therefore important that you identify all relevant features before the training process and include these, then eliminate all irrelevant features in the next step.

After you have recorded enough training data with a high variability, and then separated relevant features from irrelevant features, the quality of the data should be checked, as with any data analysis in biomechanics. Are there any outliers that should be filtered out? Are there parts of the data missing, where gaps can (and must) be filled?

The model should produce the most accurate results possible, and therefore the quality of the data is of high importance.

Not all machine learning is the same. There are many different methods that are used for different applications. It is important to select the most suitable one and then optimise it. Each individual model in turn has some hyperparameters that must be configured optimally. There is no universal method for this, and a more detailed description would go beyond the scope of this article. But don't worry, we are working on a specific article!

There are also a few points to consider in the training process to get optimal results. For a successful training process, it is not important to minimise the error between ML output and target values, but to optimise it. It is important to find a good tradeoff between underfitting and overfitting the model. Underfitting occurs when the underlying model is not able to represent the complexity of the relationships. If this is the case, an ML algorithm with a higher complexity must be chosen. Overfitting is the exact opposite of this, and occurs much more frequently in practice. Here, it leads to the phenomenon that the learning algorithm fits the network outputs so well to the target values, that any kind of noise within the training data set is also stored and mapped (Jabbar & Khan, 2014).

The main task in training new ML algorithms can be summarised as (i) creating a model that corresponds in complexity to the real problem and (ii) training this model to optimise its internal structures.

For (i), the principle of 'Occam's Razor' can be applied. This principle states that simple solutions to a problem should be preferred to more complex ones (Sober, 2015). For ML algorithms, this principle suggests that low model complexity capable of performing the task of the network leads to high generalisation performance without overfitting. Furthermore, a reduction in model complexity leads to a reduction in computation time (Schmidhuber, 2015).

Point (ii) imposes requirements on the training data to be used and the training process itself. For the training data, the sheer amount of data points and their variability are of fundamental importance. Smaller datasets are more prone to overfitting as they may not capture the full complexity of the real-world problem and the model can easily adapt to its noise and outliers (Jabbar, 2014). However, even with a large dataset, care must be taken to avoid overfitting by stopping the training process at the right point.

For this, one can divide the training dataset into two or three sub-datasets:

1. Training set: this is the largest part and is used to reduce the error between outputs and targets
2. Validation set: used for cross-validation of the model during the training process. This is separate from the training set but is used within the training process.
3. Test set: This part of the training data is not used for the training itself but is only used to evaluate the performance of the model.

Using the three subsets of data, the early stopping regularisation technique is now applied. Usually, both the validation error and the training error will decrease in the early stages of training. The beginning of overfitting can be detected when the validation and testing error starts to increase while the training error continues to decrease. Training is stopped as soon as a significant drop in generalisation performance is detected over a predefined number and epochs. The training algorithm then returns the model parameters with the best performance up to that point (Jabbar, 2014).