SMN2: Prediction of Activity

There are about 20 assays based on SMN2 constructs in the ChEMBL DB. Among all we have chosen the reporter assay described in the data description section. The assay contains 30k records, which is an interesting amount to apply ML methodology, but beyond that, it is a blackbox assay, which means that the patterns of expression of functional SMN2 protein can be elicited by a plethora of cellular mechanisms that can be further investigated to identify protein targets involved. Below, assay description of the most relevant SMN2 assays and distribution histogram of the activities recorded for the 30k molecules screen.

The 30k molecules used in SMN2 reporter screen have also been assayed in many more experimental models in the ChEMBL DB producing almost 1M records in more than 50k assays. We can therefore use the activity of these compounds on all these assays to build ML models suitable for prediction of activity of molecules that have never faced to a SMN assay. Charts below display some examples of these assays and the global distribution of activities of their respective records.

And, of course, ahead of finding relations, or making predictions, why not to have a look upon the relationship between the activity in SMN2 and the rest of the assay these molecules participate in? The picture shows the existing correlation between SMN2 and  other targets for the cases for R2 >0.5, slope>0.4 and N>10.

Evaluation of predictive models.

For model evaluation, data have been splitted in validation and test sets The validation set has been labelled for classification (Binary and multiple) and regression and used to generate the model. The test set is tested against the model and the predictions compared to the previously hidden activity columns. Accuracy of predictions is then assessed by the classical machine learning evaluation parameters in the charts below.

Validations set are passed through decision trees (Rpart: classification and regression) random forests (classification and regression), and Naive Bayes (binary and multiple classification) algorithms. Each algorithm is tested against different combinations of variables: All possible variables, a wide combination of biological and chemical variables, just biological, just chemical, and a minimal combination of both. For each combination of classification algorithm and variable set, a confusion matrix and a bar diagram with most meaningful statistical quality descriptors is charted below. If regression, there is a correlation plot instead.

Decision Trees Classification (confusion matrix)

Decision Trees (QC descriptors)

Random Forests Classification (confusion matrix)

Random Forests Classification (QC descriptors)

Naive Bayes Classification (ConfusionMatrix)

Naive Bayes (QC descriptors)

All algorithms compared in their respective best condition tested.

And the three binary classification algorithms visual confusion matrixes compared in their best conditions…

When classification algorithms result in acceptable quality they are submitted to 10 iterations cross-validations to assess predictions robustness and stability of QC descriptors

Scores prediction by regression. Although prediction by decision tree results in a poor estimation of potencies, Random forest regression shows a high correlation with actual ChemblScore data.

Based on these quality indicators, compound selection will be carried output using a dashboard that contains classification predictions from Rpart and random forests so as forests regression.. Naive Bayes output to be included in the dashboard just as an optional end filter.

Compound selection. Description of dashboard uses.

Let’s look at a typical multimethod compound selection dashboard. It is a panel where predictions through different models are interactively compared to allow quality control of the selection.   As most compounds include multiple experimental events recorded in the database, and there is a prediction for each event, we need to calculate an index of activity for each compound. For regressions, this is the average SMN2 score (ActualPlusPredictedSMN2Score), which uses actual values when they exist. Classification will use the ratio between the number of  times a compound has been predicted active vs the total number of events (countRatio). In this example we will use binomial random forests classification.

Left chart in the dashboard below visualizes the ActualPlusPredictedSMN2Score for each unique molecule-target pair vs the number of occasions that this pair has a result in the DB. Marking the most active (either actual or predicted) in a convenient number of occasions in this chart causes the marked compounds to appear in the top right plot that compares the average SMN2 potency (actual and predicted. ) with the number of records for each molecule. Marking the desired compounds in this plot raises the down right chart with the random forests classification countRatio.

Now, it’s just a matter of making the selection at your convenience. The table here contains the 143 most potent predicted SMN2 actives with the highest count ratios, as indicated in more intense dot colors in the dashboard.

And here, some of the most potent structures