Less-Naïve Bayesian Classification

February 10th, 2010 by Dana Honeycutt, Ph.D.

The Bayesian learner in Pipeline Pilot is a so-called naïve Bayesian classifier. The “naïve” refers to the assumption that any particular feature contributes a specific amount to the likelihood of a sample being assigned to a given class, irrespective of the presence of any other features. For example, the presence of an NH2 group in a compound has the same effect on predicted activity whether or not there is also an OH or COOH group elsewhere in the compound. In other words, a naïve Bayesian classifier ignores interaction effects.

We know that in reality, interaction effects are quite common. Yet, empirically, naïve Bayesian classification models are surprisingly accurate (not to mention that they are lightning-fast to train).

But perhaps there are cases where a model with interactions would be better. How might we make the Bayesian learner less naïve? If we use molecular fingerprints as descriptors, one simple approach is to create a new fingerprint by pairing off the original fingerprint features and adding them to the list. We can then train the model on the new fingerprint with its expanded feature list.

A sparse molecular fingerprint (such as the Accelrys extended-connectivity fingerprints) consists of a list of feature IDs. These IDs are simply integers corresponding to certain substructural units. E.g., “16″ might refer to an aliphatic carbon, while “7137126″ might refer to an aryl amino group. So if our original fingerprint has the following features:

16
85
784
12662
...

our fingerprint-with-interactions would have the above features with the following ones in addition:

16$85
16$784
16$12662
85$784
85$12662
...

The “$” is just an arbitrary separator between the feature IDs. A Bayesian learner works by simply counting the features present in the two classes of samples (e.g., “active” vs. “inactive”), so the feature labels are unimportant, as long as they are unique.

To test the approach, I applied it to models of the Ames mutagenicity data that I discussed in a previous posting, and to an MAO inhibitor data set. Does it work? The short answer is, “Yes, with caveats.” Read my posting on the Pipeline Pilot forum for details (registration is free).

  • Share/Save/Bookmark

Lies, damned lies and (Oracle) statistics*

February 3rd, 2010 by Ian Buchan

While investigating the costs of joins between tables in Oracle, I came across the following, seemingly curious, result.  I had two tables that were identical in content and layout, each with indexes on the same columns but when I ran the same search on both tables, the query on one table was consistently more than 25% faster than the same query on the other.  “You must have done something differently” you cry.  Well, it wasn’t exactly obvious…

Let’s start at the beginning.  I produced 2 identical tables containing a 10,000 record sample of CAP (Chemicals Available for Purchase) using the same Pipeline Pilot protocol.  The tables differed in name only: one was CapSample, the other CapSample2.  I created indexes on the CLogP and Num_H_Acceptors columns of both tables and then timed the SQL query:

SELECT count(*) FROM CapSample WHERE CLogP>5 and Num_H_Acceptors>10

over 1,000 iterations on each table (replacing CapSample with CapSample2 as appropriate).  My intention was to then measure the time of the search taking CLogP from one table and Num_H_Acceptors from the other table, joining them by the primary key CardRef column.  However the search on CapSample consistently took about 3.85 seconds per 1000 iterations while the same search on CapSample2 consistently took about 2.79 seconds.  I was the only user on the machine and I kept re-running and switching between CapSample and CapSample2 and the results were consistent.  Weird!

The first thing was to examine the execution plans.  Aha! They were different.  Both were using hash joins on the two indexes, but the order of the two index range scan searches was different for the two tables.  Obviously, the CapSample2 order was better.  But why wasn’t it choosing it for CapSample?  At this point, I noticed a note at the end of the explain plan output for CapSample2:

Note

—–

- dynamic sampling used for this statement

This wasn’t there for CapSample.  Why not?  Because I’d imported CapSample the day before and only created CapSample2 today!  During the night the statistics had been gathered automatically on CapSample.  I’d only added the indexes after creating CapSample2, so the indexes on CapSample had no statistics, even though the table did.

All I had to do was gather default statistics for both tables again.  Then, being careful to slightly change my SQL so that I didn’t hit any cached plans, I re-explained the queries on both tables and bingo! I got consistent results and they matched those for the fast search of CapSample2.  Running the searches on both tables now gave me the 2.79 seconds I’d seen earlier.

As a final sanity check, I re-timed using the search over CapSample using the original SQL and I got the original time of 3.85 seconds again.  I was hitting the cached plan: Oracle used it even though the statistics had changed.  It seems weird running two queries that look identical except for an extra space character and finding that one runs over 25% faster than the other, but that’s what happens when you have cached plans.

So the moral(s) of this tale are:

1. When you change tables significantly or add indexes, gather table statistics for the changed tables and gather index statistics for changed or new indexes.

2. Oracle’s dynamic sampling can be very good.  However, you might want to gather proper statistics immediately after changes if you are automatically gathering statistics on your tables.  Otherwise, you could find the plan changes later (when cached plans are replaced).

3. Remember to either clear cached plans or change the SQL statement slightly after you have gathered new statistics to avoid hitting old cached plans.

*See http://en.wikipedia.org/wiki/Lies,_damned_lies,_and_statistics

  • Share/Save/Bookmark

Ensemble Models for Better Predictions

January 22nd, 2010 by Dana Honeycutt, Ph.D.

One approach to building a predictive model is to choose a powerful technique such as a neural network (NN) or support vector machine (SVM) algorithm and then tune the model-building parameters to maximize the predictive performance. Over the past 15 years or so, an increasingly popular alternative is to combine the predictions of multiple different techniques into a consensus or ensemble model, without necessarily optimizing each individual model within the ensemble. This is the approach that won the million dollar Netflix Prize last year, as well as the zero dollar challenge from the November 2009 Pipeline Pilot newsletter. I’ll be talking about the latter; for details on the Netflix Prize solution, go here.

In brief, the Pipeline Pilot Challenge was to find the model-building technique that gives the best ROC score for a particular classification problem. When we formulated the problem, we figured people would apply the various different learner components in Pipeline Pilot, and probably come up with a solution involving an SVM, Bayesian, or recursive partitioning (RP) model.

But winner Lee Herman took a clever alternative approach. He built four different models using four dissimilar techniques: Bayesian, RP (a multi-tree forest), mixture discriminant analysis, and SVM. For making predictions on the test set, he summed the predictions from each of the models to get a composite score. This ensemble model gave a better ROC score than any of the individual models contributing to it. For details, see Lee’s protocol on the Pipeline Pilot forum (registration is free).

Why does this work? In essence, each type of model captures some aspect of the relationship between the descriptors and what we wish to predict, while having its own distinct errors and biases. To the extent that the errors are uncorrelated between models, they cancel rather than reinforce each other. Thus the accuracy of the whole becomes greater than the greatest accuracy of any of its parts. It’s as if many wrongs can make a right.

  • Share/Save/Bookmark

Mad about MAD

January 8th, 2010 by Dana Honeycutt, Ph.D.

Over the past year or so, I have spent a great deal of time working with model applicability domains (MAD). Here I explain some of the what and the why.

When we build a statistical model—whether with linear regression, Bayesian classification, recursive partitioning, or some other method—we want to ensure that the model is a good one. If the goal is to make predictions with the model, then “good” means “able to make accurate predictions.” We usually use cross-validation or test set validation to convince ourselves that a model is good in this sense.

But there’s more to it than this. Even a good model makes poor predictions for samples that are too different from the samples in the training data used to build the model. In other words, the training data define the model’s applicability domain.

For example, suppose we wish to model per-capita crawfish consumption as a function of several variables, including the distance from the Mississippi River. Suppose also that our training and test sets consist solely of Louisiana residents. Even if we find that the model has good predictive ability for the test set, we would not expect it to do a good job predicting crawfish consumption in, say, Oregon (though it might do an OK job for parts of Mississippi). In other words, locations in Oregon lie outside the MAD. (See map.)

This idea appears obvious, yet models in statistical software packages often lack the ability to automatically define their own MAD and flag predictions outside the MAD as questionable. (In linear regression models, confidence and prediction bands serve this role to some extent. The bands become wider as we move away from the center of the training data.) The onus is generally on the user of the model to ensure that it is applied correctly. When the person applying the model is the same one who built it, and is thus familiar with the training data and the model’s limitations, this is not too big a problem. But when the creator and user of a model are two different people separated in space or time, a model’s awareness of its own applicability domain can be critical to the proper use of the model.

In the life sciences, it appears that the need to take the MAD into account when making predictions was first recognized for QSAR models of toxicity such as TOPKAT. TOPKAT introduced the notion of the optimum prediction space (OPS) defined by the ranges of the training set descriptors in principal component space. But the OPS is just one of several MAD measures discussed in the literature (e.g., see here, here, here, and here).

To summarize some of my own recent work in this area: In various numerical experiments, I have reproduced the research results of others who found that the distance from a test sample to samples in the training set correlates well with the model prediction error. (“Distance” can be defined in several different ways, and a lengthy essay could be written on this subject alone. But I’ll spare you for now.) This gives us the potential to estimate MAD-dependent error bars even for learning methods that do not intrinsically support them. A few of the model-building (learner) components in Pipeline Pilot now support OPS and other MAD measures, and we’re working on adding more of these.

I hope I have convinced you of the importance of paying attention to the applicability domain when making predictions with a model. I’ll have more to say on this in a future posting.

  • Share/Save/Bookmark

Good Models Require Good Data

October 1st, 2009 by Dana Honeycutt, Ph.D.

In my last posting, I touted ROC analysis as one of the best ways to evaluate and compare different methods for building classification models. To do a true apples-to-apples comparison, it also helps to have a good reference data set. In this regard, Katja Hansen et al. have done data modelers a favor by publishing a “Benchmark Data Set for in Silico Prediction of Ames Mutagenicity.” Not only did they vet and make available the data, but they also provide data splits for cross-validation to help modelers ensure that their method comparisons have a common basis.

The authors compare several techniques, including the Bayesian classifier in Pipeline Pilot. Data junkie that I am, I couldn’t resist throwing the Ames data at this and a few other Pipeline Pilot learners. Here are the results I got using the ECFP_4  molecular fingerprint as the descriptor:

Method   ROC Score
Bayesian   0.82
RP Tree   0.78
RP Forest   0.82
R SVM   0.72
kNN   0.84
     

These results show a few things. The best ROC scores in the table are comparable to those reported by Hansen et al. for various classifiers that they investigated. (The best score they obtained was 0.86 for an SVM model.) The results confirm the widely known fact that forest models give better predictive performance than single tree models. Finally, they confirm that molecular fingerprints are good descriptors for building classification models.

If you want more of the statistical details, I provide them in a posting on the Pipeline Pilot Forum at the Accelrys Community site. (Registration is free.)

  • Share/Save/Bookmark

Let’s ROC

September 14th, 2009 by Dana Honeycutt, Ph.D.

In the field of machine learning, a binary classification model is a statistical method for assigning an object to one of two categories (classes): benign vs. malignant, active vs. inactive, crystalline vs. noncrystalline, and so on. We build these models to reduce the number of experiments we need to run or to reduce the human labor required to evaluate experimental data (such as image data). The models are rarely perfect—meaning that they generally assign at least some objects to the wrong category.

In evaluating the model quality, the number of metrics we can look at is vast, with names such as: accuracy, precision, specificity, sensitivity (a.k.a. recall), positive predictive value, negative predictive value, Cohen’s kappa, F-measure, and more. But if you asked me to judge a binary classifier’s predictive power based on a single number, that number would be the ROC area-under-the-curve (AUC) score on a test data set.

The ROC AUC score comes from a ROC plot, which is simply a plot of the true positive rate (sensitivity) against the false positive rate (1 − specificity). We generate the points on the plot by varying the cutoff value we apply to the model’s output to distinguish between the predicted classes. (Note that most so-called classification models are at root ranking models, which output a numerical score corresponding to the relative likelihood of the object being in one class versus the other.) Here’s a typical ROC plot:

ROC plot from Pipeline Pilot for Bayesian model of NCI AIDS data

ROC plot for Pipeline Pilot Bayesian model of NCI AIDS data

Each point on the plot tells you this: “For a true positive rate given by the Y axis value, the X axis value is the price you must pay in false positives.” It is then up to you to decide what the best tradeoff is and to set the cutoff accordingly. Or you may decide that none of the points on the curve give you the combination of sensitivity and specificity you need, and that you need a better model.

As you might infer from the name, the ROC AUC score is just the area under the ROC curve. It ranges in value from 0.5 for a model that’s no better than random guessing to 1.0 for a perfect model. Unlike the other metrics I mentioned above, the ROC score is independent of any specific cutoff value. Because of this, its value is an intrinsic property of the model (for a given test set). It does not depend on any preference we might have for, say, reducing the number of false negatives at the price of more false positives. It gives us a single value that we can use to easily compare the performance of different classification methods or to tune the performance of a given method.

  • Share/Save/Bookmark

Look Before You Learn

June 23rd, 2009 by Dana Honeycutt, Ph.D.

Part of my job is creating and maintaining learner components for building statistical models in Accelrys’s Pipeline Pilot product. A statistical model is an empirically derived equation or set of rules for predicting some unknown property (say the toxicity of a chemical compound) from a set of known properties (say descriptors derived from the compound’s structure).

A statistical model–as contrasted to a mechanistic model–is built from a specific set of data, called the training set, using a specific learning algorithm (such as linear least-squares, recursive partitioning, etc.). The quality of the model is crucially dependent on the quality of the training data.

Pipeline Pilot makes it really easy to build statistical models from your data. All it takes is dropping in a data reader component, choosing an appropriate learner component, and specifying the variables you wish to use. Because
of this ease, you may be tempted to build models from a data set before taking a look at the data.

Don’t do it!

Here’s why: more often than you might think, data sets are dirty. Some values are missing or invalid. What you thought was a scalar property appears as an array in the data. A few extreme outliers are present which (depending on the learner) may seriously skew the results. Extra commas in your CSV file have shifted some values to the wrong columns. You’re trying to build a classification model, but all data records have been assigned the same class. You thought that your data set contained only small organic molecules, but somehow a few organometallics got in there. Unbeknownst to you, the creator of the data set used 99 as a missing value tag. And so on.

Pairs Plot of Contaminated Data Set

Pairs Plot of a Contaminated Data Set

I am sometimes called upon to diagnose problems that customers or colleagues have when trying to build a model. Often the root of the problem is that something is wrong with the input data. In many such cases, just looking at the data in a table makes the problem obvious. Other times, simple analysis (such as univariate analysis) or plots (such as pairs plots) show what’s wrong.

The more worrisome cases are the ones we may never hear about. Not all problems with a training data set will make a learner fail or produce obviously incorrect results. So even if you have gone ahead and successfully built a model before looking at the data, you should still look at the data afterward.

Whether you build models in Pipeline Pilot, R, Weka, or some other program, remember to Look before you Learn.

  • Share/Save/Bookmark

Informatics lessons from the MRS

May 19th, 2009 by George Fitzgerald, PhD

The Materials Research Society (MRS) “encourage communication and technical information exchange across the various fields of science affecting materials.” It sponsors a spring meeting held in San Francisco and a fall meeting in Boston. This year’s spring meeting covered topics ranging from amorphous materials to methods for environmental stability to multiple topics in nanotechnology. (See all symposium titles here.)

Most interesting to me was Symposium Z: “Computational Nano science — How to Exploit Synergy between Predictive Simulations and Experiment, which fits with the comments I made in my previous posting, and shows just how much active interest there is in this topic. Prof. Krishna Rajan, who heads the Combinatorial Sciences and Materials Informatics Collaboratory, demonstrated how he uses data mining as a tool to understand the formation of apatites (minerals of the form A10­(BO4) 8X2) based on data mining and statistical analysis. How do you get your head around and N-dimensional space? How do you grasp trends when there are dozens of variables to consider? Use methods like recursive partitioning and Principal Component Analysis (PCA). 

Simpler than the modeling approaches I mentioned in my earlier posting, these require only a statistical analysis of the data (some experimental results, some modeling output). The results reduce N-dimensional datasets to 2 or 3 dimensions that are “grasp-able” by mere humans. Applying these approaches to the apatite data clearly shows how the choices of cation and anion influence the stability of the crystal.

Just think how many other research problems we could understand if we had the tools to look at the data in the right way.

  • Share/Save/Bookmark