Thoughts on Interpretability in Deep Neural Networks
Traditional software programs and rules-based engines have an identified set of instructions and decision trees which operate on inputs to determine the output. There are certain areas where we have a rich interpretation of input data driving the output. But, we do not have the luxury to identify the correlation between the input data and the output data all the time — for example: — looking at a MRI scan to determine if the patient has or can get Alzheimer’s disease. These are complex problems, and in these cases Deep Neural Networks have been pretty powerful to determine the output for a test input given a training set of inputs and outputs.
As machine learning practitioners, we get to work with different problems. Some of the problems have rich datasets with data points covering a wide range in the sample space, some do not. In the case where the data is not rich to train the network, engineers have tried to understand the underlying input data to find patters of correlation with the output. In other words, there has been an attempt to interpret the data and “Interpretability” has been cited as a useful tool. Let us try to understand this need for “Interpretability” further. Deep neural nets are used to solve complex problems such as autonomous vehicles or decision making in advanced surgeries. Generally speaking, because of this technology being new, we are scared of neural nets, primarily because we do not understand how they work. These are the common reasons/thoughts on why part of the Machine Learning/Deep Learning community supports “Interpretability”.
· Understanding the data — to give us confidence about the model and it’s working.
· Troubleshooting and Debugging — if something goes wrong with the output (for example: the surgery goes wrong or a car hits a pedestrian, we want to understand how to debug the model)
· Accountability — If the model is wrong, who is guilty? Is it because of the training data or the team which built the model?
· Trust — Partially related to the first point, we are yet to see this technology deployed in massive scale to gain the trust of consumers, be it autonomous cars or advanced surgery.
These are valid reasons but these mask the main reason as to why we do not completely “trust” these deep neural networks. With time, we will start believing them just like how we now believe technology which didn’t exist a few years ago — for example Auto Pilots for planes or some of the trains. If we can verify the quality of the training/test data and a near infinite set (like the scale of ImageNet dataset), the training of the model will get better with the fine tuning of hyper parameters and epochs.
I would like to provide an example here as well. There was an experiment wherein a neural network was given a dataset of Pneumonia patients with symptoms and asked to predict whether their survival. The neural network predicted that the patients who have an existing condition of Asthma had a better chance or survival. To a trained medical eye, this obviously sounds incorrect. The reason for this prediction gets a little clearer when we analyze this further. Patients with an Asthma or heart condition pay extra attention to their breathing patterns. So, if such a patient were to see abnormalities in their breathing, they are more likely to get to a hospital to get this checked out. This does NOT mean that by understanding this data better or “interpreting” it would have lead us to a better model.
Deep Neural Networks are useful in cases where we do not have a wide understanding of the underlying data. There are other ways to make sure that we are not detecting false positives. A couple of options that I find helpful are:
1. Sensitivity Analysis: This is an analysis to determine how a neural network output is influenced by its input and/or weight changes. This is a statistical analysis which does not depend on necessarily knowing the data itself, but being aware of the effects of tweaking the hyper parameters when modeling the deep neural network
2. Dropping out a sample of neurons: When using fully connected neural networks, adjacent networks develop codependency on each other. Dropout prevents interdependent learning between neurons. When some neurons are dropped, other neurons are expected to “step in” to make up for these missing neurons. This has been proven to reduce over fitting when modeling deep neural networks.
To conclude and emphasize the importance of training data, I would like to point to two key papers which demonstrate this effectively.
· Michele Banko and Eric Bill from Microsoft wrote a paper on this topic at http://www.aclweb.org/anthology/P01-1005
· Peter Norvig and his team from Google have written an iconic paper at https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35179.pdf
