“Explainable AI” can help close the knowledge gap between AI outputs and humans, however, performance and explainability must be balanced.
Artificial intelligence’s (AI) growing powers are stretching the bounds of human comprehension. Many AI applications are “black boxes,” or decision-making processes that are opaque or at the very least difficult for humans to understand, rather than clear, explicable methods. However, calls for “Explainable AI” have started to reverberate through boardrooms, tech conferences, and research labs as government officials, activists, and the general public have grown more aware of the dangers associated with unfathomable algorithms—for example, thanks to publications like Cathy O’Neil’s Weapons of Math Destruction. Humans should be able to comprehend how and why an AI system makes a decision, according to the demand.
The problem may appear to be simple. Regulations such as the General Data Protection Regulation (GDPR) of the European Union may provide people with a “right to explanation” regarding automated decisions; advocacy organizations such as the Electronic Frontier Foundation, OpenAI, and the ACLU regularly emphasize the importance of AI transparency, particularly when the results directly affect people’s rights; and human rights groups and powerful tech companies collaborate to emphasize the transparent culture in their core values.
However, although certain systems have to be constructed with a certain level of explainability, other systems might not require that kind of openness. Take commonplace devices and software, such as smartphone autocorrect or recommendation algorithms for streaming services: Although it could be intriguing to know why a specific movie or song is recommended or why a word was selected to fix a typo, the intricacies of these algorithms don’t matter much for the typical user’s experience or level of faith in the system. The efficacy and dependability of the system are more important. The excessive amount of information that users may find redundant or even incomprehensible may result from the pursuit of unnecessary transparency in such instances. It would be like asking the driver of a car to comprehend every detail of its engine; while some may find it interesting, the majority are just happy that their vehicle will get them from point A to point B safely.
The Issue of the “Black Box” and the Need for Openness
Explainability is certainly important in some situations. Deep learning algorithms, for instance, have been applied in hospitals to forecast unexpected declines in patient health, such as sepsis or heart failure. However, the interpretive leaps that these models can uniquely give are dependent on intricate computations, even while they are capable of analyzing enormous volumes of patient data, including test results and vital signs, and warning physicians of possible issues. Clinicians may also be unclear about the precise processes and combinations of data pieces they employ to reach their judgments.
In life-or-death situations, in particular, this “black box” character of the model can make it difficult for doctors to fully trust its forecasts without knowing their reasoning. Because of this, AI models that identify skin lesions or malignancies also provide visual heatmaps with their diagnosis. These heatmaps show the precise regions of the skin image that the model deemed to be malignant, making it easier for dermatologists to comprehend the AI’s logic and focus. The AI technology enables medical professionals to “see” what the model is detecting by displaying areas of concern visually. This allows a doctor to cross-reference the AI’s findings with their expertise.
Homeland security is another example. With the ability to process enormous amounts of data at extremely fast speeds, sophisticated AI systems installed in surveillance cameras in stadiums, airports, and other large public venues have the potential to detect security threats based on behavioral patterns, facial recognition, and other factors. Though they can alert users to potentially suspicious situations or people, the complex logic that underlies these flags—which may include biometric information in addition to movement patterns and facial recognition—can be challenging to fully comprehend or explain.
Explainable systems in this area would not just flag an individual without context but would designate specific actions (e.g., an unattended bag) or behaviors (e.g., a person looking over their shoulder frequently) as indicators. The AI system provides insights into its decision-making process by labeling and describing particular actions or behaviors that are deemed suspicious. This helps security personnel not only make quick decisions and prevent errors but also refine and train the AI system further.
Within the legal arena, many jurisdictions—most notably those in the US and China—have begun utilizing artificial intelligence (AI) systems to assist in assessing the risk involved in providing bail or parole to persons. These algorithms analyze a variety of criteria, such as past behavior, family history, and more, to produce a risk score. However, even though population protection can make such systems a true asset, people are unable to reconstruct the logic that led to the decision. Although they offer a risk score, it might be difficult to determine the precise importance or weighting given to each component or how they interact. This ambiguity can be troublesome, particularly when it comes to the rights and liberties of individuals.
No Easy Solutions
One may wonder, why not just ensure that all AI systems can be explained?
First of all, the fundamental qualities of various machine learning models as well as the inconsistencies in data representation and decision-making need trade-offs between model explainability and performance. Since models are idealized, simplified depictions of reality, accuracy may have to be sacrificed to achieve tractability and explanatory power. In this respect, a model’s ability to explain and forecast phenomena can be greatly enhanced even if it is not quite “true” or exact. Certain models’ ability to capture and use all available information may be constrained by the very qualities that make them easier to interpret, such as their clarity of decision limits, simplicity, and reliance on fewer elements. Conversely, models that use larger amounts of data in more complex ways might result in powerful decision-making that is consequently more difficult for humans to express and comprehend.
To effectively utilize AI models, executives must first assess and set priorities. What is the ideal ratio between having a useful, efficient instrument for comprehension, prediction, and intervention and having an exact, accurate representation of reality?
Leaders should think about the following five factors when deciding which trade-offs are worthwhile:
1. Finding a balance between complexity and simplicity: More sophisticated models, such as deep neural networks, can attain higher accuracy because they can detect minute details and complicated relationships in data that more basic models could overlook. However, it may be the very complexity of the model that makes them more difficult to understand. For the same reasons why simpler models—like decision trees or linear regression—are simpler to comprehend, they may not fully capture all the nuances in the data.
Leaders must strike a balance between risk and reward to manage the trade-offs between complexity and simplicity.
Examine the possible risks connected with AI decision-making. If a poor choice could result in major expenses or harm, such as in medical or legal choices, give explainability priority, even if it means sacrificing some performance. To capture complex data patterns, a more complicated model could be needed. Leaders should aim to embrace appropriate complexity and make conscious choices about complexity depending on goals. However, decision-makers utilizing the AI outputs should always be aware of the interpretability limits built into the model.
2. Finding a balance between generalization and particularization: Highly interpretable models, such as shallow decision trees or linear regression, rely their conclusions on certain, general rules. However, these broad guidelines might not always identify precise or complex patterns in the data, which could result in less effective performance. More complicated models can recognize and use these finer-grained patterns, which makes it harder for humans to understand the outputs of these algorithms.
Leaders ought to recognize the significance of context. Not all uses of AI need the same level of explainability; for example, AI used to diagnose medical conditions might need more transparency than AI used to suggest movies. Stakeholders should also be educated on the subtleties of the model. It will be ensured that stakeholders are knowledgeable about the general rules and particular nuances of the AI system, as well as its limitations and any biases, by providing them with regular training.
3. Prioritizing adaptability over overfit: Overfitting refers to the tendency of extremely sophisticated models to “memorize” the training set, capturing noise instead of the true distribution of the data. Although this frequently leads to poor generalization of new, unknown data, it can result in great accuracy on training data. Therefore, although simpler and easier-to-understand models may be more robust and generalizable, they may not achieve as high accuracy on the training set.
Leaders should prioritize robustness over training accuracy and constantly evaluate and track the AI model’s performance on new data, as adaptation to new data is crucial. Additionally, leaders ought to prioritize feedback loops, particularly in domains that are crucial to their work. In other words, if an AI system makes a recommendation or forecast, human experts should have the last say and their judgments should be fed back into the AI model to improve it.
4. Engineering can make a model less readable: For simpler models to function well, significant feature engineering may be required. This may entail manually extracting new features from the data using domain expertise. However, while engineering features can improve the model’s performance, if the transformations are not clear, they may also make the model’s decisions more difficult to understand.
Leaders ought to consider interpretability while selecting engineered features. If feature engineering is important for your AI application, make sure the features don’t provide further opacity by striking a balance between improving efficiency and preserving clarity. Improved efficiency shouldn’t come at the expense of comprehension. Adopting a hybrid strategy that combines human and machine decision-making may be one way to find a solution. Human skills can give the required context and interpretability to ensure clarity, even while AI can digest data quickly and provide nuanced insights through feature engineering.
5. The balance between efficacy and computational efficiency: Highly complicated models may perform better but are computationally expensive to train and implement, even though simpler, interpretable models frequently demand less memory and processing power, making them more efficient for deployment. Consider the advantages of computational easiness and model simplicity vs the possible performance improvements of a more sophisticated, computationally intensive model when deploying AI models, especially in real-time applications. Often, if computational resources are limited, a more basic model may be adequate. Here, it’s especially critical to keep current and iterative because the field is developing quickly. Leaders should frequently review and improve AI deployments to make sure they continue to fulfill standards and requirements while maintaining the computational efficiency of the models.
A Reasonable Strategy
Neither total explainability at the expense of performance nor total performance at the expense of explainability is the intended outcome. To create a future that we can all be confident in, it is important to establish a balanced strategy that is suited to the particular risks and benefits of each AI application while also considering the effects on people and the environment.