With the proliferation of generative AI and its tendency to “hallucinate,” there is concern about how the predictions these software algorithms make can be trusted. Because that is what AI models do, predict – when AI makes recommendations or suggestions, it’s simply sharing the highest or most likely probability of what the user is looking for or may be interested in.
Uncannily accurate in many cases, large language models (LLM) used in generative AI produce text-based responses that the model thinks are the most likely words to respond to users’ questions. That is why “prompting,” or asking a question in a specific way with as much detail as possible, is very important – it literally helps narrow down the likely options for the answer.
Therefore, it is essential for people to understand where AI is being used and how, because they have a responsibility to act: do they accept a prediction or reject it?
Governments like the European Union, the State of California, and the city of New York are establishing requirements to make sure buyers and users have this awareness, know the risks of using AI models, understand how to gauge what confidence to have, and even completely opt out if they don’t trust the system.
How can we know AI can be trusted?
The U.S. National Institute of Standards and Technology (NIST) within the U.S. Department of Commerce has established a framework and guidance around delivering trustworthy AI. Referencing common definitions from the International Organization for Standardization (ISO) (whose stated purpose is “making lives easier, safer, and better”), trustworthy AI models and systems are:
1. Valid and reliable
Is there documented objective evidence that the model is working as intended? Does it continue to perform as required over time?
2. Safe, secure, and resilient
To be safe, AI systems should not “lead to a state where human life, health, property, or the environment is endangered.” Practical approaches, NIST notes, include testing, monitoring, and the ability to have human intervention. It says AI systems may be said to be secure if they maintain confidentiality, integrity, and availability through protection mechanisms that prevent unauthorized access and use. Resiliency refers to being able to withstand adverse events or unexpected changes and still perform acceptably.
3. Accountable and transparent
NIST unequivocally states: “trustworthy AI depends on accountability. Accountability presupposes transparency.” Transparency “reflects the extent to which information about an AI system and its outputs is available to individuals interacting with such a system, regardless of whether they are even aware that they are doing so.”
4. Explainable and interpretable
NIST sums up: “explainability refers to a representation of the mechanisms underlying AI systems’ operation, whereas interpretability refers to the meaning of AI systems’ output in the context of their designed functional purposes.” In simpler language: “explainability can answer the question of “how” a decision was made in the system. Interpretability can answer the question of “why” a decision was made by the system and its meaning or context to the user.”
5. Privacy-enhanced
Privacy refers generally to the norms and practices that help to safeguard human autonomy, identity, and dignity. Typically, these address freedom from intrusion, limiting observation, or individuals’ agency to consent to disclosure or control of facets of their identities, says NIST.
6. Fair with harmful bias managed
This is perhaps the most nuanced part of developing trustworthy AI. NIST explains, “Fairness in AI includes concerns for equality and equity by addressing issues such as harmful bias and discrimination” acknowledging “bias is tightly associated with the concepts of transparency as well as fairness in society.”
Transparency Above All is Essential
Of note, NIST says “a transparent system is not necessarily an accurate, privacy-enhanced, secure, or fair system. However, it is difficult to determine whether an opaque system possesses such characteristics, and to do so over time as complex systems evolve.”
So arguably, the greatest of these is transparency. The good thing is that there are many effective user experience approaches and mechanisms we can use to be transparent about AI in the workplace today.
1. AI features should be opt-in across systems
Feature toggles or activation settings within software solutions are actually a best practice given the rapid pace of cloud application innovation today and lagging end-user adoption from change management overload. These toggles should be mandatory for any AI applications.
2. Indicate where AI is being invoked when it is turned on through visual cues
People should look for consistent iconography or design language system (DLS) elements that add an appropriate level of information on where AI is being applied and what users’ responsibilities are in using it. While there seems to be a common star or spark icon for AI features emerging across consumer and enterprise apps, this does not nearly go far enough.
3. Offer guidance on features and functions
System developers should offer clear plain language guidance to users on features or system functions, through additional hover-over information expansion options or links to system aides, documentation, or training. There could even be settings to dial the level of explainability up or down based on application or organizational sophistication.
4. AI feature model details should be accessible to administrators and IT leadership
Typical “model cards” can include a description of the AI feature, the business case or intent of the model, detail on model type, version, data used to train and test the model, evaluation methods and performance metrics for the model, any limitations, and how the model will be monitored. Companies should ask for and expect this level of documentation with each feature or system update.
5. System administrations should have central control
System administrators should also have a governance console or set of capabilities to centrally control and oversee AI applications, user access, and prediction confidence settings. This is particularly true as companies look to fine-tune AI models to work with their specific organizational or industry specific information. Here is where they can toggle on and off features and monitor their performance as it relates to their organizations’ specific information, company brand, and culture.
6. Testing environments should be offered to organizations
Although AI companies have responded to consumer curiosity in AI with free “experimental access” to new algorithms or model innovations, organizations should be offered testing environments that let them experiment in controlled situations before rolling out company-wide (which is also good practice in any kind of SaaS innovation delivery.)
7. Include acknowledgment of conversational AI
You may ask, what happens when AI starts enabling conversational access and there may be nothing to see? Take digital assistants like Siri as an example of "invisible AI." There are also established mechanisms for having people acknowledge when a system may suggest something, but the user decides differently. It’s called “attestation.” With attestation, users could formally acknowledge for the record that they are aware AI is in use and what their roles are in interacting with its outputs. This could be required before access to systems or features are permitted initially and as models and features evolve.
Some of these practices are already being anticipated by users. In a 2023 survey of 2,000 people by the online media outlet The Verge:
- 78% believe AI-created digital content should clearly state it was created with AI.
- 76% believe AI models should be required to be trained on datasets that have been fact checked.
- 76% believe regulations and laws need to be developed regarding the development of AI.
Given this consumer sentiment and the growing legislation referenced above, both buyers and developers of AI-driven solutions should be looking at every method to give users trust in the AI algorithms in use and the predictions they deliver. Because as AI features get more sophisticated and interconnected models or software “agents” start to offer more complex capabilities, organizations are going to need even more confidence they understand what they are capable of and retain control of these systems.
