Artificial intelligence is like a machine that learns to make decisions by looking at many examples.
These examples are, much like for a human, experiences—experiences evaluated through statistical criteria. The more often a thought, reaction, behavior, or piece of information appears across cases, the more strongly it will shape the AI’s response patterns.
Let’s suppose the AI is shown thousands of cases of people who applied for a loan.
The AI learns patterns. And then it can decide whether a credit application is “approved” or “not approved.”
However, in this process of gathering information and reasoning toward a decision, the inner workings are not clear. This is what has been called the “black box” of AI.
Many modern AIs (especially the powerful ones) operate as black boxes: they provide an answer but do not reveal why or how they reached it.
How would you react if you went to a doctor, received a prescription, and upon asking what it was for, the doctor said: “Well, that doesn’t matter. What matters is that you take it. Don’t you trust me?”
AI operates in highly sensitive areas of our reality. This habit multiplies risks and could lead to extremely serious problems. Banking, healthcare, and justice systems use AI constantly.
What solutions have been proposed recently to address this potential source of trouble? This is where XAI, or Explainable Artificial Intelligence, comes in.
The goal is to understand how the AI processes and interprets the information it is trained on.
To achieve this, a set of techniques has been developed to make transparent the opaque information that makes up the “black box.”
In Europe, for example, the EU AI Act already requires transparency/explainability for “high-risk” systems (such as credit or scoring). This makes XAI almost mandatory in many industries.
The purpose is to:
build greater trust in the tool and its outputs; detect bias; and identify and correct errors when needed.
Let’s consider an example. Imagine a credit decision engine whose primary function is to approve or deny loans, recently adopted by a bank that wants to speed up its processes.
The model does not “think” like a human: it calculates probabilities based on patterns learned from thousands of previous loans.
It must evaluate and decide whether to “approve” or “deny” the credit request. If it provides only the final result, it behaves like a black box.
Due to regulation and the need for trust, the bank will instead require a clear explanation. That is where Explainable AI (XAI) enters.
Let’s take the case of Ana R., 29 years old, recorded by the system with a monthly income of $1,200,000. She has six months of job tenure and a current debt of 800,000 pesos. Her payment history shows two late payments in the past year. The system also calculates that the potential loan installment would represent 45% of her salary.
The verdict: “Not approved.”
Both the financial institution and the applicant—in this case, Ana R.—will naturally ask: “Why not?”
And this is where XAI provides transparency and validates its reasoning. It does so through three layers:
1) “What mattered most” explanation
XAI uses techniques such as SHAP or LIME to indicate which variables pushed the decision toward “not approved.”
The system might return something like:
- Payment history with late payments → strong negative impact
- Installment-to-income ratio 45% → high risk
- Job tenure: 6 months → low stability
- Income $1,200,000 → neutral / slightly negative
- Age 29 → almost no influence
This is extremely useful for analysts because it turns the “no” into something understandable. SHAP is widely used for exactly this reason in credit scoring.
How should the human agent explain this to the customer?
“We rejected your application mainly because of recent late payments and because the installment would be too heavy compared to your income.”
2) Counterfactual explanation: “What would need to change for approval”
This is the clearest and easiest part for a customer to understand.
The system might answer:
“With your current situation, the loan is not approved.
If your installment-to-income ratio dropped to 30%
or if you had 0 late payments within 12 months, the decision would change to ‘approved.’”
This explanation gives hope that improving these conditions may lead to approval in the future. It may also suggest that applying for a smaller amount could result in immediate approval.
In other words:
“Ana, if you request a slightly lower amount (or more installments) so that the payment burden decreases, we would approve it.”
Or: “If you go a full year without late payments, we would approve it.”
3) Audit-ready explanation: “simple rules”
Sometimes the bank needs rules it can show to regulators or internal audit teams.
XAI might summarize it like this:
“Rejected due to: recent late payments > 1; and installment/income ratio > 40%.”
This supports model risk management, a practice that banking regulators have increasingly reinforced.
This system evolved from an innovation to an obligation in a very short time. The reason is that credit systems are classified as high-risk AI in Europe, and the law requires: transparency, clear and understandable explanations,
traceability, and continuous monitoring.
This means that it isn’t enough for AI to be accurate. It must justify its decision transparently.
In summary
Without XAI, the conversation would be:
Customer: “Why was I rejected?”
Bank: “We don’t know, the model said so.”
With XAI, the bank instead says:
- “We rejected your application for three main reasons.”
- “If you change X or Y, we will approve it.”
- “Our decision complies with auditable rules.”
And the results for the financial institution are:
- Greater trust
- Lower legal risk
- Better customer experience
- More opportunities to improve each process
