Machine Learning - Ethics
Principles
- Fairness: measure and mitigate disparate impact across groups.
- Privacy: minimize data, protect PII, and apply privacy-preserving techniques (DP, FL).
- Transparency: explainability, documentation (model/data cards), reproducibility.
- Safety: monitor for harmful outcomes; define redlines and escalation paths.
Practical checks
# Examples of checks to operationalize ethics
# - Track per-group metrics: FPR, FNR, calibration, acceptance rate
# - Run counterfactual tests: flip a sensitive attribute and compare predictions
# - Document dataset provenance and known limitations
Fairness metrics (walkthrough)
Common group fairness metrics (for a positive class) include:
- Demographic Parity (DP): P(ŷ=1 | A=a) is similar across groups a.
- Equal Opportunity (EOpp): TPR is similar across groups.
- Equalized Odds (EOdds): both TPR and FPR are similar across groups.
- Calibration within groups: predicted scores are calibrated for each group.
# Pseudocode (Python) to compute per-group TPR/FPR
# y_true: true labels; y_pred: predicted labels; A: sensitive attribute
from collections import defaultdict
def rates(y_true, y_pred, A):
by = defaultdict(lambda: {"TP":0,"FP":0,"TN":0,"FN":0})
for yt, yp, a in zip(y_true, y_pred, A):
b = by[a]
if yt==1 and yp==1: b["TP"]+=1
elif yt==0 and yp==1: b["FP"]+=1
elif yt==0 and yp==0: b["TN"]+=1
else: b["FN"]+=1
out = {}
for a,b in by.items():
tpr = b["TP"]/max(1,(b["TP"]+b["FN"]))
fpr = b["FP"]/max(1,(b["FP"]+b["TN"]))
out[a] = {"TPR": tpr, "FPR": fpr}
return out