AI Evaluation Framework for Youth-Serving Organizations

A Framework for Evaluating AI Tools in Youth Programs When assessing whether to adopt, modify, or reject an AI-powered tool for your youth program, run it through these four lenses: ________________________________________

🔍 LENS 1: EQUITY Who benefits and who might be harmed? Core Questions: • Does this tool work equally well for all the youth we serve? • Could it disadvantage youth based on race, language, disability, neighborhood, or prior opportunity? • Who was included in designing and testing this tool? • Does adopting this tool widen or narrow existing gaps?

Warning Signs: • Training data from narrow populations • No disaggregated outcomes data available • Cost or technology requirements that exclude some youth

🔍 LENS 2: TRANSPARENCY Can we explain how this works to young people and families? Core Questions: • Can staff clearly explain what data is collected and how it's used? • Can youth understand why they received a particular score, match, or recommendation? • Is the "black box" problem acceptable for this use case? • What's disclosed vs. hidden about how the AI makes decisions?

Warning Signs: • "Proprietary algorithm" with no explanation available • Youth can see outputs but not inputs • Consent forms are unreadable or buried

🔍 LENS 3: AGENCY Does this increase or decrease youth choice and voice? Core Questions: • Can youth opt out without penalty? • Does the tool open doors or close them? • Are youth subjects of the AI or participants in shaping it? • Does automation replace meaningful human interaction?

Warning Signs: • Youth have no say in whether to participate • Scores or rankings are presented as fixed truths • Efficiency gains come at the cost of relationship-building

🔍 LENS 4: ACCOUNTABILITY If something goes wrong, who's responsible and what's the recourse? Core Questions: • Who owns the data and what happens if the vendor disappears? • What's the process if a youth believes they were treated unfairly? • Is there human review of high-stakes AI decisions? • Who can override the algorithm?

Warning Signs: • No appeals process for algorithmic decisions • Data deletion is difficult or impossible • Vendor contracts have no accountability clauses