Kratos · AI Tool Evaluation
Step 1 of 5
Coherense / Kratos Research

Welcome & Informed Consent

Tell us about yourself and review the consent form before beginning the AI tool evaluation.

About You

Informed Consent

Purpose

This study examines how pre-service teachers and higher education faculty evaluate AI tools for classroom use using the ECET AI Framework (Grotewold, Warren, & Beck, 2024). The purpose is to assess the usability and effectiveness of the Kratos evaluation instrument and to gather structured ratings of AI tools across pedagogical, feasibility, and ethical dimensions.

What You Will Do

You will describe an AI tool you have used or are considering for classroom use, then rate it across three bands — Idea, Feasibility, and Ethics — using a modified FMEA approach. For each dimension, you will give an overall quality score, describe any problems you identify, and (only if problems exist) rate their severity, occurrence, and learning impact.

Time Commitment

This evaluation is expected to take approximately 30–45 minutes. A follow-up focus group or interview may be requested separately.

Risks and Benefits

This study is minimal risk. Risks are similar to ordinary professional reflection activities and online surveys. There is no guaranteed personal benefit, though your feedback may inform tool revision and framework development.

Confidentiality

Data will be stored in secure, password-protected files or university-approved cloud storage. Reports will use de-identified summaries. Records will be retained for at least three years past the end of the research.

Compensation

No compensation will be offered.

Contact

Questions about the study: Dr. Scott J. Warren, scott.warren@unt.edu, 940-369-7489. Questions about your rights as a research subject: UNT Office of Research Integrity and Compliance, 940-565-4643, untirb@unt.edu.

How the Rating System Works

Each ECET band contains six evaluation dimensions. For each dimension, you will complete three steps in order:

1
Score the dimension (1–5)
Select the button that best describes overall quality. Every point is labeled so you know exactly what each score means.
2
Describe any problems (optional)
If you identified an issue, describe it briefly. You may leave this blank if there are no concerns.
3
Rate the risk with FMEA sliders (only if a problem exists)
FMEA sliders appear automatically when you describe a problem. Rate Severity, Occurrence, Learning Impact, and Usability on a 1–5 scale. Each point is labeled. Scores multiply into a Risk Priority Number (RPN) from 1–625.
Severity (S)
How serious is this problem if present? 1 = negligible · 5 = catastrophic
Occurrence (O)
How often does this problem arise? 1 = rare edge case · 5 = happens constantly
Learning Impact (L)
Effect on teaching and learning outcomes? 1 = no effect · 5 = severely disrupts learning
Usability (U)
How hard is this to address using Kratos? 1 = easily resolved · 5 = very difficult

RPN = S × O × L × U (max 625 per item). RPN ≥ 250 = high priority. RPN ≥ 400 = critical.

Step 2 of 5

Describe the AI Tool

Identify the tool you will evaluate. Be as specific as possible — this context shapes your ECET ratings.

🔍
Before you begin rating
Name and briefly describe the AI tool you want to evaluate. Your description provides context for the three rating bands that follow — Idea, Feasibility, and Ethics. The more specific you are here, the more meaningful your ratings will be.
Tool Profile Tool Identification
Please enter the AI tool name to continue.
Band 1 of 3 — ECET Framework

Idea Band

Rate the conceptual and pedagogical soundness of using this AI tool across six dimensions. See the quick-reference guide below if you need a reminder of how the three-step process works.

Idea
—
Feasibility
—
Ethics
—
Total RPN
— /11,250
How to complete each question
1
Score it (1–5). Select the button that best fits. Every button shows a plain-language label — hover or read the text below each number.
2
Describe any problem in the text box. If you see no issue, leave it blank.
3
FMEA sliders appear automatically only when you enter a problem. Rate Severity, Occurrence, Learning Impact, and Usability on 1–5 — each point is labeled. The four scores multiply into a Risk Priority Number (RPN).
Idea Band
Pedagogical Alignment & Purpose
Does the concept of using this tool make sense for your teaching context?

Idea Band — Open Feedback

Band 2 of 3 — ECET Framework

Feasibility Band

Rate the practical realities of implementing this AI tool across six dimensions. The reminder card below keeps the process visible — no need to go back to the introduction.

Idea
—
Feasibility
—
Ethics
—
Total RPN
— /11,250
Same process as Band 1
1 Score 1–5 → 2 Describe any problem → 3 FMEA sliders appear if needed
This band focuses on practical implementation — access, cost, policy, time, reliability, and sustainability. Think about your specific institutional context as you rate each dimension.
Feasibility Band
Implementation & Practical Viability
Can this tool realistically be implemented in your institutional context?

Feasibility Band — Open Feedback

Band 3 of 3 — ECET Framework

Ethics Band

Rate the ethical dimensions of adopting this AI tool across six dimensions. The reminder card below recaps the rating process and explains what each score means in an ethics context.

Idea
—
Feasibility
—
Ethics
—
Total RPN
— /11,250
Same process as Bands 1 and 2
1 Score 1–5 → 2 Describe any problem → 3 FMEA sliders appear if needed
This band focuses on ethical responsibilities — privacy, bias, transparency, academic integrity, learner autonomy, and broader social impact. A score of 5 means no significant concern was identified. A score of 1 means a serious concern with no clear mitigation.
Ethics Band
Responsibility, Equity & Integrity
Does using this tool raise ethical concerns that must be addressed before or during adoption?

Ethics Band — Open Feedback

Cross-Band Wrap-Up

Step 5 of 5

Review & Submit

Your evaluation is complete. Review the summary below, then download your files and send them to the research team.

Evaluation Summary

Download Your Files

Please complete all three steps below.

1
Click Download CSV to save your raw rating data.
2
Click Download HTML Report to save your formatted evaluation summary.
3
Email both files to the research team using the button below. Attach both downloaded files to the same message.

Email to Research Team

Send both downloaded files as attachments to scott.warren@unt.edu. Use the subject line below — the button will pre-fill it for you.

Kratos Evaluation — [Your Name] — [Tool Name]
Thank you for completing this evaluation. Your feedback will directly inform the refinement of the Kratos instrument and the ECET AI Framework.