A 2 × 2 factorial randomized controlled trial crossing role modeling (present / absent) with goal setting (present / absent), delivered online in a single scheduled class session.
Figure 1. STEM intervention design and condition content
Panel A. Trial design and process
Panel B. Condition content and dose
Every condition had the same 35 minutes of content. To fit both components into that time, the combined condition used a shorter version of each: three of the five role-model videos, and the goal-setting sequence up to the SMART goal, without the goal review, strategy choices or if–then plan.
Table 1. Condition content and planned dose
| Component element | Active control | GS only | RM only | Combined |
|---|---|---|---|---|
| Introductory videos (STEM importance, SMART goals, setting goals) | — | 3 videos | — | 1 of 3 (SMART goals) |
| Task-value prompt (open written response on STEM relevance) | — | 1 prompt | — | 1 prompt |
| Effort, help-seeking and prior-goals questions | — | 3 items | — | 3 items |
| SMART goal (5 elements, menu or own wording) | — | 5 elements | — | 5 elements (different menu options) |
| Worked SMART example (shown on the page) | — | 1 | — | 1 |
| Goal review (specific, timeframes, measurable, achievable, effort, realistic) | — | 6 ratings | — | Not delivered |
| Strategy choices (distraction, difficulty) | — | 2 items | — | Not delivered |
| Free-text rewrite of goal with if–then plan | — | 6 fields | — | Not delivered |
| Closing items (obstacles, open to adjusting, confidence) | — | 3 items | — | Not delivered |
| Role-model videos (ATSE, women in STEM, ≈ 3 min each) | — | — | 5 videos | 3 videos |
| Post-video career ratings (4 items per video) | — | — | 20 responses | 12 responses |
| Most inspiring video (single choice) | — | — | 1 item | 1 item |
| Active-control videos (TED-Ed) | 4 videos | — | — | — |
| Post-video response items (engagement, attention, ease, effort, use of time) | 12 items (3 per video) | — | — | — |
| Condition-specific chatbot (GPT-4o) | Available | Available | Available | Available |
| SEVT survey (12 items × 4 domains, T1 and T2) | 48 × 2 | 48 × 2 | 48 × 2 | 48 × 2 |
| Session length | 35 min | 35 min | 35 min | 35 min |
Data were collected in six Victorian secondary schools during a single scheduled class session, using REDCap.
Newcomers. Students who missed the scheduled session (e.g., excursions, illness or technical problems) completed the same sequence when they next attended and were randomized the same way. These 192 students (22.5%) were evenly spread across conditions (χ²(3) = 0.57, p = .904) and school locations (χ²(1) = 2.11, p = .147), and newcomer status was included as a covariate.
Manipulation checks. Students who received goal-setting content were more likely to report having made a SMART goal than those who did not (90.0% vs. 36.7%), and students who received role-modeling content were more likely to report seeing a STEM role model (95.3% vs. 49.8%; both p < .001).
Twelve items drawn from established scales (Kosovich et al., 2015; Gaspard et al., 2015; Flake et al., 2015) measured four beliefs from situated expectancy-value theory. Each item was asked separately for biology, chemistry, physics and STEM overall (48 responses per wave), on a 7-point scale from Strongly disagree to Strongly agree. The order of subject blocks and items was randomized. Reliability was good to excellent (α and ω = .83–.97).
Table 2. Survey items · [subject] replaced by each domain
| Construct | Item |
|---|---|
| Expectancy | I know I can learn the material in [subject]. |
| Expectancy | I believe that I can be successful in [subject]. |
| Expectancy | I am confident that I can understand the material in [subject]. |
| Intrinsic value | [Subject] is fun for me. |
| Intrinsic value | I enjoy doing [subject]. |
| Intrinsic value | [Subject] is interesting. |
| Utility value | I think [subject] is important. |
| Utility value | I value [subject]. |
| Utility value | I think [subject] is useful. |
| Cost | [Subject] requires too much time. |
| Cost | I have to give up too much to do well in [subject]. |
| Emotional cost | [Subject] is too stressful. |