A practical guide to reviewing facial expressions in video
Reviewing a face in video starts with observation, not a conclusion about someone's feelings. This guide offers a practical process for keeping movement descriptions, interpretation, and uncertainty separate.
Published and updated November 1, 2026.
Set a purpose and get consent
Write down the question before collecting or opening any participant video. A useful question concerns visible movement during a clearly defined recording task. Avoid questions that require guessing someone's honesty, diagnosis, or suitability from facial appearance.
Explain the review purpose, who will see recordings, and how results will be discussed. Ask for informed consent appropriate to the project and keep participation genuinely voluntary. Decide how long recordings and annotations are needed before work begins. These are planning recommendations, not a claim that one process satisfies every legal requirement.
Check capture quality before interpretation
Inspect lighting, face angle, occlusion, and playback quality before marking any facial movement. Ask whether shadows, head turns, glasses, or hands obscure the feature under review. Note the recording's frame rate when available, without assuming every camera preserves brief events.
Review sample footage at its original playback speed before selecting a coding method. If a movement cannot be distinguished reliably, label that interval uncodeable rather than guessing. A missing annotation should not silently become evidence that no movement occurred.
Build a codebook around observable movement
FACS describes visually discernible facial movement through anatomically based Action Units, according to Paul Ekman Group. Its vocabulary provides a way to describe a movement without naming an internal emotion. Start with the features your question actually requires instead of adding every possible label.
Define each code, its inclusion criteria, its exclusions, and how onset and offset are marked. Include an explicit option for uncertainty and a separate option for insufficient visibility. Record whether your project uses FACS coding, a simplified movement scheme, or software estimates.
Wikipedia's FACS overview describes intensity from A, trace, through E, maximum. It also describes side and asymmetry modifiers, which require careful definitions in your codebook. Do not claim formal FACS proficiency simply because your spreadsheet contains AU names.
Code at the right speed
Haggard and Isaacs reviewed filmed psychotherapy sessions using slowed playback in their 1966 work. That historical method supports a practical distinction between viewing an interaction and inspecting a brief interval. Review the original sequence, then slow the relevant moment when your tools permit.
Paul Ekman Group describes macro expressions as lasting about one-half second to four seconds. It describes micro expressions as one-half second or less, sometimes as short as one twenty-fifth second. Definitions vary between researchers, so state the duration convention used in your own project.
Record time boundaries and explain how reviewers selected them, rather than treating playback impressions as precise measurements. Return to normal speed before interpreting the surrounding exchange. Preserve nearby context, including head movement and what happened before and after the selected moment.
Use independent coders and measure agreement
The FACS overview recommends agreement between at least two independent coders for reliable coding. Have reviewers code the same practice material independently before discussing their differences. Resolve unclear instructions in the codebook, then use separate clips to check the revised rules.
Decide whether agreement concerns presence, intensity, event boundaries, or another clearly stated variable. Do not combine these different questions into an unexplained overall quality score. Preserve original annotations when producing an agreed version so disagreement remains visible.
As of November 1, 2026, Noldus describes Cohen's kappa in The Observer XT's inter-rater reliability features. Choose an agreement measure suited to your annotation task and explain how it was applied. A software option alone does not establish that your coding decisions are dependable.
Choose tools around the review task
As of November 1, 2026, BORIS supports video, audio, and live-observation event logging under GPL v3. Friard and Gamba's 2016 paper provides its scholarly reference. Its published platform list covers Linux, Windows, and macOS.
As of November 1, 2026, ELAN provides time-aligned annotations across multiple hierarchical tiers, saved as EAF XML. As of November 1, 2026, Datavyu is a free, open-source Java-based video coding tool for Windows and Mac. Its published description includes Ruby scripting and its origins in OpenSHAPA.
Select tools by whether your codebook needs event intervals, parallel annotation tiers, or scripted review steps. Check exports using a small sample before committing a whole collection. Keep clip references understandable outside the original application so another reviewer can inspect the same moment.
Treat software output as a second opinion
Decide what a software score is allowed to contribute before reviewers see it. You might use estimates to suggest moments for inspection, while preserving independent human annotations. Avoid rewriting uncertain observations merely to make them match a model's labels.
Return to the source moment whenever a score seems important or surprising. Compare it with visible movement, recording quality, nearby context, and the codebook. Keep a clear distinction between a system's expression category and a participant's reported experience.
Know the interpretation limits
Barrett and colleagues describe substantial variation across people, cultures, and situations in their 2019 review. The same facial configuration can accompany different emotions or no emotion. A face alone therefore cannot establish the person's internal experience.
Jack and colleagues reported differences between Western and East Asian representations of basic expressions in 2012. A reply argued that commonalities outweigh differences, so describe this as an active debate. Do not use either position as permission to ignore an individual participant's context.
Consider law before selecting a use case
EU AI Act Article 5(1)(f) prohibits workplace and educational emotion inference, subject to medical or safety exceptions. Those prohibitions apply from February 2, 2025. Calling an output a signal does not automatically resolve whether a proposed use falls within that restriction.
Talk to counsel about your use case. This guide is not legal advice. Do not repurpose a consented research review into hiring, student scoring, diagnosis, surveillance, or threat assessment.
Keep a practical review checklist
Before review, confirm the purpose, consent process, access rules, and retention plan. During review, record visibility, timing, movement definitions, uncertainty, and independent coding decisions. After review, inspect disagreements and explain what the annotations do not establish.
Check that every reported observation can be traced back to its source interval. Remove claims that go beyond the codebook or available evidence. Make sure the final explanation distinguishes observation, software estimates, interpretation, and unresolved questions.
Where Mental Edge fits
Mental Edge surfaces time-coded facial-expression and movement signals from recorded video for contextual human review. It associates frame-by-frame scores with seven expression categories, rather than proving feelings or intent. Those patterns may indicate moments worth reviewing, not conclusions about someone's inner state.
Use the source video and your own review rules alongside any scores. Mental Edge is not for hiring, promotion, diagnosis, deception detection, threat assessment, student scoring, or surveillance. The guide remains openly readable, and the optional signup below offers future guides and product updates.