Multimodal AI · Case study
Hate & Threat Detection in Forensics
A reproducible, image-centric forensic pipeline that distinguishes embedded text, associated message context, and image-only evidence before producing a traceable case-level decision record.
Multimodal AIProject overview
Hate & Threat Detection in Forensics
A reproducible, image-centric forensic pipeline that distinguishes embedded text, associated message context, and image-only evidence before producing a traceable case-level decision record.
- Context
- Multimodal digital forensics
- Role
- Author and developer
- Evidence
- Image, OCR, and associated SMS text
- Output
- Fused decision CSV with source scores
- Paper
- arXiv:2604.08609 (2026)
The problem
Forensic meaning is often split across several evidence sources
A single image can contain visual symbols, embedded text, and surrounding language that change the interpretation of a case. A single-model decision can miss this context and make it difficult for an analyst to understand which evidence influenced the result.
The system
Specialized analysis before transparent score fusion
The pipeline first determines whether the image contains embedded text, nearby message context, both, or neither. It then routes only the available modalities through OpenCLIP and DeBERTa before preserving source scores in the final output.
What I built
A reproducible pipeline built for analyst inspection
Created source-aware routing for visual evidence, OCR-derived text, and associated language.
Applied OpenCLIP-based visual scoring and DeBERTa-based language analysis within one controlled workflow.
Implemented score-level fusion across multiple evidence signals without hiding the contributing sources.
Added CSV evidence logging so cases, model outputs, and fused assessments can be inspected after a run.
Structured the project for command-line execution, repeatable experiments, and regression testing.
Documented the research system and connected the public implementation to the paper.
Quality and evaluation
Traceability supports both error analysis and responsible use
The workflow keeps modality-specific outputs available for error analysis instead of presenting only a final label.
The system is research software that supports forensic review. Its assessments require human interpretation and should not be treated as autonomous legal or investigative conclusions.
Technology
Evidence