04Generative AI · Quantitative UX · Read 10 min

Evaluating a vision-language model against human UX judgments

Testing whether a multimodal model recovers human first impressions of real websites, then locating precisely where its judgment diverges.

IN ONE SENTENCE

The model partially recovers the ordering of human judgments, with an overall correlation of 0.42, but its positive bias prevents direct use of its absolute scores.

CONTEXT

A product team may want to shortlist variants before a user study or A/B test. I investigated whether a vision-language model could provide that first filter from a screenshot by testing it against a scientific dataset of homepages evaluated by human participants.

SUMMARY

I built a pipeline that sends the same 100 fashion-site screenshots to GPT-4.1 mini using six questions from the original protocol. The resulting 600 scores were joined with aggregated human ratings and analyzed through Pearson and Spearman correlations, mean absolute error and a leave-one-out comparison across human raters.

Key diagram

From a real website to comparison with human judgment

01 · DATASET100real website screenshots

Fashion ecommerce · full page

→EVALUATE

02 · VISION-LANGUAGE MODELGPT-4.1mini

AEAVGEXMPLTRUTYPUS

−3.0 ↔ +3.0 · 600 scores

↔COMPARE

03 · HUMAN REFERENCE

Aggregated consensus

Same screenshot · same dimensions

04 · PEARSON r

Aesthetics0.34

Resemblance0.51

Exemplar0.54

Trust0.43

Typicality0.44

Usability0.49

05 · INTERPRETATION

r = 0.416overall ranking+1.60 → +2.15model means

Useful relative signal · insufficient calibration

Research prototype · human validation required

The same design receives six model scores and six reference scores. Correlations measure ranking consistency; the level gap exposes calibration bias.

A case study focused on transferable methods, decisions and learnings. No client data or confidential information is disclosed.

Contribution & result04

Can a vision-language model shortlist designs before UX research or an A/B test?

I built a pipeline aligned with a human benchmark, ran 600 evaluations with GPT-4.1 mini and measured both relative ranking and calibration. The 0.416 overall correlation reveals a useful research signal, while positive bias rules out treating the scores as absolute verdicts.

Evidence & guardrails

Pearson and Spearman correlations

Absolute error and mean bias

Out-of-domain replication

Key idea

The model captures a useful signal for coarse interface comparison, especially on usability and representativeness. Calibration, replication across domains and product validation remain necessary before any decision is automated.

01 — Research question

Can a multimodal model reproduce a UX first impression?

The public benchmark combines full-page homepage screenshots with human ratings. Its dimensions cover aesthetics, perceived usability, trustworthiness, typicality, family resemblance and exemplar goodness.

The experiment tests one measurable capability: recovering the relative ordering of human judgments on websites that were not designed for this benchmark. It does not measure actual use after interaction and therefore does not replace UX research or A/B testing.

02 — Experimental pipeline

Items, scale and screenshots remain aligned with the reference study.

Each screenshot is sent to the model with the six original items and anchors. The model returns one continuous score between −3 and +3 plus a structured qualitative rationale. An incremental cache avoids recomputing completed evaluations.

The reported experiment covers 100 fashion websites, producing 600 image-dimension pairs. Outputs are stored in Parquet and joined with human scores through each screenshot identifier, enabling analysis of relative ranking and absolute error.

03 — Results

The ranking contains signal; calibration remains insufficient.

Overall Pearson correlation reaches 0.416. It ranges from 0.343 for aesthetics to 0.535 for exemplar goodness, with 0.486 for usability and 0.513 for family resemblance.

The model nevertheless assigns overly positive values: dimension means range approximately from +1.60 to +2.15 while standardized human ratings remain centered near zero. Overall mean absolute error reaches 1.98 points on the six-point scale. The model ranks better than it calibrates.

04 — Decision

Using the model as a comparison instrument requires explicit guardrails.

The result supports exploration as a shortlisting and research tool: comparing variants, finding disagreements and generating hypotheses to validate with users. It does not support automatic judgments of design quality.

The next methodological steps include calibration on a held-out sample, replication across homeware, banking and higher-education websites, uncertainty intervals, and comparison with decisions and behavior observed in actual product experiments.

Evidence frame

Research prototype using a public design-perception benchmark

An executed experiment on 100 fashion-site screenshots from the Web Design Prototypicality Dataset. Results describe a research prototype, not a production service.

Methods and references

Miniukovich & Figl (2024) — Web Design Prototypicality Dataset
GPT-4.1 mini with image input
Pearson and Spearman correlations
Mean absolute error and leave-one-out analysis