The goal of this project is to create a benchmark dataset for the research field of visual working memory. This idea is inspired by the critical role such datasets have played in the AI revolution. Benchmarks help shift research from the competition of storytelling to the more objective challenge of beating the best score on a benchmark. If this approach can be applied to psychology, it could create a level playing field and perhaps lead to a similar breakthrough.

This benchmark project shares some similarities with traditional adversarial collaboration but differs in key aspects:

Similarity: It brings together researchers with differing perspectives to agree on a methodological approach.

Difference (Pre-Data Collection): Unlike adversarial collaboration, we will not define specific hypotheses or predictions beforehand. Instead, the focus will be on identifying the most informative experimental manipulations and measures. This reduces the burden of upfront theoretical specification.

Difference (Post-Data Collection): Unlike adversarial collaboration, we will not seek a mutually agreed-upon interpretation, model, or theory for the benchmark dataset. Instead, each contributor may independently develop and publish their own interpretations. This approach removes the need for premature consensus, which often discourages collaborators from tackling the most challenging questions. Moreover, after a protection period, the dataset will be made openly accessible. Therefore, researchers—including but not limited to the original contributors—can propose and test competing models or theories. Over time, the most robust solutions will emerge through fair, community-driven scrutiny—a strategy that has proven highly effective in AI and other data-intensive disciplines.

Basic information

This section provides essential resources for understanding the project:

1. FAQ Document: Addresses frequently asked questions.

2. Methodology Files: Archives all versions of the methodology documentation.

Methodology 1.0 – Published: December 10, 2025

Coordinators (for all inquiries, please email Zhou Su) 

Other 49 contributors