MineBench evaluates spatial reasoning through 3D voxel construction.
Models receive natural-language prompts and construct the requested scene programmatically. Their outputs are rendered into 3D builds and compared through blind human pairwise voting.
The evaluation primarily tests spatial reasoning, 3D planning, structural decomposition, proportions, composition, prompt interpretation, and coherence across large structured outputs.
Models generate JavaScript using MineBench's voxel-building API.
The API exposes primitives for constructing geometry, such as blocks, boxes, and lines. The model writes JavaScript that calls those primitives to construct the requested scene.
MineBench executes that JavaScript in a sandbox, collects the generated voxel operations, converts them into the final normalized voxel JSON, validates the result, and renders it into the 3D scene shown on the site.
The model is therefore generating a program that constructs the build. It is not generating an image, directly controlling Minecraft, or receiving a reference image to reproduce.
The system prompt intentionally allows models to use creativity, environmental context, detail, and composition when constructing the requested scene.
Additional content is not automatically rewarded. A model can add substantially more geometry and still lose if those additions hurt prompt adherence, composition, or the quality of the requested object.
Likewise, a simpler build can be better if it represents the prompt more effectively.
MineBench does not use block count, scene size, or raw complexity as a scoring metric.
MineBench uses blind head-to-head voting.
Two builds generated from the same prompt are shown without revealing which model produced either one. Pairwise preferences are then aggregated into the leaderboard rating.
There is no canonical correct voxel output for a castle, spacecraft, locomotive, or most other MineBench prompts. Using a single reference build would primarily measure similarity to that particular interpretation rather than the quality of the generated 3D structure.
This is also why properties such as block count, symmetry, or geometric complexity are not used as automatic measures of quality.
Potentially, but MineBench does not have a fixed answer key that a model can simply memorize.
There is no canonical correct output for a castle, fighter jet, city, or any other prompt. Simply knowing that a prompt asks for a castle does not give a model the ability to actually construct a good castle in 3D. The model still has to reason about its structure, proportions, geometry, spatial relationships, and composition, then generate the JavaScript required to construct the build.
More importantly, the raw JavaScript generated by models for official evaluations is not released as an answer set that future models could train on or memorize. The normalized build data must be delivered to the site for rendering and export, but it is not labeled as a canonical correct response.
Public exposure can still matter. Models may become more familiar with MineBench's prompts, task format, voxel API, or the types of outputs that tend to perform well. That is a real form of evaluation contamination.
It is fundamentally different, however, from a benchmark where the model can memorize a fixed question and its correct answer. Knowing the prompt does not solve the spatial reasoning task, and the actual official responses are not published as an answer key.
New prompts can also be introduced if existing ones become overly familiar or stop distinguishing models effectively.
Models compared on the same leaderboard need to be generated under the same conditions.
Grid size matters because it determines how much 3D space is available. A larger grid gives models more room for larger structures, scenery, separation between objects, and fine detail. Builds generated on different grid sizes therefore cannot be fairly mixed into the same leaderboard.
Different grid sizes could instead be separate leaderboard settings. A smaller-grid setting could emphasize spatial efficiency and abstraction, while a larger-grid setting could test how models use additional space.
The same applies to fixed block limits. A fixed-block setting would test how effectively models use a constrained voxel budget, while the standard setting tests what models produce without that additional restriction.
Each of these settings would require a separate set of generations across the models being compared.
The primary limitation is API cost.
A new prompt needs to be generated across the model set. A new grid size, block limit, or other configuration effectively creates another evaluation suite and requires another set of generations across the relevant prompts and models.
There are many additional settings and harder prompts worth testing. The bottleneck is running them consistently across enough models for the results to remain useful.
Harder prompts are generally the highest priority because they improve separation between stronger models without requiring an entirely separate leaderboard configuration.
Models do not see renders of their builds and iteratively revise them based on visual feedback.
They receive the prompt and voxel-building interface, generate the JavaScript used to construct the scene, and that program produces the final voxel output.
Technical validation is separate from visual iteration. Handling output that cannot execute or render correctly is not equivalent to showing the model its completed scene and allowing it to improve the geometry.
A workflow where models repeatedly inspect renders and modify their builds would test an agentic 3D-editing system and would need to be evaluated separately.
Yes.
The MineBench Sandbox allows direct comparisons of up to four models at once for the same prompt and evaluation setting.
If the models have already been evaluated under the same conditions, their builds can be viewed side by side without requiring a new evaluation run or waiting for a dedicated comparison post.
Yes.
The Sandbox Import workspace allows MineBench instructions to be used with models that are not directly integrated. The model can generate the expected output externally, and the result can then be loaded into MineBench and rendered.
Those generations are useful for experimentation but should not be treated as official leaderboard results unless they were produced under the same controlled settings.
The usual reasons are API availability, integration work, or API cost.
New models are released much faster than the full MineBench evaluation can reasonably be rerun, so models have to be prioritized.
API cost is usually the largest constraint.
Yes.
MineBench supports WorldEdit .schem for Minecraft-compatible workflows, GLB for general 3D workflows, and STL for applications such as 3D printing.
Older comments stating that export support was still planned are outdated.
No.
MineBench uses its own constrained voxel-building interface so that models are evaluated with the same tools.
Giving models access to a full coding agent, shell, Blender environment, computer-use loop, or substantially different MCP toolset would introduce additional capabilities and make it difficult to isolate the model's spatial reasoning.
Those systems could be evaluated separately, but their results would not be directly comparable with standard MineBench generations.
MineBench is open source. Useful contributions include harder prompts, model integrations, renderer improvements, methodology improvements, UI changes, and bug fixes.
Financial support primarily goes toward API inference, which is the largest constraint on expanding the set of models, prompts, and evaluation settings.
