Does Your Model Run On AWS Trainium? Look It Up

TL;DR

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Hugging Face has published a public lookup Space and dataset showing whether selected Transformers models run on AWS Trainium. Results include test outcomes, device and software details, and scripts, but a pass does not establish performance or support for longer generations.

Hugging Face has published a model-support lookup for AWS Trainium, allowing users to check whether selected Transformers models run on the accelerator and view the test script used. The tool addresses a practical compatibility question for developers considering Trainium, but its PASS results do not measure throughput and cover only a limited selection of models and tasks.

The lookup is a Hugging Face Space built around tests using TorchNeuron, the native PyTorch backend for AWS Trainium provided by the torch-neuronx package. Hugging Face says Transformers runs its own modeling code through that backend. The tool reports whether a tested model runs unmodified, along with the number of Trainium cores used, wall-clock runtime including compilation, and peak device memory.

Users can browse a selected set of popular Hub models or search the Hub’s Trending and Downloads views by task. Those views follow the Hub rankings whether or not a model has been tested. Where a model is omitted intentionally, the Space gives a reason; the report says this is usually because it was not prioritized, though some models may be incompatible. Users can also select which Neuron compiler and Transformers versions to inspect.

For each listed model, the Space provides the tested software stack, installation packages, and a run.py script. Hugging Face says the script loads the model, shards it when needed, then generates using a static cache and torch.compile(backend=”neuron”), a setup covered in AWS foundation model training and inference. On a compatible Trainium2 instance, the user can run the script after installing the listed packages. A first run compiles the model graphs; the report also describes a command to retrieve cached compiled graphs from a public bucket.

What the Lookup Tells Trainium Users

The tool gives developers a way to distinguish a model that has actually been tested on Trainium from one that is merely available on the Hugging Face Hub. That can help teams narrow their options before setting up an AWS instance, and the supplied script makes it possible to try the same basic run rather than relying only on a compatibility label. Reporting the core count, compile-inclusive wall time, and peak memory also offers useful setup information.

Those figures should not be read as a general performance ranking. Hugging Face explicitly says a PASS does not indicate throughput, and the tests generate only 32 tokens. Runtime includes compilation, which can make it different from the time for later runs using already compiled graphs. The tool is therefore a compatibility and reproducibility resource, not a benchmark establishing how quickly or cheaply a model will serve real workloads.

The public dataset also makes the results easier to inspect outside the Space. It contains a record for each model and tested version combination, including failures and their errors, according to Hugging Face. That matters for developers comparing software stacks: model support can depend on the particular compiler and library versions, so a result should be read alongside its version information.

How Hugging Face Tests Models

The lookup follows Hugging Face’s earlier introduction of TorchNeuron as a PyTorch backend for Trainium. The report says the tests run on Trainium2 (trn2) instances and shard a model over the smallest number of Trainium cores that can hold its weights. Most models use tensor parallelism; mixture-of-experts models without a tensor-parallel plan use expert parallelism alone.

Each test generates 32 tokens. For a model running on one core, the generated tokens are compared with output from a CPU run of the same checkpoint. For sharded runs, a small judge model checks whether the text is fluent English. The report says the current tests include only models that fit within four Trainium2 devices, or up to 16 cores.

At the time described in the report, the selection covers Transformers models for text generation and image-text-to-text. Hugging Face says it plans to expand testing to Diffusers, sentence-transformers, and other Transformers tasks; those expansions are plans, not features the report says are already available.

“A PASS says nothing about throughput, or about longer generations.”

— Hugging Face

Limits of the Current Test Results

The report does not state that every model on the Hub has been tested. The selected list is limited, and the Trending and Downloads views include models that may have no test result. For some omitted models, the stated reason is lack of prioritization rather than technical incompatibility, so absence from the tested list does not by itself prove a model cannot run.

A PASS is also narrow: it reflects the report’s test setup, selected software versions, hardware limits, and 32-token generation. It does not establish performance for longer outputs, production workloads, different model configurations, or other Trainium devices. The report provides no general throughput comparison or cost analysis. It also gives no schedule for the planned expansion to additional model families and tasks.

Because the results are tied to specific Neuron compiler and library versions, users should check the version row and script before applying a result to their own environment. The report does not claim that every version combination has been tested or that results remain unchanged as software evolves.

Planned Coverage and User Checks

Hugging Face says it plans to extend the testing to Diffusers, sentence-transformers, and additional Transformers tasks, but it has not provided a release schedule in the report. Until coverage expands, users can search the current Space, inspect the tested compiler and library versions, and use the supplied script to reproduce a listed run on a compatible Trainium2 setup.

For teams evaluating a model for deployment, the next step is to test their own workload beyond the lookup’s 32-token check, including the output length and performance requirements they expect in practice. The public dataset can be queried directly for model outcomes across recorded version pairs, while the Space supplies the corresponding run details. Whether a result applies to a different stack or workload remains a matter for further testing.

Key Questions

What does the Trainium model lookup show?

It reports whether selected Hugging Face Transformers models ran in the stated Trainium test setup, alongside core count, runtime including compilation, peak device memory, software versions, and a script for reproducing the run.

Does PASS mean a model will be fast in production?

No. Hugging Face says PASS does not measure throughput. The test generates 32 tokens, and its runtime includes compilation, so it is not a general production performance result.

Does a missing model mean it is incompatible with Trainium?

Not necessarily. The report says some models were left out because they were not prioritized, while others may be incompatible. The Hub ranking views can show models whether or not they have been tested.

Which models and tasks are covered?

At the time of the report, the tested selection covers Transformers models for text generation and image-text-to-text, and only models fitting within four Trainium2 devices, up to 16 cores.

Can developers reproduce a listed test?

Yes. The Space provides the tested stack, packages, and run.py script. Hugging Face says users can run it on a Trainium2 instance after installing the listed packages; the first run compiles the model graphs.

Source: rss