MI355X GPU vs B300

I've been following the developments in AI hardware, and one thing that's caught my attention is the MI355X GPU. It's offering surprisingly competitive performance to the B300 when it comes to running Kimi K3, and that's largely due to its massive 288GB of VRAM. What's really interesting, though, is that the MI355X is managing to keep up with the B300, which is a much more established player in the field.

Running Kimi K3 at around 952 tok/s/node, AMD is continuing to make a strong case for itself as the winner in terms of performance per dollar. Over the past few months, we've seen a huge explosion in the capabilities of open source models, and it's clear that the hardware is starting to catch up. One of the key factors in getting Kimi K3 to its current throughput number was the use of speculative decode, which required some additional work to get right.

I'm still trying to wrap my head around what this means for the future of AI hardware, though. On the one hand, it's impressive to see AMD pushing the boundaries of what's possible with its GPUs. On the other hand, I'm not convinced that the MI355X is going to be the game-changer that some people are making it out to be. Can the MI355X really hold its own against the more established players in the field, or is this just a flash in the pan?

Introduction to Kimi K3 and MI355X

The Kimi K3 and MI355X are two notable players in the GPU landscape, with the MI355X standing out for its substantial 288GB of VRAM. To put that into perspective, this amount of VRAM is typically associated with high-end NVIDIA GPUs, but AMD's MI355X is a non-NVIDIA GPU that matches this spec. The importance of VRAM in performance cannot be overstated, as it directly impacts a GPU's ability to handle complex workloads and large datasets.

In terms of serving throughput versus interactivity, the Kimi K3 is an interesting case. When compared to the MI355X, B300, and B200 (in a 2-node TP16 configuration), the Kimi K3's numbers are noteworthy. For instance, the B200's numbers are somewhat deflated due to the cross-node all-reduce on the decode critical path, which utilizes RoCE v2 at approximately 195 Gb/s. This factor affects the B200's performance, making its numbers not entirely comparable to the others.

Benchmark results show the Kimi K3 achieving 952 tokens per second (tok/s) per node, while the single-stream performance is around 118 tok/s. In a 16-GPU setup, the Kimi K3 reaches 498 tok/s, and 249 tok/s per node. These benchmarks demonstrate the Kimi K3's capabilities, but it's also important to consider the context of these numbers. As one expert notes, "decode tok/s is fool's gold — decode is over-glorified, while time-to-first-token, number users feel" is a more meaningful metric. This highlights the complexity of evaluating GPU performance, where different metrics can tell different stories.

The discussion around the Kimi K3 and MI355X also raises questions about the CUDA moat, with some asking, "Is the CUDA moat dead?" This refers to NVIDIA's historical dominance in the GPU market, largely due to its CUDA platform. However, with advancements in non-NVIDIA GPUs like the MI355X, the landscape is evolving. To demonstrate the configuration of such a system, consider the following setup command:

sudo apt-get update && sudo apt-get install -y nvidia-driver

This command updates the package list and installs the necessary NVIDIA driver for the system. Note that this is a simplified example and actual setup may vary depending on the specific use case and environment.

In conclusion, the Kimi K3 and MI355X represent significant developments in the GPU space, with their substantial VRAM and notable benchmark performance. As the market continues to evolve, it will be interesting to see how these and other GPUs shape the future of computing. To get a better understanding of the system's configuration, you can use a configuration file like this:

config = {
    "gpu": "MI355X",
    "vram": 288,
    "benchmark": {
        "tok_per_sec": 952,
        "single_stream": 118
    }
}

Performance Comparison

The Kimi K3's performance on different GPUs is a mixed bag. On the MI355X, it achieves a serving throughput of 952 tokens per second per node, which is impressive. However, when it comes to interactivity, the numbers are less stellar, with a single stream throughput of 118 tokens per second. The B300 fares slightly better in this regard, with a throughput of 249 tokens per second per node. The B200, on the other hand, is hindered by its cross-node all-reduce on the decode critical path, which results in a lower throughput of 25 tokens per second.

One thing that's worth noting is that the MI355X has a whopping 288GB of VRAM, which is comparable to some NVIDIA GPUs. This is particularly interesting because it suggests that non-NVIDIA GPUs can hold their own in terms of memory capacity. The B200's numbers, for example, are somewhat deflated by the fact that it has to pay a performance penalty for its cross-node communication. To give you a better idea, here's a simple benchmarking script in Python:

import time

def benchmark_kimi_k3(gpu):
    # simulate token processing
    tokens_per_second = 0
    start_time = time.time()
    for _ in range(1000):
        # process token
        pass
    end_time = time.time()
    tokens_per_second = 1000 / (end_time - start_time)
    return tokens_per_second

print(benchmark_kimi_k3("MI355X"))  # example usage

This script doesn't actually do anything meaningful, but it gives you an idea of how you might structure a benchmarking test. In reality, the Kimi K3's performance is more complex and depends on a variety of factors, including the specific use case and system configuration.

As someone who's been following the development of AI and GPU technology, I have to say that I'm intrigued by the idea that non-NVIDIA GPUs can compete with their NVIDIA counterparts. The question on everyone's mind, of course, is whether the CUDA moat is dead. While it's still too early to say for sure, the fact that GPUs like the MI355X can hold their own in terms of performance is certainly a promising sign. However, it's also worth noting that decode throughput can be a bit of a red herring - as one expert noted, "decode tok/s is fool's gold - decode is over-glorified, while time-to-first-token, number users feel" is often a more important metric.

Understanding the Results

The performance of these systems is influenced by several factors, including cross-node all-reduce on the decode critical path and RoCE v2 speeds. For example, the B200's numbers are somewhat deflated by the fact that it pays a cross-node all-reduce on the decode critical path, with RoCE v2 speeds of around 195 Gb/s. This is an important consideration, as it can significantly impact the system's overall performance.

To put this into perspective, the B200 achieves 249 tok/s/node, while the MI355X reaches 952 tok/s/node. However, it's worth noting that the MI355X has a significant advantage in terms of VRAM, with 288GB available. This extra memory can be a major factor in certain workloads, particularly those that involve large amounts of data.

Here's an example of how you might configure a system to take advantage of this extra memory:

import torch

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

tensor = torch.randn(1000, 1000, device=device)

This code sets up a device with 288GB of VRAM and allocates a large tensor to demonstrate the available memory. As you can see, the extra memory provided by the MI355X can be a significant advantage in certain workloads.

It's also worth considering the trade-offs between different systems. For example, the Kimi K3 serves throughput vs interactivity, per node, with the MI355X, B300, and B200 (2-node TP16) all having different strengths and weaknesses. The B200, for instance, achieves 118 tok/s single stream, while the MI355X reaches 498 tok/s (16-GPU). These differences can have a significant impact on the overall performance of the system, and should be carefully considered when choosing a system for a particular workload.

One expert notes that "decode tok/s is fool's gold — decode is over-glorified, while time-to-first-token, number users feel" is a more important metric. This highlights the complexity of evaluating the performance of these systems, and the need to consider multiple factors when making a decision. Ultimately, the choice of system will depend on the specific needs of the user, and a careful consideration of the trade-offs between different systems.

Conclusion

The MI355X matching B300 in Kimi K3 throughput isn’t just a footnote in AMD’s GPU story—it’s the point where the numbers stop being theoretical and start feeling inevitable. We’ve seen open-source models explode in capability, but the real shift isn’t in the models themselves; it’s in the hardware that can actually serve them without bleeding money or sanity. AMD’s 288GB VRAM isn’t a gimmick when you’re running models that expect to keep whole parameter sets resident, and the fact that they’re doing it at a price point that makes NVIDIA’s B200 look like a fancy paperweight—well, that’s the story.

What’s left to figure out is whether AMD’s lead here is enough to pry loose the last holdouts still chasing NVIDIA’s coattails. The B200’s cross-node penalty is a real bottleneck, not some abstract performance tax, and if MI355X owners can dodge that bullet consistently, the buying decisions might stop being about brand loyalty. Then again, maybe the next model update from either side will flip the script again. For now, the only certainty is that if you’re picking a GPU for serving LLMs today, you’d better have a damn good reason to look past AMD.