AIMLLLMMarch 29, 2026 · 5 min read

Competing in OpenAI Parameter Golf: My Journey into Model Craft

Recently, I decided to take on a unique and exciting challenge: the OpenAI Parameter Golf Challenge. It's a competition to train the best language model possible that fits within a strict 16MB artifact size and can be trained in under 10 minutes on 8x H100 GPUs. The models are evaluated by how well they compress the FineWeb validation set, using a tokenizer-agnostic bits-per-byte metric.

Why Parameter Golf?

What drew me to this challenge was its focus on creativity and efficiency. Unlike many machine learning competitions that reward brute force and massive compute, Parameter Golf is all about doing more with less. The constraints force you to think outside the box, whether through clever model architectures, aggressive parameter sharing, quantization tricks, or novel tokenization methods.

Getting Started

The OpenAI team made it easy to get started, providing a public repo and clear instructions for both local and cloud-based training. I began by experimenting on my local machine, running small-scale tests to get a feel for the training scripts and the dataset. The documentation was refreshingly thorough.

Once I felt comfortable, I scaled up to a remote GPU environment using Runpod, which is recommended for its proximity to the challenge's official evaluation setup.This allowed me to iterate faster and test ideas at scale.

The Creative Process

Parameter Golf is not just about tuning hyperparameters; it's about rethinking the fundamentals. I found myself exploring ideas in several directions:

ideas.ts

const quantization = "1-bit, ternary, low-bit schemes"

const parameterTying = "sharing weights across layers"

const recurrence = "depth recurrence, universal transformers"

const tokenization = "custom tokenization strategies"

const training = "compression-aware training loops"

Some ideas worked, others did not, but the process was always engaging. The challenge actively encourages weird and creative approaches. I loved seeing what others in the community were trying as well.

The Community and Support

One of the best parts of this challenge is the community. OpenAI has set up forums, compute grants, and a transparent leaderboard. There is a real sense of collaboration, even as we compete. I have learned a ton from reading others' submissions and discussing ideas in the forums. It feels less like a competition and more like a shared research effort.

Lessons Learned

If there is one thing I have taken away from Parameter Golf, it's that breakthroughs often come from constraints. When you can't just throw more parameters or compute at a problem, you're forced to innovate. I've gained a deeper appreciation for efficient model design and the art of making every byte count.

Engineering custom training loops with gradient accumulation, mixed-precision arithmetic, and memory-efficient data loading, all while staying under 16MB, is a completely different challenge from standard model training. It's made me a better engineer.

Looking Ahead

I'm still iterating on my submission, aiming to push the limits of what's possible within the 16MB and 10-minute constraints. Whether or not I top the leaderboard, I'm proud of what I've learned and built. If you're interested in machine learning, efficiency, or just love a good challenge, I highly recommend giving Parameter Golf a try.

Stay tuned for updates on my progress. If you want to talk about quantization, compression, or efficient architectures, contact me here

𝒷𝑒𝓇𝑒𝓀𝑒𝓉 𝓁𝑒𝓂𝓂𝒶