HumanEval on a MacBook — 81.7% pass@1, Wi-Fi off

The M5 Max MacBook Pro with 128 GB of unified memory is the first laptop that can hold a frontier-class coding agent entirely in RAM. No GPU rack. No cloud. No subscription.

I just ran HumanEval on it. Wi-Fi off the entire run.

- 81.7% pass@1 on the full 164-problem benchmark - Qwen 3 Coder 30B-A3B-Instruct (8-bit MLX) - 14 minutes wall-clock, $0/month after the model download

YouTube walkthrough (three real problems, code streaming live, tests going green): https://www.youtube.com/watch?v=muq7VdgxqRk

## Why this number matters

The Qwen team didn't publish HumanEval scores for any Qwen3-Coder variant — they consider the benchmark saturated and went straight to agentic ones (SWE-bench Verified, BFCL, Aider-Polyglot). For the 30B variant — the one that actually fits on a laptop — there were no published HumanEval/MBPP numbers. Until this run.

I also ran MBPP (sanitized): 83.3% pass@1 on a 168-problem sample. Pass rate stable since n=120; full 427-run was impractical because a few outlier tasks induce very long model responses (10+ minutes each).

## Methodology

| Setting | Value | |---|---| | Benchmark | HumanEval — 164 Python tasks (full) | | Metric | pass@1 (first attempt only) | | Temperature | 0.0 — deterministic | | Sampling | single sample per problem, no best-of-N | | Execution | Python subprocess, 10s timeout | | Hardware | M5 Max MacBook Pro · 128 GB unified memory | | Model | Qwen3-Coder-30B-A3B-Instruct-MLX-8bit | | Network | Wi-Fi OFF the entire run | | Wall clock | 14 minutes |

## For context — Qwen3-Coder 480B's official agentic benchmarks

The Qwen team's published numbers for the 480B flagship sibling (the bigger sibling of the 30B running on this MacBook):

| Benchmark | Qwen3-Coder 480B | Claude Sonnet 4 | GPT-4.1 | |---|---|---|---| | SWE-bench Verified (500-turn) | 69.6 | 70.4 | — | | Terminal-Bench | 37.5 | 35.5 | 25.3 | | BFCL-v3 | 68.7 | 73.3 | 62.9 | | Aider-Polyglot | 61.8 | 56.4 | 52.4 |

Source: Qwen team's official blog.

## Why the offline part matters

If a tool needs the internet, three things are true:

1. Someone else can read what you sent. 2. Someone else can charge you for it. 3. Someone else can take it away.

If the same tool runs locally, none of those are true. That's a different category of software — and for law firms, medical practices, and accountants handling client material, it's the only legal one.

## Reproduce it yourself

- Open-source launchers: github.com/nicedreamzapp/claude-code-local - HumanEval dataset: github.com/openai/human-eval - Hardware: any M-series MacBook with ≥32 GB RAM (128 GB Max preferred for full 8-bit weights) - Total monthly cost: $0 after the model download

For law firms, medical practices, and accountants who want help getting this stack running on their own hardware — that's what AirGap is. 14-day pilot, fixed scope, the data never leaves your machines.

— matt

Originally published at Marijuana Union. For premium vaporizers visit iNeedHemp, wholesale at Nice Dreamz, and seeds at Tribe Seed Bank. Explore the 3D cannabis marketplace at The Farmstand.

Comments

Post a Comment

Popular posts from this blog

Exploring the Divine Tribe V5 with Hubble Hydratube: A Comprehensive Review

Image
In the ever-evolving world of vaporizers, the Divine Tribe V5 paired with the Hubble Hydratube stands out as a top choice for vaping enthusiasts. Known for its innovation and superior performance, this combination offers a unique and enhanced vaping experience. Here’s a closer look at what makes the V5 with Hubble Hydratube a must-have for both novice and seasoned vapers. Unpacking the Divine Tribe V5 The Divine Tribe V5 is a state-of-the-art vaporizer designed for concentrate use. It features: Advanced Heating Technology : The V5 utilizes a ceramic heating rod which ensures even and efficient vaporization of concentrates, providing pure and flavorful hits. Temperature Control : Precise temperature settings allow users to customize their vaping experience, ensuring optimal performance for different types of concentrates. Durable Build : Constructed from high-quality materials, the V5 is designed to withstand regular use while maintaining peak performance.           The Hubble ...
Read more

How the Richest Man in America Killed Hemp to Protect His Investments

Originally published on Marijuana Union The Three Men Who Killed Hemp The story of how hemp became illegal in America isn’t a story about public health or safety. It’s a story about three incredibly wealthy men who stood to lose fortunes if hemp remained legal — and had the political power to make sure it didn’t. Their names: Andrew Mellon, William Randolph Hearst, and the DuPont family . Together, they orchestrated one of the most successful corporate lobbying campaigns in American history, and we’re still living with the consequences nearly 90 years later. Andrew Mellon: The Banker Who Bought a Bureau Andrew W. Mellon was the Secretary of the United States Treasury from 1921 to 1932 — and simultaneously one of the richest men in America. His banking empire, Mellon Bank, was one of the largest financial institutions in the country. More importantly for our story, Mellon was a major investor in DuPont , the chemical giant that was about to make a massive b...
Read more

Why Hemp Clothing Is More Than Just a Trend: Exploring the Benefits of Hemp Hoodies

Image
In recent years, hemp clothing has gone from niche to mainstream—and for good reason. As more people look for sustainable, eco-friendly alternatives to fast fashion, hemp is stepping into the spotlight. Among the standout products in this growing movement are hemp fleece hoodies. These cozy, durable garments aren’t just a fashion statement—they represent a shift toward sustainability, comfort, and conscious consumption. What Makes Hemp Clothing Stand Out? Hemp is a fiber with a long history. From ropes to paper to clothing, it has been used for centuries, yet only recently has it been appreciated for its incredible benefits in the fashion industry. Unlike cotton, hemp requires minimal water to grow and doesn’t rely on pesticides or harmful chemicals, making it far more environmentally friendly. In fact, hemp regenerates the soil it’s grown in, which helps maintain soil health and promote sustainable agriculture. In today’s fashion world, hemp is a star. Its fibers are naturally dur...
Read more