Richard Eyre loathed the Olivier when he ran the NT. It gave him insomnia, panic attacks and hemorrhoids. In his diaries, the Olivier comes across as a nightmarish antagonist – a "giant mouth" demanding a diet of big, meaty hits; the sinkhole that could swallow the organisation whole. It's so hostile a theatre that Eyre likens it to "a watering can that doesn't hold water."

I understand that now – and I'll bet Rufus Norris does too. The Olivier is facing a double-dud summer. Salomé was infuriating – pretentious, overblown and indulgent. Common is merely skewiff – never boring, always baffling. The ambition and imagination in DC Moore's epic is laudable, but it just doesn't cohere. Were it not for a fine programme essay on the subject, I'm not entirely sure I'd have clocked that it concerned enclosure – that is, the gradual privatisation of what was once common land, collectively owned and collectively farmed.

Set somewhere in rural England in 1809, Common plays out against a backdrop of the industrial revolution. The sense is of the end of an era; an unknown new order on the way in. Anne-Marie Duff plays a woman apparently back from the dead, Mary, but in reality, returned from London. Somewhere between a scam-artist and a soothsayer, newly rich thanks to prostitution, she comes back to tear her ex-lover Laura (Cush Jumbo) away from her abusive brother King (John Dagleish) and his incestuous urges.

But it's the background that's key: Tim McMullan's fruity landowner pushes once self-sufficient locals into low paid work on his estate, undercutting them with the cheap labour of Irish immigrants. His private paramilitary force, led by Trevor Fox's snarling Heron, lay down their own law. Beyond that, though, the plot's muddy as hell. Turning points bubble up out of the blue.

Moore writes in a mangled, mongrel tongue – an ersatz attempt at a sort of olde countryspeak. It comes in broke-backed sentences of jumbled-up words and burrish chatter, as if grammar were a London thing. The speech never settles into anything comprehensible, hitting the ear and ratting emptily around. It sounds great, but it makes so little sense that Moore might as well be writing in tongues.

If the plot falls out, Jeremy Herrin's sumptuous production is undeniably atmospheric – and that's where you glean what meaning you can. Richard Hudson's dried mud and bare stone stage conjures a barren land under a dirty sky – a nation stripped of its natural resources. Muscular silhouettes scythe the air to no avail, or carry saplings home for meagre firewood. Crowds suggest overpopulation, without enough to go round. Dead animals recur, as do open graves, and bloodied men batter each other senseless in the dry earth.

Its starkest image is of collective kills, as if a community that once farmed together, now hunts together. They form a mass of masked bodies; individuals hiding behind animal skulls and vegetal matter as they set on individual enemies and tear them limb from limb. Innards are ripped out, skulls caved in with rocks. It would be deeply unsettling were the rest not so confounding.

As for the cast, it's high praise to say they play this as if it made total sense. It's easy to sniff at the idea that acting takes courage, but to step out onstage, in front of 1,200 people, in a play this full of potholes is ballsy beyond measure. Duff deserves the George Cross for the brio she brings to Mary, and Jumbo plays Laura with a committed sincerity. The Olivier, however, is still hungry.

Common runs at the National Theatre until 5 August.