City of Glass was always going to be a mind-twister of a show. Paul Auster's 1985 novella is a story inside a story inside a story; a detective thriller concealing multiple layers of fiction. One writer - the unnamed narrator - tells the story of another, who goes by numerous monikers: Daniel Quinn, William Wilson, Paul Auster. Reality takes on different faces. Identities slip and slide.

A wrong number lures Quinn, a reclusive author of mystery novels, into a mystery of his own. Mistaken for private detective Paul Auster, he's hired to protect Peter Stillman from his father, a disturbed theologian who locked his son away for nine years in an 'experiment' to discover God's language. Here begins a strange story that only gets stranger, as all of Quinn's certainties begin to flake away.

The trouble is, this is not a story that invites theatricality. Auster's book is literature about literature. It's stuffed with literary allusions - Poe, Milton, Don Quixote - and fascinated by the slipperiness of words. How to transport it to the stage is a question that adaptors 59 Productions (the company who have brought video and projection magic to everything from Katie Mitchell productions to the London 2012 Olympic opening ceremony) never fully answer.

That's not to deny the show's pleasures. If you've read the novella, then watching this adaptation is a satisfying cerebral exercise in rediscovery. Writer Duncan Macmillan's version is restrained and respectful, carefully underlining Auster's mind-bending sentences. The book's narration is maintained, delivered via voiceover, and chunks of prose are copied pretty much wholesale from page to stage.

The main concession to theatricality comes in the form of 59 Productions' technically accomplished video projections, which take their lead from Paul Karasik and David Mazzucchelli's graphic novel version of Auster's story. The apartment interior that Jenny Melville has constructed on stage is a stylish blank canvas for the landscapes painted by Lysander Ashton's videos, breathlessly whisking us from location to location. Astonishingly, these projections create the illusion of texture as well as image. A neon sign leaps out from a wall. Bricks look rough to the touch.

The design is best when doing this simple but brilliant scene-setting and atmospheric work. Subtle shifts in the backdrop melt the edges of the fiction, as once substantial walls suddenly feel like things of make-believe. But in the more ambitious and visually dazzling sequences, the video becomes distancing rather than evocative, lifting us out of the labyrinthine world of Quinn's peculiar investigation.

Another deft touch in director Leo Warner's production is the casting of two different actors, Mark Edel-Hunt and Chris New, in the central role of Quinn. The shifting identity that the novella so brilliantly conveys finds a visual equivalent in these two similar but different figures, uncannily switching in and out of the action. "Who are you?" Quinn asks of himself at one point, leaving the same question reverberating around the stage.

Still, though, something is lacking. In his adaptation of 1984 with Robert Icke, Macmillan invented a brilliant theatrical solution for the problem of George Orwell's appendix, slyly turning the whole narrative on its head. Here, what's missing is a similar revelation. Ultimately, City of Glass feels like a staged exploration of a book rather than a transformation from one medium into another. It's always theatre about literature, rather than theatre in its own right.

City of Glass will run at HOME, Manchester until 18 March before running at the Lyric Hammersmith from 26 April to 13 May, with previews form 20 April.