'Aristocrats? The same ugly bodies and physical uncleanliness, the same toothless old age and disgusting death, as with market women."
This observation comes from a notebook that Anton Chekhov kept during the last 12 years of his life, between 1892 and 1904. In it he jotted down snatches of conversation he had overheard, anecdotes, aphorisms, interesting names and embryonic ideas for short stories. This entry about aristocrats and market women belongs to the last category. The more one has read of Chekhov the more one can envisage the short story that might have grown from this bleak comparison. The point is well made and as true today as it was in 19th-century Russia - death is the great leveller - but more interestingly, these 20 words can lead us towards an initial way of understanding the short story as opposed to its larger sibling, the novel. I would argue that you could write a short story inspired by Chekhov's words but they wouldn't be sufficient for a novel.
William Faulkner regarded a short story as harder to write than a novel. Some writers rarely tackle it, or else, in a full career, write just half a dozen. Others seem perfectly at home with the form and then let it drop. And then there are those for whom the novel appears the challenge.
Yet many of the greatest short-story writers have steered clear of the long form, by and large: Chekhov, JL Borges, Katherine Mansfield, VS Pritchett, Frank O'Connor. My own case is perhaps typical: I have written eight novels but I cannot stop writing short stories - something about the short form lures me back again and again.
What draws a writer to the short story? It's important to remember that the story as we know it is a comparatively recent phenomenon. The arrival of mass-market magazine publication and a new generation of literate middle-class readers in the mid- to late-19th century saw a boom in the short story in the US and Europe that lasted maybe 100 years. Many writers were initially drawn to the form as a way of making money, particularly in America: Nathaniel Hawthorne, Herman Melville and Edgar Allen Poe all subsidised their less well-remunerated novel-writing careers by writing stories. In the 1920s, F Scott Fitzgerald was paid $4,000 for a story by the Saturday Evening Post (a vast sum today - multiply by 10 to get some idea of a comparison). Even John Updike, in the 1950s, reckoned he could support his wife and young family by the sale of five or six stories a year to the New Yorker. Times have changed. While magazines such as the New Yorker, Esquire and Playboy pay handsomely, more than any British equivalent, no one today could replicate Updike's achievement.
The popularity of the short story - indeed its very availability - has, more so than in the case of the novel, always been somewhat at the mercy of commercial considerations. When I published my first collection of stories, On the Yankee Station, in 1981, many British publishers routinely brought out short-story collections. Not any more. Moreover, there was a small but stable marketplace where a story could be sold. A short-story writer could place his or her work in all manner of outlets. The stories in my first collection, for example, had been published in Punch, Company, London Magazine, the Literary Review and Mayfair, and had been broadcast on the BBC. As a young writer I started writing short stories because at the time it seemed logical: this was my best chance of getting published. All this talk of money and strategy masks the tenacious appeal of the form. In the end writers write short stories because a different set of mental gears is engaged. Melville wrote short stories as he laboured with Moby-Dick, saying, "My only desire for their 'success' (as it is called) springs from my pocket and not my heart." And yet, in the process, he wrote works of short fiction ("Bartleby" and "Benito Cereno" among others) that have become classics.
Something occurs in the writing - and reading - of a short story that is on another level from the writing and reading of a novel. The basic issue, it seems to me, is one of compression versus expansion. To go back to the remark I made apropos Chekhov's little memento mori about aristocrats and market women: we see that the ideas, the inspiration, that will drive a novel, however succinctly expressed, have to be capable of endless augmentation and elaboration. The essence of almost every short story, by contrast, is one of distillation, of reduction. It's not a simple question of length, either: there are 20-page short stories that are far more charged and gravid with meaning than 400-page novels. We are talking about a different category of prose fiction altogether.
A common analogy is to see the novel as an orchestra and the short story as a string quartet. But the analogy strikes me as false because it is all about size, and this leads us in the wrong direction. The music produced by two violins, a viola and a cello cannot ever sound anything like the music produced by dozens of instruments, but a paragraph or a page from a short story is indistinguishable from a paragraph or a page from a novel. The short story draws on exactly the same resources as does the novel - language, plot, character and style. None of the literary tools that novelists require to write their novels is denied the short-story writer. A more pertinent comparison - to try to pin down the essence of the two forms - is poetry: to compare the epic with the lyric. Let us say that the short story is prose fiction's lyric poem, contrasted with the novel as its epic.
There are many definitions of the short story. Pritchett defined it as "something glimpsed from the corner of an eye, in passing". Updike has said: "More closely than my novels . . . these efforts of a few thousand words each hold my life's incidents, predicaments, crises, joys."
Angus Wilson observed that, "Short stories and plays go together in my mind. You take a point in time and develop it from there; there is no room for development backwards." All things to all writers, then: the quotidian epiphanic moment, the submerged autobiography, a question of structure and direction: I could cite other definitions - some contradictory, some far-fetched - but all, in their own way, possessing some cogency. If the house of fiction has many windows so too, it seems, does the house of short fiction.
I have published three collections of short stories over two decades, a total of 38 stories. Perhaps there are another four or five uncollected ones out there - juvenilia in university magazines, the odd one-off commission for an anniversary. In any event, what repeatedly draws me to the short story is its variety - the enticing possibility of adopting different voices, structures, styles and effects.
Consequently, I decided it might be worth trying to categorise the short story in a bit more detail, to try to classify its multifarious forms. Looking at collections by other writers, I gradually came to the conclusion that there are seven types of short story, and that within these seven categories almost every kind of short story can be accounted for. Some of them will overlap, or one category will borrow from a seemingly unrelated type, but these denominations seem, by and large, to subsume all the species of the genus. In this diversity we may begin to see what short stories have in common.
1 The event-plot story This term was coined by the English writer William Gerhardie in 1924 in a short, fascinating book he wrote on Chekhov. Gerhardie uses this appellation to distinguish Chekhov's stories from everything that preceded him. Up until Chekhov, all short stories, virtually without exception, were event-plot ones. In these stories the skeleton of plot is all important, the narrative is shaped, classically, to have a beginning, middle and end. The revolution that Chekhov set in train - and which reverberates still today - was not to abandon plot, but to make the plot of his stories like the plot of our lives: random, mysterious, run-of-the-mill, abrupt, chaotic, fiercely cruel, meaningless. The stereotype of the event-plot story is the "twist-in-the-tail" famously developed by O Henry but also used widely in genre stories - ghost stories (WW Jacobs, for example) and the detective story (Conan Doyle). I would say that today its contrivances make it look very dated, though Roald Dahl made something of a mark with a macabre variation on the theme, and it is also a staple of yarn-spinners such as Jeffrey Archer.
2 The Chekhovian story Chekhov is the father of the modern short story and his influence is still massive and everywhere. James Joyce pointedly claimed not to have read Chekhov when he published Dubliners in 1914 (most of Chekhov's work had been available in English since 1903), but the pointedness of the disclaimer is highly disingenuous. Dubliners, one of the great est short-story collections ever, owes a great deal to Chekhov: or to put it another way, Chekhov liberated Joyce's imagination in the same way Joyce's example later liberated others.
What is the essence of the Chekhovian short story? Chekhov wrote to a friend that, "It was time writers, especially those who are artists, recognised that there is no making out anything in this world." I would say that the Chekhovian point of view is to look at life in all its banality and all its tragic comedy and refuse to make a judgment. To refuse to condemn and refuse to celebrate. To record the actions of human beings as they are and to leave them to speak for themselves (insofar as they can) without manipulation, censure or praise. Hence his famous retort when he was asked to define life. "You ask me what is life? That is like asking: what is a carrot? A carrot is a carrot and that's all there is to it." But the effect of this world-view as expressed in his stories has had an astonishing influence. Katherine Mansfield and Joyce were among the first to write in the Chekhovian spirit, but his cool, dispassionate, unflinching attitude to the human condition resounds in writers as diverse as William Trevor and Raymond Carver, Elizabeth Bowen, John Cheever, Muriel Spark and Alice Munro.
3 The 'Modernist' story I choose this title to introduce the other giant presence in the modern short story - Ernest Hemingway. I use the term to convey the idea of obscurity and deliberate difficulty. Hemingway's most obvious revolutionary contribution to the short story was his style: pared down, laconic, unafraid to repeat the most common adjectives rather than reach for a synonym. But his other great donation was a purposeful opacity. When you read Hemingway's early stories (far and away his best work, as it happens) you understand the situation at once. A young man is going fishing, he camps out for the night. Some waiters gather in a café. In "Hills Like White Elephants" a couple at a railway station wait for a train. The mood is tense between them. Has she had an abortion? And that's about it. Yet somehow Hemingway invests this story and the others with all the covert complexities of an obscure modernist poem. You know there are hidden meanings here and it is the inaccessibility of the subtext that makes the story so memorable. Wilful obscurity in the short story works: over the length of a novel it can be very tiresome. This idea of modernist obscurity overlaps with the next category.
4 The cryptic/ludic story Here the story presents its baffling surface more overtly as a kind of challenge to the reader - Borges and Vladimir Nabokov spring immediately to mind. In these stories there is a meaning to be discovered and deciphered, whereas in Hemingway it's the tantalising out-of-reachness that entrances. A Nabokov story, such as "Spring at Fialta", is meant to be unravelled by the attentive reader - and it may take several goes - but the spirit behind its teasing is fundamentally generous: dig deep and you will discover more, is the implied message. Try harder and you'll be rewarded: the reader is on his mettle. One of the great cryptic short story writers is Rudyard Kipling, something of an unacknowledged genius of "suppressed narration", as it is sometimes known: stories like "Mary Postgate" or "Mrs Bathurst" are wonderfully complex and multilayered. Critics still argue passionately about the correct readings.
5 The mini-novel story It establishes its remit in its title. Like the event-plot story this is one of the first forms the short story took. In a way it is something of a hybrid - half novel, half short story - trying to achieve in a few dozen pages what the novel achieves in a few hundred: a large cast of characters, lots of realistic detail. Chekhov's great story, "My Life", for example, belongs to this category. It has a span of many years, characters fall in love, marry, separate, children are born, people die. All the matter of a Victorian three-decker is somehow compressed into its 50 or so pages. These stories tend to be very long, almost becoming novellas, but their ambition is clear. They eschew ellipsis and allusion for an aggregation of solid fact, as if the story wants to say, "See: you don't need 400 pages to paint a portrait of society."
6 The poetic/mythic story In strong contrast, the poetic/mythic story seems to wish to get as far away from the realistic novel as possible. This category is wide and includes writers as varied as Hemingway (his terse and brutal one-page vignettes that interleave his In Our Time story collection), the stories of Dylan Thomas and DH Lawrence, JG Ballard's moody riffs on inner space and the long prose-poems of writers such as Ted Hughes and Frank O'Hara. This is the short story-quasi-poem and it can range from stream-of-consciousness to the impenetrably gnomic.
7 The biographical story This is the one category that seems harder to define. One way of putting it would be to describe it as the short story deliberately borrowing and replicating the properties of non-fiction: of history, of reportage, of the memoir. Borges's stories play with this technique regularly. The overweening love of footnotes and bibliographical annotation in younger contemporary American writers is a similar example of the genre (or to be more precise, they represent a hybrid of the modernist story and the biographical). Another variation is to introduce the fictive into the lives of real people. I've written short stories about Brahms, Wittgenstein, Braque and Cyril Connolly, for example - imagined fictive episodes in their real lives - yet have drawn on all the research that would be required if the pieces were essays. A very valid definition of biography is that it is "a fiction conceived within the bounds of the observable facts". The biographical story plays with this paradox and in so doing attempts to have its cake and eat it, to capture the strengths of fiction and the non-fictional account simultaneously.
Today, in the UK especially, it has never been harder to get a short story published. The outlets available to a young writer that I benefited from in the 1980s have virtually dried up. Yet, despite these practical difficulties, the short story seems to me to be undergoing something of a revival, both here and in the US. The socio-cultural explanation for this would perhaps be the massive increase in creative-writing degree courses. The short story is the perfect pedagogical tool for this kind of education and conceivably the tens of thousands of stories being written (and read) in these institutions are cultivating a taste for the form in the way that the mass-circulation magazines did in the late 19th and early 20th centuries.
However, I feel that there may be a different reason why readers of the short story have never really gone away. And this has nothing to do with its length. The well-written short story is not suited to the soundbite culture: it's too dense, its effects are too complex for easy digestion. If the zeitgeist is influencing this taste then it may be a sign that we are coming to prefer our art in highly concentrated form. Like a multivitamin pill, a good short story can provide a compressed blast of discerning, intellectual pleasure, one no less intense than that delivered by a novel, despite the shorter duration of its consumption. To read a short story like Joyce's "The Dead", Chekhov's "In the Ravine" or Hemingway's "A Clean Well-Lighted Place", is to be confronted by a fully achieved, complex work of art, either profound or disturbing or darkly comic or moving. The fact that it takes 15 minutes to read is neither here nor there: the potency is manifest and emphatic. Perhaps that's what we are looking for, as readers, more and more these days - a sort of aesthetic daisycutter bomb of a reading experience that does its work with ruthless brevity and concentrated dispatch.
As writers, we turn to the short story for other reasons. I think, finally, it comes down to the opportunity that the short story offers to vary form, tone, narrative and style so quickly and so dramatically. Angus Wilson said he began writing short stories because he could start and finish one in a weekend before he had to return to his job at the British Museum. There is a real investment of effort, to be sure, but it's not the long haul of the novel with its years of generation and execution. You can write a plot-event story one week and a ludic/biographical one the next. Chekhov referred to this same pleasure in the notebook I quoted from above. He had copied down something Alphonse Daudet had written, and it obviously resonated strongly with him too. All short-story writers will know what he means. Daudet's words were these:
"Why are thy songs so short?" a bird was once asked. "Is it because thou art so short of breath?"
The bird replied: "I have very many songs and I should like to sing them all."
Ten Truly Great Short Stories in no particular order
"Spring at Fialta" by Vladimir Nabokov
"My Dream of Flying to Wake Island" by JG Ballard
"Funes, the Memorious" by JL Borges
"Prelude" by Katherine Mansfield
"The Dead" by James Joyce
"Mrs Bathhurst" by Rudyard Kipling
"Day of the Dying Rabbit" by John Updike
"In the Ravine" by Anton Chekhov
"Bang-Bang You're Dead" by Muriel Spark
"Hills Like White Elephants" by Ernest Hemingway
· Fascination, a new collection of stories by William Boyd, is published by Hamish Hamilton on October 7 at £16.99