Following the inception of Charles Darwin's theory of natural selection in 1838, the development of Darwin's theory to explain the "mystery of mysteries" of how new species originated was his "prime hobby" in the background to his main occupation of publishing the scientific results of the Beagle voyage. He was settling into married life, but suffered from bouts of illness and after his first child was born the family moved to rural Down House as a family home away from the pressures of London.
The publication in 1839 of his Journal and Remarks (now known as The Voyage of the Beagle) brought him success as an author, and in 1842 he published his first major scientific book, The Structure and Distribution of Coral Reefs, setting out his theory of the formation of coral atolls. He wrote out a sketch setting out his basic ideas on transmutation of species, which he expanded into an "essay" in 1844, and discussed his theory with friends as well as continuing with experiments and wide investigations. In the same year the anonymous Vestiges of the Natural History of Creation brought wide public interest in evolutionary ideas, but also showed the need for sound evidence to gain scientific acceptance of evolution.
In 1846 he completed his third geological book, and turned from supervising the publication of expert reports on the findings from the voyage to examining barnacle specimens himself. This grew into an eight-year study, making use of his theory to find hitherto unknown relationships between the many species of barnacle, and establishing his expertise as a biologist. His faith in Christianity dwindled and he stopped going to church. In 1851 his treasured daughter suffered a long illness and died. In 1854 he resumed his work on the species question which led on to the publication of Darwin's theory.
Background
Charles Darwin became a naturalist at a point in the history of evolutionary thought when theories of Transmutation were being developed to explain discrepancies in the established faith based explanations of species. He considered these problems at first hand during the Beagle survey. On its return in 1836 his ideas developed rapidly. His collections and writings established him as an eminent geologist and collector.
Darwin read Malthus's Essay on the Principle of Population in the context of his findings about species relating to localities, enquiries into animal breeding, and ideas of Natural "laws of harmony". Around late November 1838 he compared breeders selecting traits with a Malthusian Nature selecting from variants thrown up by "chance" so that "every part of newly acquired structure is fully practical & perfected", thinking this "a beautiful part of my theory" of how species originated.
His theory of how species originated had now come together in principle, but he was vividly aware of the difficulties he would face in getting it accepted by his friends and colleagues in the scientific establishment. On 19 December 1838 as secretary of the Geological Society of London Darwin witnessed the vicious interrogation by Richard Owen and his allies of Darwin's old tutor Robert Edmund Grant in which they ridiculed Grant's Lamarckian heresy, showing establishment intolerance of materialist theories.
Married life In 1839, now married to Emma and settled in London, Darwin continued to look to the countryside for information and began a Questions & Experiments notebook with ideas that would have seemed bizarrely mundane to the "philosophical" scientists of the time. He printed Questions about the Breeding of Animals and sent them out to gentlemen farmers, asking for information on animal husbandry from their nurserymen and gamekeepers on how they crossed varieties or selected offspring. Of only three who responded one simply found the questions too overwhelming to answer. He found agreement with the visiting Swiss botanist Alphonse de Candolle whose father Augustin had used the idea of "nature's war". However, when he tried explaining his theory to Hensleigh Wedgwood, his cousin "seemed to think it absurd... that [a] tiger springing an inch further would determine his preservation".
The publication in May of Darwin's Journal and Remarks (The Voyage of the Beagle) brought reviews accusing him of theorising rather than letting the facts speak for themselves. He turned his attention to expanding his investigations and theory of the formation of coral atolls as the first part of his planned book on geology.
In December as Emma's first pregnancy progressed, Charles fell ill and accomplished little during the following year. He did accept a position on the Council of the Geographical Society in May 1840. In 1841 he became able to work for short periods a couple of days a week, and produced a paper on stones and debris being carried by ice floes, but his condition did not improve. Having consulted his father he began looking for a house in the countryside to escape a city suffering from economic depression and civil unrest. Owen was one of the few scientific friends to visit Darwin at this time, but Owen's opposition to any hint of Transmutation made Darwin keep quiet about his theories.
First writings on the theory In January 1842 Darwin sent a tentative description of his ideas in a letter to Lyell, who was then touring America. Lyell, dismayed that his erstwhile ally had become a Transmutationist, noted that Darwin "denies seeing a beginning to each crop of species".
Darwin's book The Structure and Distribution of Coral Reefs on his theory of atoll formation was published in May after more than three years of work, with Part 4: Fish of Zoology of the Voyage of H.M.S. Beagle also going to print. Illness was a continuing problem, and he and Emma left London on 18 May, visiting her parents at Maer Hall before moving on to Shrewsbury on 15 June for rest and quiet. Now Darwin "first allowed myself the satisfaction of writing a very brief abstract of my theory in pencil in 35 pages", the '"Pencil Sketch"' of his theory. This discussed farmers breeding animals, gave the analogy of overpopulation and competition leading to "Natural Selection" through the "war of nature" and the mechanism of descent. Every living thing was related in a branching pedigree, not ascending a Lamarckian ladder, and this pedigree was the proper basis for classification. He thought it "derogatory" to argue that God had made every kind of parasite and worm on an individual whim. Already, a rough form of the phrasing and ideas which he went on to publish 17 years later in the closing paragraph of On the Origin of Species can be seen in his conclusion in this first draft:
Essay
They returned on 18 July to a London seething with Chartist unrest, and Darwin copied and scribbled changes to his "Sketch" until it was almost illegible. He returned to house hunting and found a former parsonage in the rural hamlet of Downe at a good price. A general strike led to huge demonstrations all over London, but was crushed by troops by the time Darwin moved. On 17 September 1842 the family moved into Down House (around 1850 the village changed its name to Downe to avoid confusion with County Down in Ireland, but the house kept the old spelling). After a series of alterations Darwin settled in, and in 1843 returned to writing his Volcanic Islands. In May he began a (mostly geological) country diary he called The General Aspect.
In response to a request from George Robert Waterhouse for advice on classification, Darwin replied that it properly "consists in grouping beings according to their actual relationship, ie their consanguinity, or descent from common stocks". He followed this up with another letter expressing belief that "all the orders, families & genera amongst the Mammals are merely artificial terms highly useful to show the relationship of those members of the series, which have not become extinct", before cautiously asking for the letter to be returned. Waterhouse was influenced by Owen and in a paper attacked such heresies, setting his species in the symbolic circles of the Quinarian system, not hereditary trees. Darwin sent a sharp response about these "vicious circles".
Darwin became a close friend of the botanist Joseph Dalton Hooker, and on 11 January 1844 wrote to with melodramatic humour that he was "almost convinced (quite contrary to opinion I started with) that species are not (it is like confessing a murder) immutable. Heaven forfend me from Lamarck nonsense of a "tendency to progression" “adaptations from the slow willing of animals" &c,—but the conclusions I am led to are not widely different from his—though the means of change are wholly so— I think I have found out (here's presumption!) the simple way by which species become exquisitely adapted to various ends." Hooker's reply was cautious but friendly, saying that "There may in my opinion have been a series of productions on different spots, & also a gradual change of species. I shall be delighted to hear how you think that this change may have taken place, as no presently conceived opinions satisfy me on the subject."
Darwin worked up his "Sketch" into a 189-page '"Essay"' and in July entrusted the manuscript to the local schoolmaster to copy. He then wrote a difficult letter to be opened by his wife in the event of his death requesting that the essay be published posthumously. He started his Geological Observations on South America, and corresponded with Hooker about this, feeding in questions related to his "Essay". The copied "Essay", now 231 pages, was returned to him for corrections in September. Then one day he brought it to Emma and asked her to read it. She went through the pages, making notes in the margins pointing out unclear passages and showing where she disagreed.