I focus here on the trajectory of the Abrahamic family of highly theological monotheisms. (Other forms of religiousish behaviour, such as the atheist Asian religions, require their own accounts.) This kind of religion rests on a metaphysical unification of the divine, the social-order, and nature. It gives us an enchanted world and a guarantee that we know our true place in it. The enemies of theological religion have always been philosophy and politics because both present inherently secular ways of grappling with the nature of the world that bypass religion. Their rise has shattered both enchantment and Truth. Religion still creeps about the place, but in a thoroughly subordinate role.
Philosophy is always trying to see the world - natural and social - from a non-orthodox perspective. Philosophers are those annoying people who are always saying, Yes, but..... As in Yes of course God exists, but just suppose that the heavens are like a giant clockwork mechanism that goes automatically after it's been set up? Wouldn't that be an interesting way of looking at things?
Religions can be seen as rich communal storehouses of practical knowledge, about subjects as diverse as ethics, agriculture, hygiene, and economics, that provide access to more wisdom about how to live than any one person could ever come to on their own. (That is a significant reason for their attractiveness even in the modern world, apart from the social club benefits.) But its metaphysical constraints (the ultimate answer is always that God did it) mean that religion can't compete with philosophy in the theoretical knowledge game.
Philosophy constantly spews out alternative approaches to understanding the world, most perniciously by creating dedicated collaborative epistemic communities interested in particular subjects - previously. (Every major academic discipline, from physics to economics to mathematics was started by philosophers.) These lovers of theoretical knowledge for its own sake leave the speculative orientation of pure philosophy behind as they develop specialist methodologies and employ them systematically upon different bits of the world. If they are given space and freedom (universities) they generate vast quantities of robust theoretical knowledge about how the world really works. Religion of course knows this full well, and always strove to keep the keys to knowledge to itself, for example in medieval Europe by exercising tight control over literacy, libraries, and universities.
But the Enlightenment let philosophy out of its bottle for good, and since then it has succeeded to an astonishing degree in producing much much better accounts of how the world works than any religion. It has convincingly shown that all religions make serious mistakes about matters of fact (astronomy, history, medicine, etc), and about how facts are connected (evolution, cosmology, etc). The sacred triangle between the divine, human society and nature just isn't out there, so, obviously, your prayers cannot affect the weather or your marriage prospects.
Academic inquiry has shown that the world is actually much more complicated than religion ever claimed - exposing the limits of religious knowledge communities. But at the same time it has also shown that the world doesn't need God, or any other supernatural agency, to make it work. Even in ethics, it has shown up the inconsistencies, biases, and questionable foundations of religious teachings and identified a plethora of alternatives that beat iron age patriarchal tribal mores hands down. (How many genocides does God order in the Old Testament? Death for picking up sticks on the Sabbath? We can do better than that.)
Religion can cope with atheism since atheists at least take religion seriously enough to argue about. But over time, the exercise of applied philosophy produces naturalists who don't even find religion compelling enough to reject (any more than fairies or UFOs), and who consider it only as an object of study, for example by anthropologists and sociologists.
Politics is about earthly power and has an underlying Machiavellian pragmatism that has always been in tension with religious accounts of the divine and True social order. Politicians are the ones who have bright ideas like this: Hey, what if I stop killing and torturing the people who disagree with the official church about the Doctrine of the Trinity. Think of all the money I would save and all the extra taxes I could raise.
The pragmatism of politics is a threat to the theological domination of society in more than one way. (Hence the truism that the separation of church and state actually protects religion.) On the one hand religion can easily be corrupted by politics if it is seduced by the possibility of using direct earthly power to make sure the social world follows the divine script properly. The exercise of that power can't be done without yielding to the logic of politics and becoming embroiled in the profane arts of government (like the prince bishops of medieval Europe or Khamenei). And then the people stop taking the clerics' heavenly status claims seriously.
On the other hand politics can reach out to religion as a source of legitimacy for its own Machiavellian strategies - blessing kings, wars, and taxes for example. Here the seductions themselves are earthly: access to the king's court, monetary rewards, and extra-legal privileges for clerics. Sooner or later the clerics are selling indulgences to build gorgeous palaces and raping peasant children with impunity. And then, again, the people tend to lose faith
Despite these tensions, politics and religion can co-exist in a reasonably stable arms-length relationship since each recognises that it needs what only the other can provide (churches need funding; kings need right as well as might). So even when the relationship broke down (e.g. in the Protestant Reformation), it tended to come back. That is, until politics invented something new and enormously powerful: large nation states combined with democratic forms of government. These replaced kings, whose legitimacy as rulers was always based on a dubious metaphysics of divine social order (backed by raw ruthless power), with professional politicians. The legitimacy of political power holders now depends on their ability to appeal to the people, not the clerics; the legitimacy of their actions depends on their adherence to temporal constitutions not divine law.
Since the unleashing of politics and philosophy - since the arrival of modernity - religion has been forced to adapt. Out went the mighty all-encompassing metaphysics in the face of the top-down scrutiny of querulous university academics. Out went the claim to absolute truth in the face of the bottom-up revolution of peasants demanding to be consulted. The new religion is individualised, marketised, and deracinated.
IndividualisationLiberal democracy is all about dealing with the fact of reasonable disagreement, that as soon as one relaxes totalitarian social control one will find that other people have completely sincere beliefs and judgements that differ from yours. Luther of course when he said, 'Here I stand. I can do no other', didn't think he was making a plea for individual freedom of conscience. He thought he was right. But when lots of people start thinking they have the right to stand up for what they believe, one has to find a modus vivendi (or face interminable civil war). One has to find a way to live in the same political society as people whom you believe are utterly - metaphysically - wrong.
The trouble is that once you set up some rules for keeping the peace - state neutrality on religious matters, no religion in the public square, etc - they do much more than keep the peace. One ends up finding one's Lutheran, Catholic, Jewish or even atheist neighbours actually very pleasant. It becomes harder and harder to remember that the really important thing in the relationship is that they're going to hell. Politeness and civility come to matter, and people decide to keep their revelations to themselves so as not to be an ass. Instead of religious credos being too dangerous to utter in public, mentioning them becomes something of a social gaffe, like blurting out your taste in pornography.
Meanwhile the damned philosophers are busy creating a secular intellectual space open to all and independent of any religious dogma. Religion's storehouse of wisdom is bypassed and then surpassed by towering cathedrals of secular knowledge. Then it is subjected to criticism. Of its science, history, ethics; even of its literary qualities! If you try to talk about miracles and divine providence these people ask if you also believe in UFOs or fairies; and ethnographers may ask if they can study your interesting sub-culture.
For the great majority of the world’s faithful - those who do not live in sequestered theocratic communities - religion is a personal matter that takes place largely within the mind of the individual. Outside this inner world they must act as if their beliefs were irrelevant. Above them, the world turns according to a godless physics, biology, economics, etc. Around them, social interactions are shaped by the fact of pluralism: the faithful must bear in mind the different beliefs of others, and limit their talk and actions to what heretics and the godless can go along with.
MarketisationPluralism and individualisation also brings markets. Of course, marketing is nothing new to the monotheistic religions, but previously they were often in the comfortable state of being monopolies able to charge whatever price for their spiritual services they wanted in their customers’ attention and property. That meant that they tended to set a 'price' for their services that maximised their profits, even though it left a great many people unserved or underserved (what economists call the deadweight loss of monopoly). Indeed, in the heyday of religion most people probably weren't particularly religious because they weren't the core market and the core product wasn't designed or priced with them in mind.
Nevertheless, monopoly not customer service is what every corporation dreams of and struggles for. In this light, monotheism's visceral distaste for polytheism can be seen as typical protectionist behaviour (3 out of the 10 commandments are about monopoly and trademark protection). Under monotheism when your prayers aren't answered it's your fault and you need to pay/pray more. In contrast, if you have a choice of gods you can play them off against each other, as the Greeks did, by switching to a different god whenever one fails to deliver. Multiple gods means market competition between divine service providers and lower prices for the ordinary consumer (and so also more freedom to think for yourself).
Monopolist religions used some of their excess profits to invest in R&D in their principal technology - theology. On the one hand they were able to train and employ the best minds of their time, which led to powerful innovations. The invention of the afterlife for example is a brilliant way of getting even more out of your present customers. As Pascal pointed out, by extending people's utility function into eternity you alter their cost-benefit analysis in favour of greater commitment to the church in this life (however much you discount future well-being, eternity trumps present consumption). On the other hand this theological research was quite inefficient for society as a whole, since all those great minds were only allowed to think about one area. And it turns out that when great minds take up the serious systematic study of God, they raise at least as many problems as they solve (which then have to be kept quiet - Luther!).
The modern state of affairs requires religions to compete with each other like regular companies in a free market. Monopoly profits from captive consumers are a thing of the past – in America more than 50% of the population has changed their religious affiliation at least once. Since people tend to think all religion is pretty much the same thing (different paths to the same ultimate truth) all the religious denominations are desperate to avoid being fully commoditised, like pork bellies, as fully substitutable with each other and compared purely on price. So they strive to differentiate themselves, like supermarkets or airlines. They identify and target distinct types of customers like single mothers, high-flying business types, or specific ethnic groups, and then woo them with free lunches, parking, daycare, and sermons they like to hear. To increase retention rates they pay close attention to customer feedback and tweak their offerings to meet changing tastes. (Vatican II: Latin isn't working for us anymore.) They have to come up with competition strategies against new start-up religions and franchise operators moving into their areas.
All this marketing is of course quite profane. The success of a religion is now judged in terms of the numbers of 'bums on seats' it can corral into its services, and the amount of time and money those bums can be persuaded to give up. Not by the truth of its theology. The True religion must compete in the marketplace with the mass of heresies and trivial spiritualisms (like horoscopes and crystal-ball gazing), distinguishable only by the success of their marketing strategies. And the marketing affects the content of modern religions as well, turning them into a set of recognisable hack genres such as one finds in airport book-stores: self-help/personal-growth; horror (I’m thinking of that weird one with the snakes); Dan Brownian secret mysteries; Twilight fantasies; fantasy epics (Catholicism); etc. Many of the fastest growing religions hardly bother with theology at all, because their customers don't have time for all that wordy stuff.
One of the interesting consequences of this is that modern religion has diversified its offerings to supply every consumer niche, rather than demanding that the spiritually inclined accommodate themselves to the one True church. Tastes which went somewhat unrecognised and uncatered to before are now met by specialist providers. For example, fundamentalists eschew the established religious providers altogether and start their own co-operatives to meet their peculiar concerns with purity. So in some countries with a thriving free market for religion (most prominently, America) there is probably more religious activity than ever before.
DeracinationContemporary religion has been deracinated - separated from its cultural context. In the enchanted world religion was always felt as much as thought, because it was literally embedded into the social landscape and rituals of everyday life. That gave religion a solid foundation, and helped make defection difficult to even imagine. It also protected religion from theological excesses by requiring that religion take a form that a human society could actually live by. But now that the enchanted world has shattered, theology can no longer depend on the solid foundations of social practices. Almost the whole of Western Europe, for example, enjoys its quaint Christian culture (history, holidays, recipes, nice old buildings, church weddings, etc.) as part of a cultural identity, but hasn't the faintest interest in Christianity. Religion for us is an ethnic identity thing, not a belief thing.
On the other hand, the falling away of the cultural roots of religion means that those people interested in personal spirituality have more freedom than ever to download and try out a new theology. That drives the thriving market for religion already discussed. But it also has implications for the kind of religion that people can have. In particular, accessing the wisdom of religion about the human condition (all those brilliant minds focussed on analysis and commentary for generations) requires more than reading the sacred book by itself (or, in the case of many evangelicals, the few quotations they like).
The reason many fundamentalists are so unpleasantly and rigidly righteous is that they are people of the book who lack the lived culture of the book. They mistake religious knowledge of the kind one can get by reading a book in the way one might read Euclid's geometry, for religious wisdom, which requires a much deeper immersion in and personal subordination to the culture of that religion.
Fundamentalists, and their worried observers, make an even bigger mistake in believing that religious movements based on literalist readings of sacred texts constitute a revival of old-time religion. Instead they are a fundamentally modern and secular phenomena of individuals searching for meaning in their own lives and fitting together a personal theology from bits and pieces of texts that suit them, and then joining clubs that see things the same way.
ConclusionThe secularisation thesis actually happened, though not exactly in the way 20th century sociologists predicted. They thought levels of religiousity would decline as religion lost its hegemonic dominance over society, whereas in fact it is the nature of religiousity which has changed. Contemporary religion is a thoroughly secular affair. It is matter of individual conscience projected outwards, and as such hard to distinguish from secular faiths like environmentalism or socialism.
Religious individuals, like socialists, see the world a certain way, but they generally recognise that others see it differently and they understand the significance and legitimacy of their faith in terms of private personal belief. The deinstitutionalisation of religion in liberal democratic societies has led to cut-throat competition which has shifted religions from price-makers to price-takers. Religions now recognise that the customer is always right and make offerings to the public (instead of vice versa). Religion has been cut free from its cultural base and now floats freely, as bite-size memes that anyone can download from the cloud when they feel like it and assemble for themselves.
Religious readers particularly may appreciate Rod Dreher's discussion of this post over at his blog on The American Conservative. He considers the phenomenon of secular religion from the religious perspective, and in the context of the heavy-weight religious philosophers Charles Taylor and Alasdair MacIntyre.