The presenter said his documentaries had often been about “morally fraught” people whose stories “made me nervous”.
But he said broadcasters like the BBC now had “a temptation to lay low” and avoid difficult subjects for fear of causing offence.
There is an “atmosphere of anxiety” in the TV industry, he suggested.
Theroux was giving the annual keynote MacTaggart lecture at the Edinburgh TV Festival.
“I want to take the risk of going out speaking to people I profoundly disagree with and making documentaries about them,” he said.
‘Made me nervous’
He recalled making his name by “investigating worlds viewed as stigmatised or controversial”, including the porn industry, the far right, Nazis, gangs and sexual predators.
“Often the stories made me nervous. They felt risky,” he said.
“But it was also true that those shows that had real moral complexity to them were the ones that worked best.
“The less morally fraught episodes – the ones that were safer – haven’t aged so well.”
There has been a welcome shift in mindset so broadcasters today are “more thoughtful about representation” and aware of “the need not to wantonly give offence”, he said.
“But I wonder if there is something else going on as well. That the very laudable aims of not giving offence have created an atmosphere of anxiety that sometimes leads to less confident, less morally complex film-making.”
From his time working for the BBC, he said he could see “all-too-well the no-win situation it often finds itself in”.
The corporation, he said, was “trying to anticipate the latest volleys of criticisms, stampeded by this or that interest group, avoiding offence”.
“Often the criticisms come from its own former employees, writing for privately owned newspapers whose proprietors would be all too happy to see their competition eliminated.
“And so there is a temptation to lay low, to play it safe, to avoid the difficult subjects.
“But in avoiding those pinch points, the unresolved areas of culture where our anxieties and our painful dilemmas lie, we aren’t just failing to do our jobs, we are missing our greatest opportunities. For feeling. For figuring things out in a benign and thoughtful way. For expanding our thinking. For creating a union of connected souls.
“And what after all is the alternative? Playing it safe? Following a formula? That may be a route to success for some. It never worked for me.”
He called for television that is “confrontational, surprising and upsetting”, and urged producers: “Take risks. Sail close to the wind.”