What is the real death toll in Iraq?

• Jonathan Steele and Suzanne Goldenberg
• 

  People gather at the Baquba hospital morgue next to caskets containing the bodies of six of their family members who were killed by a roadside bomb south of Baquba, Iraq. Photograph: Ali Yussef/AFP/Getty images

  The following correction was printed in the Guardian's Corrections and clarifications column, Tuesday July 1 2008.

  In the article below we were wrong to say that a household survey conducted by the World Health Organisation and the Iraqi health ministry found that the rate of violent deaths had doubled in Iraq after the invasion. The survey did not make this finding. Figures that were unadjusted for under-reporting showed a doubling of the rate of all deaths and a violence-related death rate about 11 times higher. The article said the survey estimated that 151,000 civilians had been killed since the invasion. That figure included combatants. The article below should have also made clear that the Lancet and Opinion Research Business surveys included combatants as well as civilians.

  ---

  Lieutenant General Tommy Franks, who led the invasions of Iraq and Afghanistan during his time as head of US Central Command, once announced, "We don't do body counts." This blunt response to a question about civilian casualties was an attempt to distance George Bush's wars from the disaster of Vietnam. One of the rituals of that earlier conflict was the daily announcement of how many Vietnamese fighters US forces had killed. It was supposed to convince a sceptical American public that victory was coming. But the "body count" concept sounded callous - and never more so than when it emerged that many of the alleged guerrilla dead were in fact women, children and other unarmed civilians.

  Iraq was going to be different. The US would count its own dead (now close to 4,000), but the toll the war was taking on Iraqis was not a matter the Pentagon or any other US government department intended to quantify. Especially once Bush had declared "mission accomplished" on May 1 2003 - after that, every new Iraqi who died by violence would be a signal that the president was wrong, and would show that a war conducted in the name of humanitarian intervention was exacting a mounting humanitarian toll of its own.

  But even though the Americans were not counting, people were dying, and every victim had a name and a family. Wedding parties were bombed by US planes, couples driving home at night were shot at checkpoints because they missed a flashlight warning them to stop, and hundreds of other unarmed civilians were killed for no legitimate cause. In just the last three weeks of April 2003, after Saddam's statue and his regime were toppled, US forces killed at least 266 civilians - a pattern of overeager resort to fire which has continued to this day.

  So five years after Bush and Tony Blair launched the invasion of Iraq against the wishes of a majority of UN members, no one knows how many Iraqis have died. We do know that more than two million have fled abroad. Another 1.5 million have sought safety elsewhere in Iraq. We know that the combined horror of car bombs, suicide attacks, sectarian killing and disproportionate US counter-insurgency tactics and air strikes have produced the worst humanitarian catastrophe in today's world. But the exact death toll remains a mystery.

  There is no shortage of estimates, but they vary enormously. The Iraqi ministry of health initially tried to keep a count based on morgue records but then stopped releasing figures under pressure from the US-supported government in the Green Zone. The director of the Baghdad morgue, already under stress because of the mounting horror of his work, was threatened with death on the grounds that by publishing statistics he was causing embarrassment. The families of the bereaved wanted him to tell the truth, but like other professionals he came to the view that he had to flee Iraq.

  An independent UK-based research group, calling itself the Iraq Body Count (IBC), collates all fatality reports in the media where there are two or more sources as well as figures from hospitals and other official sources. At least four household surveys have been done asking Iraqis to list the family members they have lost. The results have then been extrapolated to Iraq's total population to give a nationwide estimate.

  The results range from just under 100,000 dead to well over a million. Inevitably, the issue has become a political football, with the Bush administration, the British government and other supporters of the US-led occupation seizing on the lowest estimates and opponents on the highest.

  Trying to cut one's way through the statistical jungle quickly becomes a battle over methodology, and sometimes over motives. Critics even raise the spectre of fraud, claiming survey interviewers fake the evidence by inflating the figures for political reasons.

  The initiators of the Iraq Body Count, which has produced the lowest figures, were no advocates of the invasion. John Sloboda, an Oxford-based psychologist who co-founded the group, was provoked to do so precisely because he saw no official institution willing to count the human cost of the war. His organisation decided to use "passive surveillance", a statistically conservative method that only deals with facts on the ground. The IBC lists all cases where at least two media sources report an incident causing one or several deaths, keeping a careful tab of the victims' age, gender, occupation, manner and place of death, where information is available. In addition to providing a running total (which now numbers just under 90,000), its figures have allowed the IBC to produce some of the best data and analysis on how the Iraq war has changed over five years. They reveal that the Americans killed four times more civilians in the first two years of the war (thereby provoking armed resistance to the occupation) than al-Qaida-linked insurgents did, in spite of the media's emphasis on car bombs and suicide attacks. They also show the explosion of criminal violence since 2003, one byproduct of the removal of Saddam Hussein's draconian security methods - or what Bush would call Iraq's new "freedom".

  The World Health Organisation (WHO) and the Iraqi health ministry conducted a survey of 10,860 households in 2007. Ministry employees questioned 10 households in each of more than 1,000 clusters across Iraq's 18 provinces, picked to give a representative sample of the country's population. People were asked to list any family deaths in the two years before the invasion and the first three years after. Some 115 (11%) of the clusters, mostly in Baghdad and the mainly Sunni province of Anbar, could not be approached because of insecurity. The organisers decided to calculate the probable number of deaths there.

  The results showed that the national rate of killing between April 2003 and June 2006 averaged just over 120 a day. This was double the number killed during Saddam's last two years in power. The study's figures ignored deaths from accident, disease or suicide. They estimated the civilian death toll in the occupation's first three years as 151,000. The true figure could be anywhere between 104,000 and 230,000 allowing for misreporting, they said. But even the lowest figure on this range is more than twice as high as the IBC's figure of 47,000 for the occupation's first three years. In December 2005, Bush gave a figure of 30,000 civilian deaths.

  The WHO/Iraqi health ministry study was published in January this year in the New England Journal of Medicine. Dr Salih Mahdi Motlab al-Hasanawi, the health minister appointed after the ministry's ban on releasing official morgue figures, said the survey was prompted by controversy over civilian casualties.

  What he had in mind was the storm aroused by two studies led by researchers from Johns Hopkins University in Baltimore and published in the prominent British medical journal, the Lancet. In the first survey in 2004, 990 randomly selected families in representative locations across Iraq were asked to produce the death certificates and list the names of members who died between January 1 2002 and the start of the invasion, and those who died thereafter. Subtracting the former from the latter, this produced an "excess" rate. This was then used to calculate the deaths in excess of normal fatality rates in Iraq's total population.

  The first survey found at least 98,000 such deaths up to October 2004. The second survey, in the summer of 2006, interviewed a separate but also randomly chosen sample of 1,849 households and found an excess of 655,000 deaths up to June 2006, of which 601,027 were said to be from violence rather than natural causes. This amounts to 2.5% of Iraq's population, or more than 500 deaths a day since the invasion.

  The estimates were explosive and were widely reported in the Middle East and around the world. They met instant dismissal from the White House and Downing Street. "I don't consider it a credible report," Bush told reporters. Blair's spokesman said the study's result "was not one we believe to be anywhere near accurate".

  Sitting in his office in Camden in north London, where every surface is covered with wobbly piles of files, the Lancet's editor, Richard Horton, admits that the figure "seems crazy". "But the second study validated the first one. The pre-invasion mortality rate is the same in both, and the upward lines of the post-invasion rate are exactly the same", he says.

  He is particularly pleased by information unearthed last year by a Freedom of Information request by the BBC's Owen Bennett-Jones. This found that the chief scientific adviser to the Ministry of Defence described the methods used by the second survey as "close to best practice" and added that the "study design is robust". The adviser warned the government to be "cautious" about criticising the survey findings .

  The study in the Lancet was led by researchers at the Centre for Refugee and Disaster Response at Johns Hopkins University's School of Public Health. It was an epidemiological survey, one of the fundamental methods of charting disease in a population or the long-term effects of a disaster. Like Horton, Gilbert Burnham, the centre's co-director, recalls his initial surprise when the figures from the team of Iraqi researchers started coming out: "It was like, 'Wow. Is this really the case? What is happening here?'"

  Epidemiology is not a field for the faint-hearted. Like most things in the realm of statistics, it is innately political, touching on matters of health policy, international disaster aid in the aftermath of a disaster and, in the case of Iraq, the consequences of war. Leslie Roberts, a lecturer at Columbia University's Mailman School of Public Health, who was the driving force behind the Baltimore project, had a record of moral engagement in the horrors of war.

  A high-school physics teacher who switched careers to take a degree in public health, he did mortality surveys charting genocide in Rwanda and Congo in the 1990s.

  Burnham's career route was circuitous, too. In 1970, while performing compulsory US military service as an army doctor in South Korea, Burnham and a team of medics began setting up village clinics. This led him into public health issues, which would later see him surveying the human cost of war in Kosovo and elsewhere.

  The two men's collaboration on Iraq began in 2004, when most Americans still had faith in their president and his war. Burnham was uneasy about the news coming out of Iraq, but even more, about what he did not know. How was the population coping with the violence? How was Iraq's healthcare system functioning amid the breakdown of electricity and sanitation networks? "There was already a concern that we really don't know what is happening," he says. "There was clearly a need that had to be addressed to try to find some solid way of estimating what was happening in the conflict."

  Roberts distrusted the official estimates of the war's toll from the outset. "People had no clue what was going on in Iraq," he says. "At that point, if you had looked at the official numbers coming out from various sources - the Iraqi government, Iraq Body Count - the violence rate in Baltimore would have been far higher than in Iraq. That is, if you believe those official numbers of 12,000 by the time our study came out, and that's just crazy."

  The Baltimore findings ran into criticism from all sides, including the Iraq Body Count. Sloboda wondered where the physical evidence was for the huge numbers of bodies or the vast crowds of wounded people, an estimated 800,000, that would be expected to come to Iraq's hospitals if violence was really producing a death toll of 655,000. "People may be frightened or hospitals are closed," Horton responds. As for the "missing bodies", he assumes people are often buried without first being taken to morgues where they can be counted.

  While the first survey chose household clusters by means of a global positioning system, the second one made a random selection of main streets within a series of areas chosen by population size. Twelve of these clusters were in Baghdad, five in Mosul, and three each in other areas such as Basra, Diyala and Anbar. A residential street that crossed the main street was then picked and two teams, each with two men and two women, would go to 40 households along it. Critics writing in Science magazine pointed to "main-street bias", since most car bombings take place on main streets or markets and people living nearby would be at greater risk than other Iraqis.

  Madelyn Hicks of the the Institute of Psychiatry, King's College London, cast doubt on the Baltimore survey because of its interviewers' apparent haste. The study says "one team could typically complete a cluster of 40 households in one day". It does not say how many hours they worked or whether the four interviewers worked individually, in small groups, or all together. If they worked as a large group over 10 hours - a long time in Iraq's heat and tough security conditions - this would allow only 15 minutes per household. In the conditions that prevailed, she said, "sometimes no one answers the door while residents peek through the curtains or check first with someone else at home. Once the door is opened, the team must be introduced and they must explain why they are there. If invited in, further introductions are made to household members, chairs are offered and refreshment is served ... Prospective interviewees do this not just to be polite, but for the opportunity to assess who these researchers really might be, their underlying motives, how they might choose to engage with these strangers, and what they would have to gain or lose in this engagement," she wrote in a critique (available at www.hicn.org).

  She highlights a comment that Roberts made to the BBC, that in 2004 "it usually took a two-person team about three hours to interview a 30-house cluster" (an average of six minutes per house) and that this was "about twice as long" as in 2006. She finds it hard to believe that conducting a serious interview on such an emotional issue could be done so rapidly. The study said the choice of adjacent houses in a cluster was done deliberately for security reasons. "It was felt that the benign purpose of the survey would spread quickly by word of mouth among households, thus lessening risk to interviewers," it said. Hicks wonders whether this created short cuts in which the interviewers did not follow ethical and scientific norms by taking time to explain the survey in a calm atmosphere and get informed consent from every head of household.

  Sloboda applauds the spirit behind the Baltimore survey and some of its findings. "It's a valuable contribution in that it confirms trends over time, and shows a rising trend year on year," he says. But he shares Hicks's doubts. "There is a lot of debate in the research community about how these figures should be interpreted. Several of the most important comments centre on the lack of detail in the report about the precise methodology and the protocols that were adopted by interviewers in a highly volatile and tense situation," he says.

  The most detailed criticism has appeared in a long study in the January 2008 issue of the National Journal, a right-of-centre magazine aimed at Washington policy-makers. It raised three sets of questions: possible flaws in the study's design and execution; a "lack of transparency in the data which has raised suspicions of fraud"; and political preferences held by the authors and the funders, who include George Soros's Open Society Institute.

  The National Journal described the Baltimore study's Iraqi field director, Riyadh Lafta, as "a child-health official in Saddam Hussein's ministry of health when the ministry was trying to end international sanctions against Iraq". They say he claimed high rates of child malnutrition during the sanctions period without giving data from the pre-sanctions period by which they could be measured. They allege he and his interviewers for the Baltimore study worked "under brutal political pressure" at a time when the health ministry was under the control of Moqtada al-Sadr, the anti-occupation Shia religious leader. They say Lafta had little supervision and has rarely appeared in public or been interviewed. He presented his study to an off-the-record meeting of experts in Geneva last May but none of the attendees has agreed to describe his remarks. He refused to respond to emailed questions from the National Journal.

  The Baltimore study's authors have declined to provide the interviewers' reports and questionnaires, a non-transparency issue which also worries the IBC researchers.

  Responding to the criticism that it was impossible to visit so many homes in a day, the researchers said in earlier Guardian interviews that most visits were terminated after the first question - if the household did not register any deaths. As for the allegation that their study suffered from "main-street bias", and that their results were skewed by car bombs which generally detonate on larger roads, the researchers argued that car bombs and other explosions account for only 13% of deaths.

  In a long reply to the National Journal recently released on their website (www.jhsph.edu), the Johns Hopkins School of Public Health defends Lafta, saying he was one of small number of doctors who never joined Saddam's Ba'ath party. He has frequently worked with other international groups since 2003 and "asked that he not be contacted by the media out of concern for his safety and that of his family, a not unreasonable request in Iraq where doctors and academics are major assassination targets". On the alleged speed of the interviewers' work, the Baltimore people say most houses were in walled compounds and doorstep interviews were "judged to be the best survey techniques from the point of security and cultural acceptability". They say the interviewers' data with names and addresses cannot be released for fear of endangering the families. As for the funders' political views, they say this is irrelevant since the interviewers were not told who the funders were.

  Finally, they point out that more recent data confirm their findings and even suggest a higher figure. The British polling firm Opinion Research Business (ORB) asked 1,720 Iraqi adults last summer if they had lost family members by violence since 2003; 16% had lost one, and 5% two. Using the 2005 census total of 4,050,597 households in Iraq, this suggests 1,220,580 deaths since the invasion. Accounting for a standard margin of error, ORB says, "We believe the range is a minimum of 733,158 to a maximum of 1,446,063."

  One expert also believes the number of civilian casualties may be higher than the Baltimore/Lancet figure. Frederick "Skip" Burkle is a professor in the department of public health and epidemiology at Harvard University who ran Iraq's ministry of health after the war but was sacked by the US and replaced by a Bush loyalist. He says the survey ignored the occupation's indirect or secondary casualties - deaths caused by the destruction of health services, unemployment and lack of electricity. Two surveys by non-government organisations found a rise in infant mortality and malnutrition, he notes, so why are those figures not reflected in the second study that appeared in the Lancet?

  The controversy will clearly run and run, probably long after the Iraq war eventually ends. One thing is certain, and it provides no comfort for Bush, Blair and other occupation supporters. They continue to claim that, whatever errors may have been committed since the invasion, the judgment of history will be that the toppling of a brutal dictatorship was an unmitigated benefit. That alone means the invasion was a blessing for the people of Iraq.

  Alas for Bush and Blair, most statisticians do not support their case. Nor can any journalist or other independent witness who has seen the pain of the bereaved still living in post-invasion Iraq or the millions who have escaped to Jordan and Syria. Estimates of the Iraqi deaths caused by Saddam's regime amount to a maximum of one million over a 35-year period (100,000 Kurds in the Anfal campaign in the 1980s; 400,000 in the war against Iran; 100,000 Shias in the suppressed uprising of 1991; and an unknown number executed in his prisons and torture chambers). Averaged over his time in power, the annual rate does not exceed 29,000.

  Only the conservatively calculated Iraq Body Count death toll credits the occupation with an average annual rate that is less than that - some 18,000 deaths in the five years so far. Every other source, from the WHO to the surveys of Iraqi households, puts the average well above the Saddam-era figure. Those who claim Saddam's toppling made life safer for Iraqis have a lot of explaining to do.

  247 dead: Last week's death toll (as counted by Iraq Body Count)

  Monday March 10 - 34 dead
  Including Dr Khalid Nasir, the only neurosurgeon in Basra; sheikh Thair Ibrahim and his five-year-old niece, killed by a female suicide bomber; 10 people killed by a suicide bomber; and a mother and son killed by gunmen.

  Tuesday March 11 - 90 dead
  Including a couple kidnapped the week before; 16 members of a family returning from a funeral, killed by a roadside bomb; three killed in a US air strike; and 20 people whose bodies were found in a mass grave.

  Wednesday March 12 - 24 dead
  Including a 10-year-old girl killed by US forces; five shot and beheaded at a checkpoint; and three truck drivers killed in a roadside bomb.

  Thursday March 13 - 39 dead
  Including a journalist killed by gunmen; 18 people killed by a car bomb in Baghdad; a 15-year-old girl shot dead by police; and Archbishop Paulos Faraj Rahho.

  Friday March 14 - 15 dead
  Including ex-footballer Munther Khalaf, killed outside his home by a group of armed men; a street sweeper killed by a roadside bomb; an Iraqi interpreter, killed by a suicide bomber; and the son of the chief of al-Kharaj tribes, killed during a raid by joint forces.

  Saturday March 15 - 19 dead
  Including Hussein Awda, killed by gunmen; three brothers; and an Iraqi contractor, Athir Ibrahim.

  Sunday March 16 - 26 dead
  Including two policemen killed in an armed assault and 16 others whose bodies were found, including that of an 11-year-old boy.

  · Jonathan Steele's book Defeat: Why America and Britain Lost Iraq is published by IB Tauris in Britain, and Counterpoint Press in the US.

  · This article was amended on Thursday March 20 2008. In the article above we originally said that Madelyn Hicks was affiliated to the Institute of Development Studies at Sussex University. This was wrong; she is in fact affiliated to the Institute of Psychiatry, King's College London, and was so at the time of her paper's posting. This has been corrected.

• Original here