Is it difficult to determine how pollution affects health? I'm guessing it may be since many factors often contribute to a disease.
For a specific case yes, overall no.
Epidemiology (the study of disease distribution and incidence) and epidemiologists tell us a lot about general trends and that certain things such as a disease or exposure to a certain pollutant are inextricably linked. Just like anything, however, correlation is not causation (just because someone smokes does not mean that they will get cancer we do know for a fact however that smoking does cause cancer and increases one's risk of getting cancer).
Let's have fun with a case theoretical scenario:
A man goes to his doctor, he's coughing non-stop, the coughing even wakes him up at night. The doctor is concerned and has the man go for an MRI, after the MRI and a battery of tests they discover he has lung cancer. The patient just so happens to be employed at the local coal-fired power plant and is a smoker, the exposure to smoke definitely increased his risk of contracting lung cancer (it kills the cells in his lungs) however we are not certain that it was what caused his cancer (he could have been genetically predisposed to it, people can get lung cancer without ever having touched a cigarette).
Something we know for a fact has a direct effect on health is carbon monoxide poisoning. Carbon monoxide competes with oxygen in the body (essentially the red blood cells mistake the carbon monoxide for oxygen) your body can't use carbon monoxide. Since someone's body can't use carbon monoxide and they don't have enough oxygen they eventually pass out and if they are luck enough to be discovered may have severe brain or other bodily tissue damage. Unfortunately most of this data supported by the all too common occurrence of attempted suicides involving carbon monoxide.
I hope this helped!