How was the formation of coal affected by the atmosphere?
It really wasn't.
"Formation of coal" is the complex process of compression and decomposition of organic matter in the earth. For it to occur, it presumes and requires the existence of solid organic material. Further, that material must be covered by subsequent layers of soil and rock.
Thus, there was no impact on the atmosphere due to the formation of the coal. There were few, if any, gaseous compounds released in the process. If the question intends a "sequestration" effect of carbon, it is incorrect to look to coal formation. The exchange and/or sequestration of carbon dioxide already occurred in the organic material, long before it became coal.