How do scientists know how the earth's layers formed?
We can prove the existence of several layers through experiments on the ground. Furthermore we can theorise on what they are made of as well.
For example, the magnetic field on earth is caused by the molten core - which is the only way a magnetic field that could encompass the entire planet could be created.
Furthermore, earthquake tremors travel more slowly through denser rocks, therefore when there is a huge earthquake, a facility in L.A would detect it at a different time from another in London. We can thus calculate the number of layers, their thickness and their material.
We can theorise on how they were formed by seeing their thickness and material. The core is molten due to the intense pressure it is under. Bedrock is incredibly dense due to the same pressure (albeit to a much lesser extent).
If there is another, more specific, question you have on this topic, please comment on this answer - I feel like I may be rambling here...