We can think of sqrt(108x^5y^8 as the same as sqrt(108 xx x^5 xx y^8. Then we can can square root each number individually to get 6x^2y^4sqrt(3x)
So the square root of 108 is 6sqrt(3) because sqrt108 can also be written as sqrt(36 xx 3). We know the square root of 36 to be 6, so we write 6 outside of the square root. Thus we have 6sqrt(3) so far.
Then we can do the same for x^5 xx y^8. However, when square rooting powers, we divide the power by the root. For example sqrt(y^8) becomes y^4. For sqrt(x^5), we get x^2sqrt(x) because we can only square root 4 so we have a remainder.
We can then put all of these answers together to get 6x^2y^4sqrt(3x)