How do you calculate the mass of a galaxy?
By counting its stars.
The difficulty, and its huge, is in cataloguing the relative size of all the stars within a particular galaxy. The average galaxy has between 100 billion and 300 billion stars. That number in itself makes such a calculation to be at best an educated scientific guess. The amount of error associated with such a calculation could prove to be quite large.
Scientists at the Keck Observatory in Hawaii have dedicated themselves to cataloguing stars within our own galaxy and their attentions are particular focused on those stars closest to the black hole at the center of the Milky Way. The number and size of those stars has proven to be extremely elusive.
Now, when the look at the large galaxy closest to us, Andromeda, the task is easier in the sense that we can see it in its entirety and differentiate stars more easily. The difficulty comes from the distance it is from us. At only 2.5 million light years, it is relatively close. But as the distance a galaxy is from us the level of difficulty increases.
In summary, they guess and will continue to do so until our detection equipment improves dramatically from what we have today.