Since the notation for the composition (g\circ f)(x)=g(f(x))(g∘f)(x)=g(f(x)) means to first apply ff to xx to get f(x)f(x) and then to apply gg to f(x)f(x) to get g(f(x))g(f(x)), a point xx will be in the domain of g\circ fg∘f if and only if xx is in the domain of ff and f(x)f(x) is in the domain of gg.
For example, let f(x)=\frac{x+3}{x-4}f(x)=x+3x−4 and g(x)=\frac{x-2}{x+5}g(x)=x−2x+5. The number x=4x=4 is not in the domain of ff and is therefore not in the domain of g\circ fg∘f. Is there another number that's not in the domain of g\circ fg∘f? Yes, any value(s) of xx such that f(x)=-5f(x)=−5 is/are not in the domain of g\circ fg∘f since -5−5 is not in the domain of gg. In order for f(x)=-5f(x)=−5, we must have x+3=-5(x-4)=-5x+20x+3=−5(x−4)=−5x+20 or 6x=176x=17 or x=17/6x=176. The domain of g\circ fg∘f is therefore \{x\in R: x!= 4 \mbox{ and } x!=17/6\}{x∈R:x≠4 and x≠176}.
If you find a simplified formula for g\circ fg∘f in such an example, you can be misled in what the correct answer is. In the example above, we can write (g\circ f)(x)=\frac{f(x)-2}{f(x)+5}=\frac{\frac{x+3}{x-4}-2}{\frac{x+3}{x-4}+5}=\frac{x+3-2(x-4)}{x+3+5(x-4)}=\frac{-x+11}{6x-17}(g∘f)(x)=f(x)−2f(x)+5=x+3x−4−2x+3x−4+5=x+3−2(x−4)x+3+5(x−4)=−x+116x−17.
This simplified formula can mislead you into thinking that 17/6176 is the only number not in the domain of g\circ fg∘f, but it's not. The number 44 is also not in the domain as we saw above.
Why does this happen? The reason is that the equality \frac{\frac{x+3}{x-4}-2}{\frac{x+3}{x-4}+5}=\frac{x+3-2(x-4)}{x+3+5(x-4)}x+3x−4−2x+3x−4+5=x+3−2(x−4)x+3+5(x−4) is not true if x=4x=4, because the expression on the left-hand side is undefined there. So, in doing the simplification above, we were implicitly assuming that x!=4x≠4.