Let #x = f^-1(x)#:
#f(f^-1(x))= 3(f^-1(x))^2+5(f^-1(x))+1#
A property of a function and its inverse is #f(f^-1(x))=x#, therefore, the left side becomes x:
#x= 3(f^-1(x))^2+5(f^-1(x))+1#
Divide both sides by 3:
#1/3x= (f^-1(x))^2+5/3(f^-1(x))+1/3#
Add #a^2-1/3# to both sides:
#1/3x-1/3+a^2= (f^-1(x))^2+5/3(f^-1(x))+a^2#
Using the pattern #(x+a)^2 = x^2+2ax + a^2# we observe that
#2a = 5/3#
#a = 5/6#
#a^2 = 25/36#
#1/3x-12/36+25/36= (f^-1(x)+5/6)^2#
#(f^-1(x)+5/6)^2 = (12x+13)/36#
#f^-1(x) + 5/6 = +-sqrt(12x+13)/6#
#f^-1(x) = +-sqrt(12x+13)/6-5/6#
We know that #g(x) = f^-1(fog(x))#:
#g(x) = +-sqrt(12(3x^4+6x^3-4x^2-7x+3)+13)/6-5/6#
#g(x) = +-sqrt(36x^4+72x^3-48x^2-84x+49)/6-5/6#
#g(x) = +-sqrt((6x^2+6x-7)^2)/6-5/6#
We shall choose the positive value, because #fog(x)# would have a different form if the negative were the correct choice:
#g(x) = x^2+x-7/6-5/6#
#g(x) = x^2+x-2#