Proposing #f(x)# such that
#f(x)=(a x + b)/(c x + d)# and after doing
#f(f(x))=(b (a + d) + (a^2 + b c) x)/(b c + d^2 + c (a + d) x)#
and then identifying
#{(b (a + d) = -4), ((a^2 + b c) = 1), (b c + d^2 = 4), (c (a + d) = 1):}#
and solving for #a,b,c,d# obtaining
#((a = sqrt[1/7 (16 sqrt[2]-13)]),( b = -4 sqrt[
1/7 (4 sqrt[2]-5)]), (c = sqrt[1/7 (4 sqrt[2]-5)]), (d = sqrt[
8/7 + (16 sqrt[2])/7]))#
There are four solutions, with two real and two complex. We decided to show only the most simple.
With this solution we have
#f(x)=((1 + 2 sqrt[2]) x-4)/((x+2 sqrt[2] + 4)#
Note. The relations associated to the coefficients for #f, h# can be obtained by doing
#((a, b),(c, d))^2=((a^2 + b c, a b + b d),(a c + c d, b c + d^2))#
For #f@f@f# the relations are given by
#((a, b),(c, d))^3#
so generally
#f@f @ cdots @ f# for a #n#-composition we have
#((a, b),(c, d))^n#
so this bilinear transformation follows the composition rules of linear operators.