Proposing f(x) such that
f(x)=(a x + b)/(c x + d) and after doing
f(f(x))=(b (a + d) + (a^2 + b c) x)/(b c + d^2 + c (a + d) x)
and then identifying
{(b (a + d) = -4), ((a^2 + b c) = 1), (b c + d^2 = 4), (c (a + d) = 1):}
and solving for a,b,c,d obtaining
((a = sqrt[1/7 (16 sqrt[2]-13)]),( b = -4 sqrt[
1/7 (4 sqrt[2]-5)]), (c = sqrt[1/7 (4 sqrt[2]-5)]), (d = sqrt[
8/7 + (16 sqrt[2])/7]))
There are four solutions, with two real and two complex. We decided to show only the most simple.
With this solution we have
f(x)=((1 + 2 sqrt[2]) x-4)/((x+2 sqrt[2] + 4)
Note. The relations associated to the coefficients for f, h can be obtained by doing
((a, b),(c, d))^2=((a^2 + b c, a b + b d),(a c + c d, b c + d^2))
For f@f@f the relations are given by
((a, b),(c, d))^3
so generally
f@f @ cdots @ f for a n-composition we have
((a, b),(c, d))^n
so this bilinear transformation follows the composition rules of linear operators.