As the radius of the circle is not relevant we can assume we have a circle of unitary radius centered in the origin, whose equation is:
#x^2+y^2=1#
The triangle will then have three vertices #A,B,C# lying on the circle. Because of the circular symmetry we can assume without loss of generality that the side #AB# is parallel to the #y# axis and intersecting the positive #x# axis, so that the coordinates of #A# and #B# are:
#{(x_A = cos(alpha)),(y_A = sin(alpha)):}#
#{(x_B = cos(alpha)),(y_A = -sin(alpha)):}#
where #2alpha = angle (AOB)# so that the length of #AB# is:
#bar(AB) = 2sin(alpha)#
Now, as a first step let's consider for any possible choice of #AB# the triangle #ABC# letting #C# be the third vertex with coordinates:
#{(x_C = cos(theta)),(y_C=sin(theta)):}#
where #theta# is the angle between the #x# axis and #OC# as we generally define in trigonometry.
The height of the triangle is going to be parallel to the #x# axis.
As #x_A# is positive by construction the height is:
#h = x_A -x_C = cosalpha - cos theta#
and the area of the triangle is:
#S = (h bar(AB))/2 = sinalpha(cosalpha-costheta)#
and as #1- <= cos theta <=1# clearly the are is maximum when:
#cos theta = -1#
#theta = pi#
that is when the triangle #ABC# is isosceles and then the area is:
#S(alpha) = sinalpha(cosalpha+1)#
Maximize now this function with #alpha in (0,pi/2)#:
#(dS)/(d alpha) = cosalpha(cos alpha +1) -sin^2 alpha#
#(dS)/(d alpha) = cos^2alpha-sin^2 alpha +cos alpha #
#(dS)/(d alpha) = cos^2alpha -(1-cos^2 alpha)+cos alpha #
#(dS)/(d alpha) = 2cos^2alpha +cos alpha -1#
The critical points are then:
#cos alpha = (-1+-sqrt(1+8))/2 = (-1+-3)/2 = {(-1),(1/2):}#
Excluding #cos alpha = -1# because by construction #cos alpha >0# we have:
#alpha = pi/3#
where:
#(d^2S)/(dalpha^2) = -4sin alpha cos alpha -sin alpha = -sin alpha(1+4 cosalpha)#
#(d^2S)/(dalpha^2)|_(pi/3) = -sqrt3/2(1+4 (1/2)) = -(3sqrt3)/2 < 0#
so that in fact #alpha = pi/3# is a local maximum.
Remembering that:
#angle(AOB) = 2alpha = (2pi)/3#
as the angle #angle(ACB) = 1/2 angle(AOB)# we have:
#angle(ACB) = pi/3#
and as the triangle is isosceles the angles at the base are each:
#angle (BAC) = angle(ABC) = 1/2 (pi -pi/3) = pi/3#
which means the triangle is equilateral.
