Firstly we will pass the geometrical objects to a more convenient representation.
#L_1->x+3y=0# to #L-1->p =p_1+lambda_1 vec v_1 #
#L_2->3x + y +8 =0# to #L_2->p=p_2+lambda_2 vec v_2#
#C_1-> x^2+y^2-10x-6y+30=0# to #C_1->norm(p-p_3)=r_3#
Here
#p_1 = (0, 0)#
#p_2 = (-2, -1)#
#vec v_1 = (1, 3)#
#vec v_2 = (3, 1)#
#p_3 = (5, 3)#
#r_3 = 2#
Now, given
#L_1->p=p_1+lambda_1 vec v_1#
#L_2->p=p_2+lambda_2 vec v_2#
#C_1->norm(p-p_3)=r_3#
and
#C->norm(p-p_0) = r#
with #p=(x,y)#
If #C# is tangent to #L_1, L_2# and #C_1# then
#p_0 in L_{12}# where
#L_{12}->p_(12)+lambda_(12) vec v_(12)#
with #p_(12) = L_1 nn L_2# and #vec v_(12) = vec v_1/norm(vec v_1) + vec v_2/norm(vec v_2)#
Other conditions for #p_0# and #r# are
#norm(p_(12)-p_0)^2- << p_(12)-p_0, vec v_1/norm(vec v_1) >>^2 = r^2#
#norm(p_0-p_3) = r + r_3#
but
#p_0=p_(12)+lambda_(12)vec v_(12)#
so
#lambda_(12)^2norm(vec v_(12))^2-lambda_(12)^2 << vec v_(12),vec v_1/norm(vec v_1) >> ^2= r^2#
and
#norm(p_(12)+lambda_(12) vec v_(12) - p_3)^2=(r+r_3)^2#
#norm(p_(12)-p_3)^2+2lambda_(12) << p_(12)-p_3, vec v_(12) >> +lambda_(12)^2 norm(vec v_(12))^2=(r+r_3)^2#
The essential set of equations to obtain the solution is
#{(lambda_(12)^2(norm(vec v_(12))^2 - << vec v_(12),vec v_1/norm(vec v_1) >> ^2)= r^2),
(norm(p_(12)-p_3)^2+2lambda_(12) << p_(12)-p_3, vec v_(12) >> +lambda_(12)^2 norm(vec v_(12))^2=(r+r_3)^2):}#
two equations and two incognitas #r, lambda_(12)#
Solving for #r, lambda_(12)# we obtain
#((lambda_(12) = 1.64171, r = 1.31337),(lambda_(12) = 8.00809, r = 6.40647))#
The attached plot shows the answer in red and the initial elements in black.
