You were on the correct track with your setup on number 8, but when you took the derivative of the y/z term on the right side with respect to time t, you did not use the Quotient Rule to perform the derivative, which rendered the rest of the problem incorrect.
The proper setup is sin theta = y/z as you noted. From there:
(cos theta) (d theta)/(dt) = (z*(dy)/(dt) - y*(dz)/(dt))/(z^2)
It is difficult to read the image included, but I believe the setup involves y = 8, z = 10, (dy)/(dt) = 10, (dz)/(dt) = -6, and cos theta = 8/10, which means you now have:
(cos theta) (d theta)/(dt) = (z*(dy)/(dt) - y*(dz)/(dt))/(z^2)
(8/10) (d theta)/(dt) = (10(10) - 8(-6))/(10^2)
(4/5) (d theta)/(dt) = (100+48)/100
(4/5) (d theta)/(dt) = 148/100
(d theta)/(dt) = 37/25 * 5/4 = 37/20
A quick sanity check on the sign of the result can help catch errors as they are in progress. As the ships continue moving, ship A will get closer to the lighthouse, while ship B will move farther away.
This action will serve to make the angle theta become larger and larger, much like how the angle made by a ladder to the ground increases as you push a ladder up the side of a house. (By the way, this is a common alternative word problem that tests the exact same math in many Calculus classes.)
Your original attempt ended with a negative answer, which would indicate the angle is decreasing - a result that contradicts the nature of the situation. It might not help you find the issue, but it can at least give you a hint that you're on the wrong track somewhere.