Let #I=int\ e^{5x}cos 3x\ dx#. Now let #u=e^{5x}# and #dv=cos 3x\ dx# so that #du=5e^{5x}\ dx# and #v=1/3 sin 3x#.
Then #I=1/3 e^{5x}sin 3x-5/3int\ e^{5x}sin 3x\ dx#.
For this next integral, let #u=e^{5x}# and #dv=sin 3x\ dx# so that #du=5e^{5x}# and #v=-1/3 cos 3x#. It follows that
#I=1/3 e^{5x}sin 3x-5/3(-1/3 e^{5x}cos 3x+5/3 int e^{5x}cos 3x\ dx)#
#=1/3 e^{5x}sin 3x+5/9e^{5x}cos 3x-25/9 I#.
Therefore, #34/9 I=1/3 e^{5x}sin 3x+5/9 e^{5x}cos 3x#.
Multiplying both sides by #9/34#, rearranging, and tacking on a #+C# at the end gives
#I=int\ e^{5x}cos 3x\ dx=5/34 e^{5x}cos 3x+3/34 e^{5x}sin 3x+C#.