The quantity asked about: lim_(x->oo)x^alpha(cosx/x)^2. Note that, as stated, this is not an integral, rather an expression. I assume that x is the quantity to take the infinite limit of.
Multiply out:
lim_(x->oo)x^(alpha-2)cos^2x
cos^2x periodically oscillates in the range [0,1] over the real numbers, so its behaviour at infinity is not going to be a potential problem here regarding convergence.
As x->oo, x^(alpha-2) has the following behaviours for differing alpha:
If alpha>2, then alpha-2>0 and x^(alpha-2)->oo.
If alpha=2, then x^(alpha-2)=x^0=1.
If alpha<2, then alpha-2<0 and x^(alpha-2)->0
So
If alpha>2, x^alpha(cosx/x)^2->oo as x->oo.
If alpha=2, x^alpha(cosx/x)^2->cos^2x as x->oo.
If alpha<2, x^alpha(cosx/x)^2->0 as x->oo.