To multiply these two terms you multiply each individual term in the left parenthesis by each individual term in the right parenthesis.
#(color(red)(5a) + color(red)(3b))(color(blue)(a^3) + color(blue)(2a^2b) - color(blue)(ab^2) + color(blue)(3b^3))# becomes:
#(color(red)(5a) xx color(blue)(a^3)) + (color(red)(5a) xx color(blue)(2a^2b)) - (color(red)(5a) xx color(blue)(ab^2)) + (color(red)(5a) xx color(blue)(3b^3)) + (color(red)(3b) xx color(blue)(a^3)) + (color(red)(3b) xx color(blue)(2a^2b)) - (color(red)(3b) xx color(blue)(ab^2)) + (color(red)(3b) xx color(blue)(3b^3))#
#5a^4 + 10a^3b - 5a^2b^2 + 15ab^3 + 3a^3b + 6a^2b^2 - 3ab^3 + 9b^4#
We can now group and combine like terms:
#5a^4 + 10a^3b + 3a^3b - 5a^2b^2 + 6a^2b^2 + 15ab^3 - 3ab^3 + 9b^4#
#5a^4 + (10 + 3)a^3b + (-5 + 6)a^2b^2 + (15 - 3)ab^3 + 9b^4#
#5a^4 + 13a^3b + 1a^2b^2 + 12ab^3 + 9b^4#
#5a^4 + 13a^3b + a^2b^2 + 12ab^3 + 9b^4#