if the angle between vector a & b is an actute angle , then the difference a-b

to subtract vector b from vector a we take  the sum of a and -b.So the new angle will be 180-theeta.In this parrticular case the formula becomes whole root of a²+b²+2abcos(180-theeta) or whole root of a²+b²-2abcos theeta where theeta is the acute angle.