3 masses(m,n,o) forming the corners of an equilateral triangle attract each other only according to Newton's Laws.Determine the rotational motion which will leave the relative position of these masses unchanged?

Consider total equivalent attractic force on any one of the masses.It will be directed towards the center of the equilateral triangle having radius a/(3^1/2) taking length of side as  'a' considering simple situation where all masses & forces on them are equal.

Then somehow you rotate them with such velocity so that (mv²)/r=f_eq ,where m=mass , r=a/(3^1/2) , f_eq=f(3^1/2)

or,              mv²=fa.Then their relative positions will not change as for each of them pseudo centrifugal force balnce the attractive forces on them.

Once you make masses & corresponding forces between pairs dissimilar the symmetry will be lost & there will be no simple analysis !!!

Actually you couldn't be able to maintain their relative positions unchanged as center should lie on line of resultant force for each mass which make it impossible to implement even if u've all control over value of v²/r....!!!