The smaller mass is accelerated upwards which in turn lifts the the heavier mass upward at some rate as well. The heavier mass is accelerating with 'a' while the smaller mass is accelerating with 'am', while 'ar' is the relative acceleration of smaller mass.
Here we shall establish the two equations of motion using the above diagram
T – mg = mam………….(a)
T – Mg = Ma…………(b)
Now we shall calculate the value of acceleration 'a' by equating equations (a) and (b).
so,
Mg – mg = Ma - mam
however, we know that
am = ar - a
so, the above relation becomes
g(M-m) = Ma - m.ar + ma
g(M-m) + m.ar = a(M+m)
a = [g(M-m) + m.ar]/(M+m)
now, here
M = 100 kg
m = 60 kg
ar = (5/4)g
so,
a = (-400 + 750)/160
or acceleration
a = 35/16 m/s2 = 2.1875 m/s2
now, the tension will be calculated as, using equation (b)
T = M(g+ 2.1875)
= 100 x 12.1875
i.e. T = 1218.75 N