First note that the function is injective: f(x) = f(y) implies f(f(x)) = f(f(y)) which yields x = y.
We say that a number x is fixed by f if f(x) = x.
Now, if f(a) = b, with a and b unequal, then f(b) = a. That means we can pair up numbers in the set that are not fixed. Since the given set has 5 numbers this means, we can only have an odd number of fixed numbers i.e. 1, 3, or 5.
Case 1: 5 fixed numbers. There is obviously only one such function
Case 2: 3 fixed numbers: We can choose these three numbers in 5C3 = 10 ways. Say they are 1,2,3. Notice that we must have f(4) = 5 and f(5)=4. So we have one such function for each choice of fixed numbers.
So, this case yields 10 functions
Case 3: One fixed number. The fixed number has 5 choices.
Lets say it is 1. Now, if we know, for instance, f(2) = 3, then f(3) = 2. and hence f(4) = 5 and f(5) = 4. That means, the function is decided by choice of f(2), which gives three options for f.
So this case yields 5 X 3 = 15 cases.
Thus, we have a total of 26 choices of f, such that f(f(x))=x