There are actually several different ways to interpret that question, depending on what you mean by "vector" and "rotation". But here's a sense that I've often wondered about myself: in introductory physics, the velocity vector is defined as the time derivative of the position vector (relative to some fixed point). Why is the same not true of angular velocity - that is, why isn't there an "angular position vector" which angular velocity can be the derivative of?

As a matter of fact, sometimes there is. Think about this simple case: pick a single, fixed rotation axis and only consider rotations about that axis. (2D rotations, if you prefer to think of it that way) You'd pick a certain orientation to be the "origin," and you could actually define an angular position vector, pointing along the rotational axis, with length equal to the amount of rotation relative to that "origin" orientation.

Now, suppose your object's angular position is changing over time. You can take the derivative of the angular position vector, and hopefully you can see that what you'd get is just the good old angular velocity. No problem there.

But we live in a 3D world (relativity notwithstanding), so what happens when you try to generalize that model to 3 dimensions? That's where you run into problems. As an example, take the object from the last paragraph, which was rotating around one particular axis - say, thez^axis. Now suppose it changes its motion so that it starts rotating around a different axis, maybe thex^axis. How will you represent its orientation now?

You might be tempted to use an "angular position vector" pointing in thez^direction, whose length represents the amount of rotation around thez^axis, and another "angular position vector" pointing in thex^direction, whose length represents the amount of rotation around thex^axis. After all, that works for position. But it doesn't work for angular position. The reason why is thatrotations do not commute, to use the technical lingo. What that means is that if you apply rotation A to an object and follow it up by rotation B which is around a different axis, you get a different result than if you apply rotation B followed by rotation A.

This little issue gets in the way if you try to combine your twox- andz- angular position vectors into one overall angular position vector. Presumably, you would write this overall vector as(θx,0,θz)(the zero might represent the amount of rotation around they^axis). That vector would represent the sum ofθxtimes the unit rotation around thex^direction andθztimes the unit rotation around thez^direction. But it's missing a critical piece of information: which of those rotations was performed first? If you gave that vector to your physicist friend, he would be unable to reproduce the orientation of the object because he doesn't know whether to perform thex-rotation or thez-rotation first. Sure,youmay know that the object was rotated in thez-direction first, but that information needs to be contained in the vector for it to be of any use.

The point of the last paragraph is that there's no way to sensibly create linear combinations of these "angular position vectors." And that pretty much ruins their usefulness, because the ability to be linearly combined is absolutely fundamental to the definition of a vector, and it underlies alotof the analytical techniques we use in physics.

By the way, in this view, the reason matrices work to represent rotations is that matrices offer you an additional operation, multiplication, which you can use to combine them. It happens that matrix multiplication, for certain matrices (3x3 antisymmetric with determinant 1), has the same properties as composing rotations; most notably, it's also noncommutative. Multiplying matrix A by matrix B can give you a different result from multiplying matrix B by matrix A.