Why concept of potential is meaningless in case of non conservative fields?

A system with friction is a simple example of a non-conservative force field. Let's assume that in this system, an object has potential energy E. Now let's make the object make a small excursion - up, left, down, right. It returns to the same point, but it had to do work in order to overcome the friction. If that work is W, then its potential energy must have been reduced by the same W, and is now E-W

The implication is that there is no unique potential energy associated with a state of the system (position of the object in the field). And that means that potential energy is a meaningless concept (because a particular state cannot have a unique potential energy associated with it). And that is why nobody does it...