91 lock problem

I have a 1991 century that is having a problem with one of the locks. It is the driver side door lock. At times it can be extremely difficult to unlock or lock. And then at times it may only be slightly harder then it should be to unlock/lock it. This is weird. Sometimes I can go out to my car, put the key in and try to turn as hard as I can with no success. I will then have to go to open the passenger door (which has no problems) and move the driver lock foward which will be difficult but not as hard as turning the key. Then I will get in and drive and then later try to lock my car and the lock will not lock... every door except the driver door will lock with ease, and I will have to push the button a couple of times to get it to lock. There are then times that I will put my key in the door and it will have the slightest of difficulty locking, and then I will try to lock it from the inside and it will lock right away. I have sprayed WD-40 in the keyhole, on the key, and on each siide of the little door lock thing on the interior but the problem exists. What can I do to fix this? I assume I will have to remove the door panel and do some work, but I would sure prefer not to go anywhere to have things replaced or whatnot and have to spend big bucks... as I just did this for a radiator and idle speed problem. Thanks for any input.