I just had the same problem and rebuilt both sides. It sounds to me like whomever rebuilt this for you missed a step or didn't do a good job. Several key items
The bearing load must be correct, to much slop will cause the axle seals to fail. (to tight can be problematic as well, new set of problems)
You must put an rtv sealant on darn near every compnent, shims, bearing cage, backing plate, axle spline, etc.
Brake drum should have nothing to do with your issue.
Here's a link to a great web-site that goes into detail on your rear seals.
Smith's Old Ford Tractors
Look under repairs and maintenance
Good luck!