You shouldn't have to do that. Make sure you have the latest drivers for the video card. I've got a similar rig; same graphics card, Amd X2 4000+, and I can run 1024x768 with everything maxxed out and not drop below 30 fps or so.
I'd say it's either your drivers, bunch of stuff running in the background, or maybe too much AA/Filtering at too high a resolution.
The card is good, but if it only has 256 Mb of RAM on it like mine, lots of activity in TF2 with everything up all the way @ 1280x1024 slows it down some.