If I'm reading this right, light loads your fine, but once you do something heavy, it craps out.
Test out your memory here's something you can try
http://www.memtest86.com/#install You'd have to burn a image to a CD though...not sure if you know how/have the software to do so though.
Prime 95
http://www.mersenne.org/freesoft.htm This test your CPU and memory. Download the program, install it (instructions on the site). Start it up, and click options, then click torture test. Use Blend option to test RAM more, choose inplace large FFTs and Small FFTs to test more CPU.
ATI Tool
http://www.techpowerup.com/atitool/ This is a program that will test your video card...it's also a program used for overclocking, but you don't need to mess with that part of the program. Install it, and start the program. Then click scan for artifacts.
Try runnning these over night to see what happens. If Prime errors out, post the error on why it failed. ATI Tool will just start artifacting, if it fails. This means it'll have strange dots, lines, etc. It'll look wrong.
Also, you can check how many volt your power supply is putting out. You'll need a voltmeter for this one. Pull off the side of the computer, while it's running. Find a molex connector, there should be some extra ones in there, they're little plugs/connectors. You'll be able to stick the probe thing from your voltmeter into one of the holes in the connector. The two center ones are grounds and the one with a red wire running to it is 5v and the one with the yellow wire running to it is 12v. 5v a "safe" range for the 5v is ~4.9-5.3. The 12v is ~11.9-12.5ish. Low volts here can cause all kinds of trouble for you.
I build computer/work on them in the spare time and make some money doing it for gas. If you have any other questions, let me know.
Jesse