1996 Plymouth Neon- Electrical- Aftermarket stereo stopped working while listening?

Ok, I was listening to my stereo in my 1996 Neon today after I made some modifications to my speaker box for my subs. I reinstalled the box and speakers properly, everything is hooked up properly, no wires touching each other, etc etc etc. My stereo played fine for several minutes when suddenly in the middle of a song it cut out like it shut off, then it came back on. Then a few seconds later it shut off completely and has not came back on since. I then checked all of the wiring in my aftermarket wiring harness and all of my connections and they seem to be fairly solid. The wierd thing is, when I have the ground disconnected from the stereo, but it is plugged into the harness, my amplifier comes on. However, if you hook up the ground the amplifier shuts off and when the key is in the off position I can eject and put in CDs, but the CD player does not work any other time. I think it might be the wiring in my car, or my stereo is bad. Any advice or ways I could test and find out?