Which relay harness do you have?
The morimoto relay has 2 grounds. I grounded one at that fender support over the battery, the other is grounded on one of the bolts under the airbox (no airbox mod so it's easy to get to.)
Make sure you clean the surfaces you're grounded to.
IF you only have one ground, for whatever reason when I had my cheap ass POS DDM relay harness (oh, fuck DDM BTW) my passenger side headlight was always the one that would fuck up on me. Example - When I'd fire the truck up and the AHL's would kick the lights on at the same time, the pass side would flicker and go out EVERY TIME.
I had some flickering issues and I cleaned all the contacts on the harness, fuse, relay, ground, and hit them with a di-electric grease. Problem solved.
What you're describing is a typical starvation issue though. If the ballast hasn't gone out yet, it will since this is super hard on ballasts, and the reason I park with the parking brake on to keep the auto headlight system from firing my HID's while I'm starting the truck.
Make sure everything is making a good solid connection and you should be fine!