This project is read-only.

Unhandled Exception: Cudafy.Host.CudafyHostException: CUDA.NET exception: ErrorLaunchTimeOut.

Aug 13, 2012 at 11:15 PM

I get the following error, sometimes, after running a fairly large data set on an underpowered GPU.  I don't get this on my GTX 560TI.  It seems there's an issue when I try to copy the data off the GPU.

 

Unhandled Exception: Cudafy.Host.CudafyHostException: CUDA.NET exception: ErrorL
aunchTimeOut.
   at Cudafy.Host.CudaGPU.HandleCUDAException(CUDAException ex)
   at Cudafy.Host.CudaGPU.DoCopyFromDevice[T](Array devArray, Int32 devOffset, A
rray hostArray, Int32 hostOffset, Int32 count)
   at Cudafy.Host.CudaGPU.DoCopyFromDevice[T](Array devArray, Array hostArray)
   at Cudafy.Host.GPGPU.CopyFromDevice[T](T[] devArray, T[] hostArray)

When I switch to use an async copy I get the following:

Unhandled Exception: Cudafy.Host.CudafyHostException: CUDA.NET exception: ErrorL
aunchTimeOut.
   at Cudafy.Host.CudaGPU.HandleCUDAException(CUDAException ex)
   at Cudafy.Host.CudaGPU.DoCopyFromDeviceAsync[T](DevicePtrEx devArray, Int32 d
evOffset, IntPtr hostArray, Int32 hostOffset, Int32 count, Int32 streamId)
   at Cudafy.Host.CudaGPU.DoCopyFromDeviceAsync[T](Array devArray, Int32 devOffs
et, IntPtr hostArray, Int32 hostOffset, Int32 count, Int32 streamId)
   at Cudafy.Host.GPGPU.CopyFromDeviceAsync[T](T[] devArray, Int32 devOffset, In
tPtr hostArray, Int32 hostOffset, Int32 count, Int32 streamId)

Aug 14, 2012 at 4:17 PM

Basically the NVIDIA driver default will timeout if a kernel takes more than a second or two.  Try to split your kernel into multiple shorter ones.

Nick

Aug 14, 2012 at 4:20 PM

Hi Nick,

You are correct (as usual).  Resolved this with a simply TdrDelay in Windows: http://msdn.microsoft.com/en-us/windows/hardware/gg487368.aspx.  Working fine now.

Thanks,

-Dan

Jul 6, 2013 at 10:58 AM
Edited Jul 6, 2013 at 11:00 AM
Hi guys, im getting the exact same error. As far as im aware ive already modified the TdrDelay to something extremely large like 6 days, see picture below (regedit is on the right). The kernel im trying to run should be about 6 minutes in duration.

Any ideas why this isnt working? Ive already changed the power settings, as the screen saver was initializing and seemed to stop the kernel.

Thanks.

An error has occured and the operation has been terminated.
CUDA.NET exception: ErrorLaunchTimeOut.
Stack Trace for above error:
at Cudafy.Host.CudaGPU.HandleCUDAException(CUDAException ex)
at Cudafy.Host.CudaGPU.DoCopyFromDeviceAsync[T](Array devArray, Int32 devOffset, Array hostArray, Int32 hostOffset, Int32 count, Int32 streamId)
at Cudafy.Host.CudaGPU.DoCopyFromDevice[T](Array devArray, Int32 devOffset, Array hostArray, Int32 hostOffset, Int32 count)
at Cudafy.Host.CudaGPU.DoCopyFromDevice[T](Array devArray, Array hostArray)
at Cudafy.Host.GPGPU.CopyFromDevice[T](T[,] devArray, T[,] hostArray)
at FrazerMann.Profiler.UserInterface.TaskManager.DataRetrieval() in C:.....\TaskManager.cs:line 255
at FrazerMann.Profiler.UserInterface.MainView.btnEvaluation_Click(Object sender, EventArgs e) in C:....\MainView.cs:line 338
Inner exception:

Image
Jul 6, 2013 at 11:10 AM
You may need to reboot. But honestly having a single kernel last more than a few seconds in a non-Tesla, non-dynamic parallelism environment is not a good idea. Break the kernels down to preferably less than 2 seconds each. Also reduce loops to test algorithm for a second or two. As we say time and time again here, reduce your code to the simplest, shortest form that still gives the exception.
Jul 6, 2013 at 11:49 AM
Edited Jul 6, 2013 at 11:50 AM
Hi Nick, thanks for the quick reply. Ive already rebooted (id changed the registry a couple wks back) when i 1st encountered the problem.

So your suggesting having a loop in my c# code which launches small kernels 1 after the other? Im assuming i would need to give the gpu some rest time between calls?

Yeah im planning to simplify the code and then try reducing the amount of data its processing. Was just wondering if there was possibly something else id missed. Will update later today once im done.

Regarding Tesla cards, still want to verify that the tasks im wanting to run will work on the gpu, so would prefer not to get heavily invested atm.

Thanks again for your help.
Jul 6, 2013 at 9:46 PM
What do you mean give the GPU a rest between calls? You mean it should lie down in a dark corner somewhere? No, push it as hard as you can. Overlap I/O and kernels using async copies and launches. Just don't blame me if it all explodes.
Jul 6, 2013 at 11:41 PM
Yeah now you're talking! Show that bitch who's the boss! Whip it into a frothing frenzy of flops and watts! Fill those loops with NOPs!!!
Hmm, I'll go away now. Please, carry on
Jul 7, 2013 at 1:30 AM
This is the 1st time ive experimented with GPU computing + i dont personally know anyone who is proficient with them, so im sorry if some of my questions are a bit stupid. Dont worry, i understand that if i break anything then its my fault.

@pedritolo1 lol :)