Asynchronous transfers let snapshot copies overlap later GPU computation. Before copying to the host, FDTD CUDA preserves a snapshot in device staging arrays and orders the copy stream with an event:
gpu::copy_on_device(slot.ez.data(), m_ez.data(), bytes, m_stream); gpu::copy_on_device(slot.hx.data(), m_hx.data(), bytes, m_stream); gpu::copy_on_device(slot.hy.data(), m_hy.data(), bytes, m_stream); gpu::copy_on_device(slot.ezx.data(), m_ezx.data(), bytes, m_stream); gpu::copy_on_device(slot.ezy.data(), m_ezy.data(), bytes, m_stream); slot.copied.record(m_stream); slot.copied.hold(m_async->stream);
All five arrays are staged on the simulation stream. Recording slot.copied after those copies and making the transfer stream wait prevents it from reading incomplete data. Subsequent simulation steps can update the original arrays while host copies read the staging slot.
The backend alternates two slots and uses pinned host memory. A completion event is synchronized before invoking the snapshot handler. Select an asynchronous CUDA variant such as v4+async when snapshots are enabled. Overlap reduces waiting opportunities; it does not eliminate transfer cost.
Source: fdtd_tmz_backend_cuda.cu.