Motion detection works by seeing changes in pixels. You have to provide a stable background that contrasts strongly with the subject. Cockroaches are dark, so your detection will improve with a white background. Constant illumination is also good. High contrast helps detection.
Also, the illumination has to be compatible with the camera. If you have an IR camera, you can illuminate with IR. iF you have a normal camera, continuous visible illumination works but may prevent cockroaches from entering the field of view.
The subject has to consume a significant portion of the field of view. A cockroach from 6ft away won't be noticed. A cockroach at 12 inches will change more pixels and trigger capture.
Lastly, your fake cockroach test must be preceded by a long period of non-activity where nothing happens in the field of view.