OK I came up with a fix for Bandicam here after experimenting. Install Better Rayman 3 then set it to 1080p (or whatever your monitors native resolution is) and use this
Rayman 3 Bandicam Fix
Just extract the files to your Rayman 3 install directory overwriting the other files. Let me know how that works out for you
If it doesn't work for you, just let me know what the problem is. The DXWrapper.ini is set to Fullscreen = 1 so if you want to use windowed mode just set that to 0.
It's d3d8 that was causing the problem as expected, this is another DXWrapper version of it that is set to use RibSharks BetterRayman3.dll still but also uses d3d8 to d3d9 and Bandicam detects it
EDIT: being the eejit I am I had forgotten a couple of dll's. I've added them to the zip file now. This also makes the game get picked up as DX9.
EDIT 2: I can confirm it 100% works and records at 60fps for me (I set Bandicam to record at 60fps as a test) and since it shows as DX9 now, peoples GPU software might actually have an effect on the game. Usually it just ignores you. I also feel like it's not as sluggish between exiting and entering the game so maybe this also works better for Windows 10.