Thank you @NickPick528!
I am using a MacPro 2012 with 2 x 3.46 GHz 6-Core Intel Xeon, 64 GB 1333 MHz DDR3 with latest OS X 10.12.5 and latest Nvidia Drivers.
I power the GTX1080ti with an external PSU (bequiet P11 750W). The card definitely gets enough power. And cooling shouldn't be an issue either.
Nvidia is not officially supporting the 1080ti with its Mac Web Drivers: http://www.nvidia.com/download/driverResults.aspx/119315/en-us
It is not listed under supported products.
The support staff of Nvidia just stubbornly says that this card in not supported on Macs and that I should contact Apple about it.