After further review it appears I have to change the output type of the project to "Windows Runtime Component" instead of "Class Library".
Then it requires some code changes such as no "Task" return type and I couldn't quickly figure out how to make the "Connected" event compatible -- so I deleted it. However, with those deletions it linked with my C++/CX code using nothing but a reference to the project.
EDIT: At runtime it crashes and burns with a null activation factory (3rd argument) to GetActivationFactoryByPCWSTR() when you try to create a new port with "ref new IOPi()". I could not figure out how to make it work at this time.
Perhaps consider making modifications to the libraries to allow linking with C++/CX code. Microsoft provides several of the Win10 IoT examples in both C# and C++/CX languages.