The USPTO has published a patented named “Game controller with removable controller accessory” which details a modular game controller for Xbox One that can be customized by replacing its objects like keys.
You can find the patent application here and the application is detailed below:
A user input control device, such as a game controller may be used to provide user input to control an object or a character in a video game or to provide some other form of control. A game controller may include various types of controls that may be configured to be manipulated by a finger to provide different types of user input. Non-limiting examples of such controls may include push buttons, triggers, touch pads, joysticks, paddles, bumpers, and directional pads. The various controls may be manipulated to provide control signals that may be mapped to different operations in a video game.
A game controller includes an electronic input sensor and a mounting platform. The mounting platform includes an accessory-retention feature and a sensor-activation feature. The accessory-retention feature is configured to removably affix a selected removable controller accessory to the mounting platform. The selected removable controller accessory is one of a plurality of differently configured removable controller accessories removably affixable to the mounting platform. The mounting platform is configured to move from a default posture to an activation posture responsive to finger manipulation of the selected removable controller accessory when the selected removable controller accessory is affixed to the mounting platform. The sensor-activation feature interfaces with the electronic input sensor to generate an activate control signal in the activation posture.