Programming an autonomous robot to operate in a real-world environment is extremely challenging. Machine learning techniques are seen as a promising alternative to the reliance on building accurate models of the robot platform and its environment by learning a controller while on the robot platform in the real environment. Currently, the majority of machine learning techniques applied to learning a robot controller use a uniform or pre-defined internal representation provided by a human designer. A uniform representation typically provides poor generalisation for control applications, and a pre-defined representation requires the designer to have an in-depth knowledge of the desired control policy. In this thesis, the approach taken is to reduce the reliance on a human designer by adapting the internal representation, to improve the generalisation over the control policy, during the learning process.