Carel van Schaik and Karin Isler explain:

Perhaps the most important limitation of brain size is the fact that brain tissue is metabolically very intensive and it, therefore, consumes a lot of energy. The heart, the liver and the kidneys need a similar amount of energy per gram of tissue. However, their size is much more determined by body weight than brain size i.e. it varies much less between species. Other organs, bones, muscles, skin etc. consume per gramme a fraction of the energy the brain consumes (…)

That means that it is more difficult to achieve a certain increase in brain size through natural selection than it is, for example, to achieve an equivalent increase in muscle mass or general body size.

Another difficulty is that an organism cannot temporarily shut down its brain to save energy because the brain needs exactly the same amount of energy at rest.
(Carel van Schaik, Karin Isler 2010: Gehirne, Le­bens­läufe und die Evolu­tion des Menschen, in: Fischer/ Wiegandt: Evolution, pp. 142-169, pp. 153s)