Was it really, really rare?
You should let any meat rest for a bit when it comes out of the oven. The juices that have been pushed this way and that by the heat of roasting have a chance to be reabsorbed by the proteins in the meat fibers as it rests. Madeleine Kamman, author of The New Making of a Cook (Canada, UK), says any time you see more than a few tablespoons of meat juice on the carving platter, you have carved the roast too soon after taking it from the oven. She recommends a resting time of 5 to 7 minutes for a roast of 3 pounds and up to 20 minutes for an 8-pound roast.
But more important for the purposes of your question, the heat the roast has accumulated in the oven will continue to cook it as it rests. Depending on the temperature at which you roasted it, the internal temperature of the meat should rise - must rise. Even if you cooked it slowly at 200°F (95°C) for hours, the internal temperature will go up a degree or two during a 10-minute rest. If you roasted the meat at high temperature, above 400°F (205°C), the internal temperature could rise as much as 15°F during the wait.
Did the meat reach a sufficient temperature while it was roasting, or, as many people do with lamb, was it only warmed in the center when you took it from the oven? In that case, the roast could have seemed to cool just because it didn’t have much residual heat when you took it from the oven. There is quite a tendency these days to serve lamb really rare. Do you have an instant-read thermometer? To cook lamb rare, it should reach 135°F (57°C). If you take the lamb out of the oven when it is 130°F in the center, the temperature should go up enough as it rests. (Our government Department of Agriculture, by the way, recommends that you cook lamb to at least medium rare, or 145°F (63°C) in the center).
One thing to consider is that 135°F is not a particularly hot temperature as far as the inside of your mouth is concerned. Harold McGee, coming to the rescue this time in The Curious Cook (Canada, UK) says he can tolerate a beverage at 170°F (77°C) in his mouth (the challenge is getting it past the more sensitive lips), and assumes that other mouths’ comfort levels are around that figure. If that’s the case, meat at 135°F will not seem hot.
One of our staff members is constantly griping about the lukewarm food served in most restaurants these days, but it may just be that for some foods to really taste hot, they would have to be seriously overcooked. So your question may boil down to how hot you want your roast to "taste" versus how well-done you want it. If you rest the roast in the oven or on a hot plate, you will be raising the temperature, but possibly more than you intended.
easy solution is to make sure your gravy is really hot and to serve it before it cools down.