A warm building loses energy to the colder outside by conduction through its walls. How quickly the building cools depends on two things about the walls: their thickness and their thermal conductivity.
Thermal conductivity tells you how well a material conducts thermal energy. A material with a high thermal conductivity, such as metal or dense stone, transfers energy quickly. A material with a low thermal conductivity, such as wool or foam, transfers it slowly. Walls made from a material with a low thermal conductivity make the building cool more slowly.
Thickness matters too. Energy has further to travel through a thick wall than a thin one, so the rate of energy transfer is lower. Thicker walls make the building cool more slowly, and thin walls let it cool faster.
The slowest rate of cooling comes from thick walls made of a material with low thermal conductivity. This is why cavity walls are filled with insulating foam.