In a bowl, combine the flour, baking powder, salt and nutmeg. Set the flour mixture aside.
Using an electric mixer, beat the butter and sugar for a minute. Add the eggs and beat until smooth, about 1 minute.
Preheat oven to 350 degrees F. Add the flour mixture and milk; beat until smooth, about 1-2 minutes. Fold in orange zest and walnuts.
Spray an 8x8-inch baking dish with cooking spray. Spoon the batter into the baking dish.
Bake on the middle rack of the oven until a toothpick inserted into center of cake comes out clean, about 50-60 minutes.
Remove from oven and let cool before slicing.