In a large heavy saucepan, beat eggs and sugar until well blended. Gradually stir in milk. Cook over medium low heat, stirring constantly, until mixture is thick enough to coat a spoon, about 25 minutes. Pour custard into a large bowl. Stir in vanilla and brandy, if using. Chill stirring occasionally. Cover and refrigerate at least 3 hours or overnight. To serve, beat whipping cream until soft peaks form. Gently fold whipped cream into custard. Sprinkle lightly with nutmeg. Keep refrigerated or place bowl on bed of ice. Use within 2 hours.