If using candied orange peel, make it the day before. Simply melt 1/4 cup of sugar in 1/2 cup of water. Carefully peel an orange ensuring to get just the skin and not the pith (the white stuff). Simmer the orange peel in the simple syrup for about 5-7 minutes. Remove with a slotted spoon and let dry on wax paper for a day or so. Save the simple syrup for the garnish.
Preheat the oven to 350°F. Grease an 8-inch round cake or spring form pan and line the base with parchment paper.
In a mixing bowl sift together the rice flour and almond flour with the baking powder and salt.
In a separate large mixing bowl, beat the egg yolks with an electric whisk until light in color and texture, about one minute. Add the sugar and beat for 2 to 3 minutes until well combined. Continue to whisk and add the olive oil in a slow strain. Then add the vanilla and almond extracts, grated orange zest, orange juice, and the sherry. Fold the dry ingredients into this mixture, until just combined.
Whisk the egg whites in a clean bowl using the electric whisk with clean beaters until firm peaks form. Fold the egg whites into the yolk and flour mixture until well combined. Pour the batter into the prepared cake pan.
Bake in the oven for 30 to 40 minutes until light brown and a toothpick inserted in the middle comes out clean.
Remove the cake and allow to cool in the pan for about 15 minutes, then run a greased knife around the edge of the pan before carefully inverting onto a clean, folded tea towel. Remove the paper from the base and invert onto a cooling rack (this prevents the rack from leaving marks on the top of the cake).
Garnish with flaked almonds and/or candied orange peel and serve plain or with whipped cream, or creme fraiche and berries. If using the candied orange peel, brush the top of the cake with the reserved simple syrup or melted apricot marmalade.