Windows Azure uses IIS7 so normal

        <customErrors mode="RemoteOnly" redirectMode="ResponseRedirect" >
            <error statusCode="404"   redirect="404.htm"/>
        </customErrors>

wont work. What we have to do is to add

        <httpErrors errorMode="Custom">
            <clear/>
            <error statusCode="404" path="404.htm" responseMode="Redirect"/>
        </httpErrors>

to web.config and we are safe again.