When you import an image to use as a background image as part of a composed look in SharePoint Online, the image is re-scaled to 1024 x 768 and compressed as low quality jpg. If you have an image which still looks reasonable after these modifications then please continue to use composed looks as the ensures that the page weight is kept to a minimum.
SharePoint provides no configuration around this that I am aware of and even you provide a background image URL in the composed look to a non-compressed image, the image will be compressed and the compressed image will be referenced when the composed look is applied.
The only way to support a high resolution background image is to reference it in CSS, overriding the composed look background image. I have found that the best way to do this is with the following two CSS rules. The second prevents the background being used in dialogs which occasionally don’t use the transparent overlay and can become unusable otherwise.
If the image needs to be changed later, a new image with the same name can uploaded to replace the current one. The issue with this approach is that client browsers may have cached the background image and will continue to use the cached image until the cache expires (could be years, not sure what the default cache is) or the user manually clears their browser cache. In order to force the browser to re-fetch the image a query string value should be appended (or amended, in the current example we are using ‘v=1.0’).
Note that if you are using theme-able CSS you will need to reapply the theme in order to apply changes to the CSS.