It took me a while to figure out, but it's not ScaleType.FIT_XY that does the trick but FIT_CENTER.
FIT_XY will stretch both width and height to fit the ImageView.
Remember to use ImageView.setImageDrawable() or setImageBitmap() rather than setBackgroundDrawable().