You are absolutely correct. SWR is a measurement of how well the load matches the feedline impedance, nothing more. Those who try to make something else of it are in error.