I think Dr Johnson has a point, and agree with him, perhaps this is because I am a direct (or naive) realist. Admittedly, Dr Johnson makes the point with some theatrics and unlike Berkeley doesn't take many pages to do it, but he is still making a philosophical point.
Berkeley's argument is clever, but flawed. The key sign that he is not onto a winner is when he appeals to the existence of a supernatural being that we can never experience, or even form a clear idea of, to support his metaphysics. Being a Bishop will do that to people I will try and pick apart his argument in a slightly more sophisticated manner in a different post, but there is so much wrong with it that I want to marshal my thoughts first.
I think we are entitled to start from the assumption that the world is real, it is an abductive argument perhaps, but it fits so well compared to Berkeley's fantasy.