It depends on what you want to pursue, so I would say that it’s okay. If you were to go into engineering, Physics would be a major part of your life and would highly recommend taking it as a lot of colleges would require it. If not, then there’s no compulsion to take the subject.