I think you should be the one to just kind of tell him that we want to leave and that maybe you want to leave and then we'll all go together.
Mislim da bi mu ti trebao reći da želimo otići ili da ti želiš otići, pa odemo zajedno.