Errant intermediate-mass black holes