In the Labeling recipe, you are delete "label classes". (Settings page)
To improve the current solution, an option to additional delete all related annotations from already reviewed images where the deleted class was used.
At the moment, those annotations are kept as "Assign a category". (Review page)
In addition, statistics about the deleted class should be deleted as well. (Overview page)