Studies of jet mass in dijet and W/Z+jet events