Measurements of the differential jet cross section as a function of the jet mass in dijet events from proton-proton collisions at sqrt(s) = 13 TeV