Search for resonances decaying to an anomalous jet and a Higgs boson in proton-proton collisions at sqrt(s) = 13 TeV

This paper presents a search for new physics through the process where a massive particle, X, decays into a Higgs boson and a second particle, Y. The Higgs boson subsequently decays into a bottom quark–antiquark pair, which is reconstructed as a single large-radius jet. The decay products of Y are also assumed to produce a single large-radius jet. The identification of the Y particle is enhanced by computing the anomaly score of its candidate jet using an autoencoder, which measures deviations from typical quark- or gluoninduced jets. This allows a simultaneous search for multiple Y decay scenarios within a single analysis. In the main benchmark process, Y is a scalar particle that decays into a W boson pair. Two other scalar Y decay processes are also considered as benchmarks: decays to a light quark–antiquark pair, and decays to a top quark–antiquark pair. A fourth benchmark process considers Y as a hadronically decaying top quark, arising from the decay of a vector-like quark into a top quark and a Higgs boson. Data recorded by the CMS experiment at a center-of-mass energy of 13 TeV in 2016–2018, corresponding to an integrated luminosity of 138 fb−1, are analyzed. The search covers X masses between 1.4 and 3.0 TeV and Y masses between 90 and 400 GeV, with all simulated signals produced in the narrow-width approximation. No significant excess above the standard model background expectation is observed. The most stringent upper limits to date are placed on benchmark signal cross sections for various masses of X and Y particles.