  A search for pair production of up-type vector-like quarks ($T$) with a
significant branching ratio into a top quark and either a Standard Model Higgs
boson or a $Z$ boson is presented. The same analysis is also used to search for
four-top-quark production in several new physics scenarios. The search is based
on a dataset of $pp$ collisions at $\sqrt{s}=13$ TeV recorded in 2015 and 2016
with the ATLAS detector at the CERN Large Hadron Collider and corresponds to an
integrated luminosity of 36.1 fb$^{-1}$. Data are analysed in the lepton+jets
final state, characterised by an isolated electron or muon with high transverse
momentum, large missing transverse momentum and multiple jets, as well as the
jets+$E_{T}^{miss}$ final state, characterised by multiple jets and large
missing transverse momentum. The search exploits the high multiplicity of jets
identified as originating from $b$-quarks, and the presence of boosted,
hadronically decaying top quarks and Higgs bosons reconstructed as large-radius
jets, characteristic of signal events. No significant excess above the Standard
Model expectation is observed, and 95% CL upper limits are set on the
production cross sections for the different signal processes considered. These
cross-section limits are used to derive lower limits on the mass of a
vector-like $T$ quark under several branching ratio hypotheses assuming
contributions from $T \rightarrow Wb$, $Zt$, $Ht$ decays. The 95% CL observed
lower limits on the $T$ quark mass range between 0.99 TeV and 1.43 TeV for all
possible values of the branching ratios into the three decay modes considered,
significantly extending the reach beyond that of previous searches.
Additionally, upper limits on anomalous four-top-quark production are set in
the context of an effective field theory model, as well as in an universal
extra dimensions model.
