  Frustration-induced strong quantum fluctuations accompanied by spin-orbit
coupling and crystal electric field can give rise to rich and diverse magnetic
phenomena associated with unconventional low-energy excitations in rare-earth
based quantum magnets. Herein, we present crystal structure, magnetic
susceptibility, specific heat, muon spin relaxation(muSR), and electron spin
resonance (ESR) studies on the polycrystalline samples of Ba6Yb2Ti4O17 in which
Yb3+ ions constitute a perfect triangular lattice in ab-plane without
detectable anti-site disorder between atomic sites. The Curie-Weiss fit of
low-temperature magnetic susceptibility data suggest the spin-orbit entangled
Jeff = 1/2 degrees of freedom of Yb3+ spin with weak antiferromagnetic exchange
interactions in the Kramers doublet ground state. The zero-field specific heat
data reveal the presence of long-range magnetic order at TN = 77 mK which is
suppressed in a magnetic field 1 T. The broad maximum in specific heat is
attributed to the Schottky anomaly implying the Zeeman splitting of the Kramers
doublet ground state. The ESR measurements suggest the presence of anisotropic
exchange interaction between the moments of Yb3+ spins and the well separated
Kramers doublet state. muSR experiments reveal a fluctuating state of Yb3+
spins in the temperature range 0.1 K-100 K owing to depopulation of crystal
electric field levels, which suggests that the Kramers doublets are well
separated consistent with thermodynamic and ESR results. In addition to the
intraplane nearest-neighbor superexchange interaction, the interplane exchange
interaction and anisotropy are expected to stabilize the long-range ordered
state in this triangular lattice antiferromagnet.
