  In this paper, we present various pre-training strategies that aid in
im-proving the accuracy of the sentiment classification task. We, at first,
pre-trainlanguage representation models using these strategies and then
fine-tune them onthe downstream task. Experimental results on a time-balanced
tweet evaluation setshow the improvement over the previous technique. We
achieve 76% accuracy forsentiment analysis on Latvian tweets, which is a
substantial improvement over pre-vious work
