  Environmental, social, and governance (ESG) factors are widely adopted as
higher investment return indicators. Accordingly, ongoing efforts are being
made to automate ESG evaluation with language models to extract signals from
massive web text easily. However, recent approaches suffer from a lack of
training data, as rating agencies keep their evaluation metrics confidential.
This paper investigates whether state-of-the-art language models like GPT-4 can
be guided to align with unknown ESG evaluation criteria through strategies such
as prompting, chain-of-thought reasoning, and dynamic in-context learning. We
demonstrate the efficacy of these approaches by ranking 2nd in the Shared-Task
ML-ESG-3 Impact Type track for Korean without updating the model on the
provided training data. We also explore how adjusting prompts impacts the
ability of language models to address financial tasks leveraging smaller models
with openly available weights. We observe longer general pre-training to
correlate with enhanced performance in financial downstream tasks. Our findings
showcase the potential of language models to navigate complex, subjective
evaluation guidelines despite lacking explicit training examples, revealing
opportunities for training-free solutions for financial downstream tasks.
