  With children talking to smart-speakers, smart-phones and even
smart-microwaves daily, it is increasingly important to educate students on how
these agents work-from underlying mechanisms to societal implications.
Researchers are developing tools and curriculum to teach K-12 students broadly
about artificial intelligence (AI); however, few studies have evaluated these
tools with respect to AI-specific learning outcomes, and even fewer have
addressed student learning about AI-based conversational agents. We evaluate
our Conversational Agent Interface for MIT App Inventor and workshop curriculum
with respect to eight AI competencies from the literature. Furthermore, we
analyze teacher (n=9) and student (n=47) feedback from workshops with the
interface and recommend that future work leverages design considerations from
the literature to optimize engagement, collaborates with teachers, and
addresses a range of student abilities through pacing and opportunities for
extension. We found students struggled most with the concepts of AI ethics and
learning, and recommend emphasizing these topics when teaching.
  The appendix, including a demo video, can be found here:
https://gist.github.com/jessvb/1cd959e32415a6ad4389761c49b54bbf
