  Documentation is an important mechanism for disseminating software
architecture knowledge. Software project teams can employ vastly different
formats for documenting software architecture, from unstructured narratives to
standardized documents. We explored to what extent this documentation format
may matter to newcomers joining a software project and attempting to understand
its architecture. We conducted a controlled questionnaire-based study wherein
we asked 65 participants to answer software architecture understanding
questions using one of two randomly-assigned documentation formats: narrative
essays, and structured documents. We analyzed the factors associated with
answer quality using a Bayesian ordered categorical regression and observed no
significant association between the format of architecture documentation and
performance on architecture understanding tasks. Instead, prior exposure to the
source code of the system was the dominant factor associated with answer
quality. We also observed that answers to questions that require applying and
creating activities were statistically significantly associated with the use of
the system's source code to answer the question, whereas the document format or
level of familiarity with the system were not. Subjective sentiment about the
documentation format was comparable: Although more participants agreed that the
structured document was easier to navigate and use for writing code, this
relation was not statistically significant. We conclude that, in the limited
experimental context studied, our results contradict the hypothesis that the
format of architectural documentation matters. We surface two more important
factors related to effective use of software architecture documentation: prior
familiarity with the source code, and the type of architectural information
sought.
