  We present a reproducible method to analyze the state of software development
practices in a given scientific domain and apply this method to Geographic
Information Systems (GIS). The analysis is based on grading a set of 30 GIS
products using a template of 56 questions based on 13 software qualities. The
products range in scope and purpose from a complete desktop GIS systems, to
stand-alone tools, to programming libraries/packages. The final ranking of the
products is determined using the Analytic Hierarchy Process (AHP), a
multicriteria decision making method that focuses on relative comparisons
between products, rather than directly measuring qualities. The results reveal
concerns regarding the correctness, maintainability, transparency and
reproducibility of some GIS software. Three recommendations are presented as
feedback to the GIS community: 1) Ensure each project has a requirements
specification document, 2) Provide a wealth of support methods, such as an IRC
(Internet Relay Chat) channel, a Stack Exchange tag for new questions, or
opening the issue tracker for support requests, as well as the more traditional
email-based methods, and, 3) Design product websites for maximum transparency
(of the development process), for open source projects, provide a developer's
guide.
