SIS is a platform that helps nonprofits run better evaluations. It features a secure online workspace and easy-to-use tools that support organizations to plan, conduct, and report on their evaluations.
SIS was developed by LogicalOutcomes, a Canadian nonprofit organization that provides evaluation consulting services to nonprofits and mission-driven organizations around the world. LogicalOutcomes was founded in 2013 by Dr. Gillian Kerr, an organizational psychologist with thirty years of experience in evaluation, program design and policy analysis and by Neil Price, Dean of School of Justice and Community Services at Fleming College and former CivicAction Fellow. Our second President, Martha McGuire, is a fellow of the Canadian Evaluation Society and one of the leading evaluators in Canada. We have a team of analysts and researchers who can handle any evaluation project.
LogicalOutcomes services include:
- Evaluating programs and policies
- Designing monitoring and evaluation systems
- Selecting indicators
- Setting up client management systems
SIS Design Principles
No evaluation is completely objective. They all embed values about who is important (like who should be consulted, who makes decisions and who is a passive consumer of services) and what is important (like what outcomes and outputs are essential and what are optional or unnecessary). Like any evaluation platform, SIS is not value-neutral. LogicalOutcomes has adopted the Principles for Digital Development as design principles for SIS:
- Design With the User
- Understand the Existing Ecosystem
- Design for Scale
- Build for Sustainability
- Be Data Driven
- Use Open Standards, Open Data, Open Source, and Open Innovation
- Reuse and Improve
- Address Privacy & Security
- Be Collaborative
The Principles for Digital Development are beautifully done, and well worth reading. For example, the section on Privacy and Security includes 10 core tenets, introduced by:
Organizations must take measures to minimize collection and to protect confidential information and identities of individuals represented in data sets from unauthorized access and manipulation by third parties. Responsible practices for organizations collecting and using individual data include considering the sensitivities around the data they have collected, being transparent about how data will be collected and used, minimizing the amount of personal identifiable and sensitive information collected, creating and implementing security policies that protect data and uphold individuals’ privacy and dignity, and creating an end-of-life policy for post-project data management.
The Principles acknowledge that privacy, while vital, is just one of the values that should drive digital development, and that we also have to ensure that our tools are driven by user needs, guided by user input, based on collaboration, and support open innovation.
Beyond the Principles for Digital Development, SIS design priorities can be summarized as follows:
- The purpose of evaluation is to improve services for the communities who are intended to benefit. A secondary but important objective is to report to funders, donors and managers because without accountability it is not possible to get adequate resources.
- Evaluation information should be used by service providers in improving services. That means that the data should be relevant and credible to the providers, and they should get results in a timely and accessible manner.
- Service recipients should have an opportunity to communicate their priorities and requests and have them incorporated into planning and decision-making. Priorities often include qualities like responsiveness and courtesy as well as the outcomes of the program.
- The burden of evaluation should be kept as low as possible. That includes respondents’ time for filling out surveys as well as the organization’s time for managing evaluation projects.
- Evaluation tools, measures and indicators should be technically sound, based on evidence, open to criticism and continuously improved.
- Data collection techniques should be multilingual, accessible to people with disabilities, and informed by issues of equity and diversity.
- High quality evaluations should be financially feasible for small nonprofit organizations.
- Evaluation metadata is a common good. Rigorous informatics tools are essential resources for achieving broad social goals and should be built collaboratively and shared freely. Our many inspirations include the United Nations Sustainable Development Goals and the World Health Organization’s International Classification of Health Interventions.
Any specific evaluation will need to balance these principles in its own design. For example, although SIS supports multiple languages (even the name ‘SIS’ works in French, Spanish, Portuguese, Italian, Romanian, etc.), resources are required to translate instruments, test the translations for accuracy and cultural appropriateness and enter the translated text into the system. That means that there will often be trade-offs between accessibility and financial feasibility.
As another example, user engagement in every stage of digital development is required under the Principles of Digital Development but is expensive to do well. Whenever we can, we select measures that have been tested and validated by users.
Once an item has been selected or translated, it will be freely available to everyone. This way, every project systematically builds evaluation resources for the whole sector. We reduce the financial burden of individual evaluation projects by spreading out the costs across the whole platform.
Every survey and interview instrument that collects personal data requires informed consent. Personal data includes opinions and demographic information – anything that can be used to identify someone or link their identity to a response. Consent processes can be detailed and complex, or quick and simple, depending on the evaluation design and the type of risk that is involved. For example, children under 16 should have consent from parents or guardians for sensitive questions, but it may be acceptable to ask older children for simple anonymous feedback without contacting guardians, assuming the raw data is used only to improve services, is deleted shortly afterward, and is not re-used in future studies. Surveys that ask highly stigmatizing information need more elaborate consent questions, which may reduce response rates. We have created a few consent templates that should cover most uses in SIS, but you can write your own if you wish. The SIS consent statements assume that:
- The data is used for quality improvement or program evaluation.
- If the data is used for an academic research study or for some other purpose, that use must be stated in the consent form, and the project should be approved by an Ethics Review Board.
- Responses to the surveys or interviews are completely voluntary. If clients are refused service based on whether they respond to the survey, you should get an external ethics review by, for example, the Community Research Ethics Office.
- The services provided to clients are safe and ethical, and appropriate consent has already been obtained.
- This consent form does not cover consent to the services themselves, only to the survey or interview.
Given those assumptions, your consent questions should include:
- A statement that participation is voluntary and will not affect the services that the respondent receives.
- An explanation about how the data will be used, and who will have access to it.
- A description of how long it will take to answer (unless it’s obvious, like a brief poll).
- A description of the risks, for example, that someone might figure out the identity of the respondent.
- A way to ask for more information, including contact information for a person who can answer questions about privacy.
- Consent forms often include a detailed description about the purpose of the evaluation and contact details for the evaluators as well as organizational staff. They should also, according to best practice in information privacy protection, state how long the data will be retained.
Privacy and security
Even though SIS is mainly funded by nonprofit organizations and their donors, personal data collected by SIS belongs to the individuals who provided the data, and must be handled in accordance with their consent. All of our security practices are based on this simple assumption.
The LogicalOutcomes Information Security Policy is supported by detailed procedures which are reviewed and audited by an external Information Security professional (with current CISM, CISA, CRISC qualifications) acting as Data Protection Officer.
Usability and threat models
If an organization uses portable computers that are not locked down and managed in a central I.T. facility, security processes must be easy to use, as unobtrusive as possible, and seen as valuable by team members. Otherwise they will be ignored. As Bruce Schneier writes, “People often represent the weakest link in the security chain”. Security is part of a sociotechnical system that includes human behaviour as well as technical features. It is not reasonable to expect people to read and comply with long complicated security policies, and people won’t do it. Usable policies mean policies that are embedded into the system and that minimize willpower and ethical decision-making on the part of users. As much as possible we have simplified procedures for users and made them fairly automatic through tightly managed access controls.
Secure systems demand a sophisticated and detailed understanding of risks relating to privacy, data integrity and availability. You can’t create effective protection procedures without knowing what you are protecting against. In the case of nonprofits and health-related services, threats to privacy come from both external actors (e.g., hackers) and internal actors (staff). In fact, the biggest privacy risks in health systems are staff fraud, curiosity and sloppiness, not external hackers. We need to protect the privacy of respondents in situations where either our own contractors or nonprofit staff who subscribe to SIS may share their computers with coworkers, neglect to patch their operating systems, click on infected email links, email sensitive information by mistake because they are in a hurry, have their laptops stolen, and so on. As part of our policy development we analyzed every database and every information asset in LogicalOutcomes to identify where we might be storing personal data, and where potential risks might arise. Not only emails and evaluation spreadsheets, but also IP addresses collected from web visitors, contracts from our consultants, log-in info from third-party services like Skype – everything. Then we radically reduced our exposure to privacy breaches by consolidating most of our work into Microsoft 365 and LimeSurvey, both hosted on Canadian servers.