It is assumed that ethical standards are always more important than other standards.

In order to continue enjoying our site, we ask that you confirm your identity as a human. Thank you very much for your cooperation.

Manuel Velasquez, Claire Andre,Thomas Shanks, S.J., and Michael J. Meyer

Moral issues greet us each morning in the newspaper, confront us in the memos on our desks, nag us from our children's soccer fields, and bid us good night on the evening news. We are bombarded daily with questions about the justice of our foreign policy, the morality of medical technologies that can prolong our lives, the rights of the homeless, the fairness of our children's teachers to the diverse students in their classrooms.

Dealing with these moral issues is often perplexing. How, exactly, should we think through an ethical issue? What questions should we ask? What factors should we consider?

The first step in analyzing moral issues is obvious but not always easy: Get the facts. Some moral issues create controversies simply because we do not bother to check the facts. This first step, although obvious, is also among the most important and the most frequently overlooked.

But having the facts is not enough. Facts by themselves only tell us what is; they do not tell us what ought to be. In addition to getting the facts, resolving an ethical issue also requires an appeal to values. Philosophers have developed five different approaches to values to deal with moral issues.

The Utilitarian Approach
Utilitarianism was conceived in the 19th century by Jeremy Bentham and John Stuart Mill to help legislators determine which laws were morally best. Both Bentham and Mill suggested that ethical actions are those that provide the greatest balance of good over evil.

To analyze an issue using the utilitarian approach, we first identify the various courses of action available to us. Second, we ask who will be affected by each action and what benefits or harms will be derived from each. And third, we choose the action that will produce the greatest benefits and the least harm. The ethical action is the one that provides the greatest good for the greatest number.

The Rights Approach
The second important approach to ethics has its roots in the philosophy of the 18th-century thinker Immanuel Kant and others like him, who focused on the individual's right to choose for herself or himself. According to these philosophers, what makes human beings different from mere things is that people have dignity based on their ability to choose freely what they will do with their lives, and they have a fundamental moral right to have these choices respected. People are not objects to be manipulated; it is a violation of human dignity to use people in ways they do not freely choose.

Of course, many different, but related, rights exist besides this basic one. These other rights (an incomplete list below) can be thought of as different aspects of the basic right to be treated as we choose.

• The right to the truth: We have a right to be told the truth and to be informed about matters that significantly affect our choices.

• The right of privacy: We have the right to do, believe, and say whatever we choose in our personal lives so long as we do not violate the rights of others.

• The right not to be injured: We have the right not to be harmed or injured unless we freely and knowingly do something to deserve punishment or we freely and knowingly choose to risk such injuries.

• The right to what is agreed: We have a right to what has been promised by those with whom we have freely entered into a contract or agreement.

In deciding whether an action is moral or immoral using this second approach, then, we must ask, Does the action respect the moral rights of everyone? Actions are wrong to the extent that they violate the rights of individuals; the more serious the violation, the more wrongful the action.

The Fairness or Justice Approach
The fairness or justice approach to ethics has its roots in the teachings of the ancient Greek philosopher Aristotle, who said that "equals should be treated equally and unequals unequally." The basic moral question in this approach is: How fair is an action? Does it treat everyone in the same way, or does it show favoritism and discrimination?

Favoritism gives benefits to some people without a justifiable reason for singling them out; discrimination imposes burdens on people who are no different from those on whom burdens are not imposed. Both favoritism and discrimination are unjust and wrong.

The Common-Good Approach
This approach to ethics assumes a society comprising individuals whose own good is inextricably linked to the good of the community. Community members are bound by the pursuit of common values and goals.

The common good is a notion that originated more than 2,000 years ago in the writings of Plato, Aristotle, and Cicero. More recently, contemporary ethicist John Rawls defined the common good as "certain general conditions that are...equally to everyone's advantage."

In this approach, we focus on ensuring that the social policies, social systems, institutions, and environments on which we depend are beneficial to all. Examples of goods common to all include affordable health care, effective public safety, peace among nations, a just legal system, and an unpolluted environment.

Appeals to the common good urge us to view ourselves as members of the same community, reflecting on broad questions concerning the kind of society we want to become and how we are to achieve that society. While respecting and valuing the freedom of individuals to pursue their own goals, the common-good approach challenges us also to recognize and further those goals we share in common.

The Virtue Approach
The virtue approach to ethics assumes that there are certain ideals toward which we should strive, which provide for the full development of our humanity. These ideals are discovered through thoughtful reflection on what kind of people we have the potential to become.

Virtues are attitudes or character traits that enable us to be and to act in ways that develop our highest potential. They enable us to pursue the ideals we have adopted. Honesty, courage, compassion, generosity, fidelity, integrity, fairness, self-control, and prudence are all examples of virtues.

Virtues are like habits; that is, once acquired, they become characteristic of a person. Moreover, a person who has developed virtues will be naturally disposed to act in ways consistent with moral principles. The virtuous person is the ethical person.

In dealing with an ethical problem using the virtue approach, we might ask, What kind of person should I be? What will promote the development of character within myself and my community?

Ethical Problem Solving
These five approaches suggest that once we have ascertained the facts, we should ask ourselves five questions when trying to resolve a moral issue:

• What benefits and what harms will each course of action produce, and which alternative will lead to the best overall consequences?

• What moral rights do the affected parties have, and which course of action best respects those rights?

• Which course of action treats everyone the same, except where there is a morally justifiable reason not to, and does not show favoritism or discrimination?

• Which course of action advances the common good?

• Which course of action develops moral virtues?

This method, of course, does not provide an automatic solution to moral problems. It is not meant to. The method is merely meant to help identify most of the important ethical considerations. In the end, we must deliberate on moral issues for ourselves, keeping a careful eye on both the facts and on the ethical considerations involved.

This article updates several previous pieces from Issues in Ethics by Manuel Velasquez - Dirksen Professor of Business Ethics at Santa Clara University and former Center director - and Claire Andre, associate Center director. "Thinking Ethically" is based on a framework developed by the authors in collaboration with Center Director Thomas Shanks, S.J., Presidential Professor of Ethics and the Common Good Michael J. Meyer, and others. The framework is used as the basis for many programs and presentations at the Markkula Center for Applied Ethics.

Compiled by Justin Zobel, Department of Computer Science

RMIT University, Melbourne, Australia

, http://www.cs.rmit.edu.au/~jz

May 1999

Summary

"The broad principles that guide research have been long established. Central to these are the maintenance of high ethical standards, and validity and accuracy in the collection and reporting of data ... The processes of research protect the truth. Communication between collaborators; maintenance and reference to records; presentation and discussion of work at scholarly meetings; publication of results, including the important element of peer refereeing; and the possibility that investigations will be repeated or extended by other researchers, all contribute to the intrinsically self-correcting and ethical nature of research." [ Joint NHMRC/AVCC Statement and Guidelines on Research Practice, May 1997]

The process of research - that is, the discovery of scientific truths - relies on the assumption that researchers observe ethical standards. Over the past decade, the major organisations responsible for research and science, such as the AVCC and NHMRC in Australia, have developed guidelines describing these standards. The principal topics of the AVCC guidelines are Data Storage and Retention, Authorship, Publication, Supervisior of Students, Disclosure of Potential Conflicts of Interest, and Research Misconduct.

The NHMRC/AVCC guidelines are written for the general scientific community. In a series of discussions of research ethics with computer scientists, difficulties with the guidelines emerged. In particular, amplification or clarification was needed because in some cases it is not clear how the guidelines should be applied in computer science. This document, then, concerns the ethical conduct of research in computer science. It augments the NHMRC/AVCC guidelines: each section has a brief restatement of the major points in the original guidelines (which every researcher should read in full), focusing on issues of particular relevance to research practice in computer science, followed by commentary based on discussions with practicing computer scientists.

The principal recommendations in this document are that:

• Research workers be committed to high standards of professional conduct.
• Research workers should only participate in work which conforms to accepted ethical standards and which they are competent to perform.
• Confidentiality must be observed.
• Data - the outcomes and results of research - must be recorded in a durable and appropriately referenced form.
• The minimum requirement for authorship of a publication is participation in conceiving, executing or interpreting a significant part of the outcomes of the research reported.
• Supervisors should ensure that the work submitted by research students is the work of the student.
• Supervisors must not publish a student's work without giving appropriate credit (usually authorship) to the student.
• Researchers should not referee a paper or examine a thesis where there is a real or perceived conflict of interest, or where there is some reasonable likelihood that it will be difficult for the referee to maintain objectivity.

General considerations

Three elements the NHMRC/AVCC opening statement are of particular relevance to practising researchers in computer science:

• It is a basic assumption of institutions conducting research that their staff members are committed to high standards of professional conduct. Research workers have a duty to ensure that their work enhances the good name of the institution and the profession to which they belong.
• Research workers should only participate in work which conforms to accepted ethical standards and which they are competent to perform. When in doubt they should seek assistance with their research from their colleagues or peers. Debate on, and criticism of, research work are essential parts of the research process.
• If data of a confidential nature is obtained ... confidentiality must be observed and researchers must not use such information for their own personal advantage or that of a third party ... In general, however, research results and methods should be open to scrutiny by colleagues within the institution and, through appropriate publication, by the profession at large.

Data storage and retention

• 2.1 Data (including electronic data) must be recorded in a durable and appropriately referenced form.
• 2.2 The department or research unit must establish procedures for the retention of data and for the keeping of records of data held.
• 2.3 Data must be held for sufficient time to allow reference ... the minimum period for retention is at least 5 years from the date of publication.
• 2.4 Wherever possible, original data must be retained in the department or research unit in which they were generated ... Retention solely by the individual researcher provides little protection to the researcher or the institution in the event of an allegation of falsification of data.
• 2.5 Data related to publications must be available for discussion with other researchers.

The guidelines also consider issues relating to data available through confidential sources.

Commentary

In this context, data does not necessarily include the subject of an experiment - by analogy, a chemist is not required to keep test-tubes of chemicals once the work is complete - but rather the outcomes, results, and conclusions of research. What should be held is not always clear, but at a minimum researchers should keep programs and scripts used to conduct experiments, and if possible should also keep both input and output; but it is recognised that inputs often cannot be retained, because of their size or transient nature. Output as processed for analysis should always be kept.

In other disciplines it is expected that scientists keep careful, dated notebooks describing their experiments, ideas, intentions, methodology, results, and so on. Computer scientists should also keep such notebooks, which can function as effective reminders of the differences between version of software, of parameters used and data tested, and of failed runs. In conjunction with centralised backups (which are evidence that software, inputs, and outputs were present in the system at certain dates), such notebooks - even if held privately - are an adequate mechanism for retaining data. Notebooks should be bound (that is, not bundles of loose pages), dated, and should clearly identify software, data, versions, and outcomes. Software used to produce recorded outcomes should be retained if possible.

Data related to publications should be made available for discussion with other researchers; but any requirements of confidentiality in relation to data should be observed.

Authorship

• 3.1 Each institution must establish a written policy on the criteria for authorship of a research output. Minimum requirement for authorship should accord with the 'Vancouver Protocol'. Authorship is substantial participation, where all the following conditions are met: a) conception and design, or analysis and interpretation of data; and b) drafting the article or revising it critically for important intellectual content; and c) final approval of the version to be published. Participation solely in the acquisition of funding or the collection of data does not justify authorship. General supervision of the research group is not sufficient for authorship. Any part of an article critical to its main conclusion must be the responsibility of at least one author. An author's role in a research output must be sufficient for that person to take public responsibility for at least that part of the output in that person's area of expertise. No person who is an author, consistent with this definition, must be excluded as a author without their permission in writing.
• 3.3 When there is more than one co-author of a research output, one co-author (by agreement amongst the authors) should be nominated as executive author for the whole research output, and should take responsibility for record-keeping regarding the research output.
• 3.4 Where the research is published, including electronically, all co-authors of a publication must acknowledge their authorship in writing in terms of, at least, the minimum acceptable definition at 3.1, above. This signed statement of authorship must specify that the signatories are the only authors according to this definition. It must state that the signatories have seen the version of the paper submitted for publication.
• 3.5 The written acknowledgment of authorship must be placed on file in the department or unit of the executive author.
• 3.7 The authors must ensure that others who have contributed to the work are recognised in the research output. Courtesy demands that individuals and organisations providing facilities should also be acknowledged.

Commentary

Thus the minimum requirement for authorship of a publication is participation in conceiving, executing or interpreting a significant part of the outcomes of the research reported. Note that this does not include subsidiary tasks such as implementation conducted under the direction of a researcher. Honorary authorship on any basis - seniority, "tit-for-tat", generosity, or coercion - is unacceptable.

There is no simple rule that establishes how much contribution to a paper is enough to merit authorship. Providing comments on a draft or two is almost certainly not sufficient; but conception in detail of the original idea almost certainly is.

On the one hand, a researcher should always be given an opportunity to be included as an author if their contribution has added to the quality of the paper sufficiently for it to be accepted by a more prestigious journal or conference than would otherwise have been the case. On the other hand, involvement in an extended project does not guarantee authorship on every paper that is an outcome of the project; and in most circumstances a researcher will have partipated to some degree in every part of a publication of which they are an author, from the commencement or conception of the research to completion of the publication itself. A researcher who has only minimally met requirements for authorship should consider choosing to be acknowledged instead, particularly when the researcher is a supervisor of the other authors. Co-authorship is a consequence of having made a genuine contribution to the intellectual property embodied in a paper, and simply being a student's supervisor is not sufficient to merit co-authorship.

A related issue is of author order, since many readers will assume that the first author was the primary contributor. A researcher who is clearly the primary contributor should be listed first. Where there is no obvious first author, possible approaches to ordering include: alphabetical or reverse alphabetical, with an explanatory footnote; a reversal or rotation of the order used on a previous paper by the same authors; choosing the first author based on considerations such as the value to each invididual, so that for example in a paper jointly written by a student and supervisor the student should be listed first; or if all else fails toss a coin. The order of authors should always be explicitly discussed prior to submission, and is the joint decision of all of the authors.

A publication should contain due recognition of the contributions made by all participants in the relevant research. The work of research students, research assistants, and the assistance or advice of colleagues, should be properly acknowledged. Acknowledged persons should, if the presence of their name could be interpreted as their endorsing the contents of the paper, be given the opportunity to read the paper prior to submission.

Any groups or organisations that funded or contributed significantly to the research should be acknowledged. Where, for example, the address of an author is not the institution that employed them while they conducted the research, the institution should be acknowledged explicitly.

Publication

• 4.1 Publication of multiple papers based on the same set(s) or subset(s) of data is not acceptable, except where there is full cross-referencing within the papers.
• 4.2 An author who submits substantially similar work to more than one publisher must disclose this to the publishers at the time of submission.
• 4.3 As a general principle research findings should not be reported in the public media before they have been reported to a research audience of experts in the field of research.
• 4.6 Publications must include information on the sources of financial support for the research. Financial sponsorship that carries an embargo on such naming of a sponsor should be avoided.
• 4.7 Deliberate inclusion of inaccurate or misleading information relating to research activity in curriculum vitae, grant applications, job applications or public statements, or the failure to provide relevant information, is a form of research misconduct.
• 4.8 All reasonable steps must be taken to ensure that published reports, statistics and public statements about research activities and performance are complete, accurate and unambiguous.

Commentary

When presenting research results, researchers should:

• Ensure that results are accurate;
• Describe any limitations of experiments and any considerations that may have affected their outcome;
• Provide enough detail to allow other researchers to reproduce experiments, including (where relevant) method, sources of input data, and so on;
• Report the work of others fairly and accurately;
• Report relevant negative results as well as positive results;
• Not deliberately make misleading statements of any kind, or imply falsehoods.

Publication of multiple papers based on the same results is improper unless there is full cross-referencing. It should be made unambiguously clear to what degree the results are new.

Other forms of self-plagiarism are the subject of debate. The conflicting positions are represented by two articles: "Copyrights and author responsibilities", H.S. Stone, IEEE Computer, 25:12, December 1992, pp. 46-51; and "Self-plagiarism or fair use?", by P. Samuelson, Communications of the ACM, 37:8, August 1994, pp. 21-25. The central issue is of whether it is acceptable for authors to reuse their text, even text describing background or principles (in contrast to text describing new research). The arguments are with regard to both copyright and ethics.

Researchers should be aware that some referees consider reuse of text to be unethical, on the grounds that a contribution can only be made once; even discussion of background or principles can present a new view of a topic that is itself a contribution to understanding of research; and the quality of such discussion is a factor in the decision as to whether the paper should be accepted. Moreover, any significant reuse of text by a researcher that is not explicitly referenced as a quotation can be regarded as plagiarism once the text has already appeared in a copyright form such as a journal or a conference proceedings. But the issue of how much text constitutes "significant" is open. Stone argues from copyright that reuse of two paragraphs or more is unacceptable; Samuelson argues from an informal survey of colleagues that, without specific permission, 30% of the paper being reused prose is "a grey area" and would recommend less reuse than that. (In contrast, an informal survey of my colleagues suggested that perhaps 10% - one page of a typical conference paper - was a limit and that they would be uncomfortable recommending acceptance of a paper with more than that level of reuse, an exception being a journal submission that was a substantial expansion of a preliminary conference paper.) Researchers should not reuse text to a degree that is likely to be interpreted by a referee as plagiarism. Note that the text of a paper is owned collectively by the authors, regardless of who actually wrote what.

Any plagiarism of the work of others is unacceptable. Note that plagiarism is not limited to copying of material from published papers, but also includes material in electronic form, such as technical articles made available on Internet or comments or suggestions made in e-mail.

Gender-inclusive language - that is, language that does not specify gender unnecessarily - should be used in all writing, including research publications.

Supervision

• 5.3 Each trainee should be provided with written material on applicable government and institutional guidelines for the conduct of research.
• 5.4 Supervisors should be obliged to provide guidance in all matters of good research practice.
• 5.5 The supervisor must ensure, as far as possible, the validity of research data obtained by a student under his/her supervision.

Commentary

When a researcher supervises a postgraduate or honours student, the student undertakes a research program under the supervisor's direction, culminating in a written report that is assessed. Often material in the report is the product of joint research, and must be explicitly acknowledged as such. However, if the research or report is substantially the work of the supervisor, the student is in breach of University regulations if the work is submitted as their own. As far as possible, supervisors should ensure that the work submitted by research students is the work of the student, and that the research is valid.

It is improper for a supervisor to publish a student's work without giving appropriate credit (usually authorship) to the student. Where a supervisor is enrolled in a research degree, the supervised project must be distinct from the supervisor's research. That is, the research undertaken by, for example, an honours student should not subsequently be incorporated into the supervisor's assessed work.

Published work that is generated during the course of a postgraduate degree is often jointly attributed to both student and supervisor. It is usually the case that the student has undertaken the bulk of the task: capturing some idea in text, conducting experiments, and creating the paper that that describes the idea. However, it is often the case that the paper would not have existed without ongoing input from the supervisor, and that the conception and initial development of the idea is due to the supervisor. In these cases student and supervisor should both claim authorship. This practice of shared authorship does not diminish the student's final work, and it helps to prevent the supervisor from limiting their responsibility to the student and to the quality of the research.

There are also cases in which sole authorship by a student is appropriate, such as during the latter stages of a candidacy when the student can be expected to be in contact with other experts in the field, and is by then expert. Good PhD candidates should be able to prove themselves towards the end of their candidacy by undertaking research largely unassisted. Supervisors should encourage each student to write a paper of which the student is sole author - that is, to develop an idea, conduct experiments and write a paper - and should give him/her the freedom to do so. In such cases the supervisor should be consulted prior to submission.

In no circumstances should a supervisor use their position to force a student to include him/her as an author; indeed, a supervisor who has only minimally met the requirements for authorship should consider choosing instead to be acknowledged. Nor should a supervisor assume that he/she is automatically an author of a student's paper - authorship should always be explicitly discussed. Disputes over authorship should be raised at the earliest opportunity, and taken to the research coordinator or the head of department if they cannot be resolved amicably.

A supervisor, and in particular the senior supervisor in a supervisory team, is responsible for ensuring that the student has reasonable access to resources necessary for their project and that the academic aspects of the program proceed steadily and at a sufficient rate.

Disclosure of potential conflicts of interest

• 6.2 Institutions must formulate and advertise to their staff policies and procedures regarding appropriate disclosure of affiliation with, or financial involvement in, any organisation or entity with a direct interest in the subject matter or materials of researchers ... Such disclosure should cover any situation in which the conflict of interest may, or may be perceived to, affect any decision regarding other people.
• 6.3 The procedures should require disclosure to editors of journals, to the readers of published work, and to external bodies from which funds are sought.
• 6.4 Researchers have an obligation to disclose at the time of reporting or proposing research (for example, in a grant application), any conflict of interest which has the potential to influence research and investigations, publication and media reports, grant applications, applications for appointment and promotion.

Research misconduct

Misconduct or Scientific misconduct is taken here to mean fabrication, falsification, plagiarism, or other practices that seriously deviate from those that are commonly accepted within the scientific community for proposing, conducting, or reporting research. It includes the misleading ascription of authorship including the listing of authors without their permission, attributing work to others who have not in fact contributed to the research, and the lack of appropriate acknowledgment of work primarily produced by a research student/trainee or associate. It does not include honest errors or honest differences in interpretation or judgements of data.

Examples of research misconduct include but are not limited to the following :

Misappropriation: A researcher or reviewer shall not intentionally or recklessly

• a. plagiarize, which shall be understood to mean the presentation of the documented words or ideas of another as his or her own, without attribution appropriate for the medium of presentation;
• b. make use of any information in breach of any duty of confidentiality associated with the review of any manuscript or grant application;
• c. intentionally omit reference to the relevant published work of others for the purpose of inferring personal discovery of new information.

Interference : A researcher or reviewer shall not intentionally and without authorization take or sequester or materially damage any research-related property of another, including without limitation the apparatus, reagents, biological materials, writings, data, hardware, software, or any other substance or device used or produced in the conduct of research.

Misrepresentation: A researcher or reviewer shall not with intent to deceive, or in reckless disregard for the truth,

• a. state or present a material or significant falsehood; or
• b. omit a fact so that what is stated or presented as a whole states or presents a material or significant falsehood.

Commentary

Thus misconduct in research includes:

• Fabrication of data and claiming results where none have been obtained;
• Falsification of data including changing of records;
• Plagiarism, including the direct copying of textual material, the use of other people's data without acknowledgement, and the use of ideas from other people without adequate attribution;
• Misleading ascription of authorship including the listing of authors without their permission, attributing work to others who have not in fact contributed to the research, and the lack of appropriate acknowledgement of work primarily produced by a research student or assistant;
• Any practice that seriously deviates from those commonly accepted within the research community;
• Intentional infringement of published Codes of Conduct.

Misconduct does not include honest errors or honest differences in interpretation or judgements of data; but significant errors should be corrected as promptly as possible. Accusations of misconduct or unethical behaviour are serious matters and should only be made after careful consideration.

Refereeing and examination

The guidelines in this section are adapted and extended from a resolution passed by the Transactions Advisory Committee of the IEEE Computer Society.

Researchers should not referee a paper or examine a thesis where there is a real or perceived conflict of interest, or where there is some reasonable likelihood that it will be difficult for the referee to maintain objectivity.

Examples are:

• A paper (or thesis) by an author with whom the referee has recently been a co-author;
• A paper by an author at the same department as the referee, or in a closely related department either at the same university or at another university with which the referee is closely associated;
• A paper by an author who was a recent student or supervisor of the referee;
• A paper by an author with whom the referee has had recent close interaction, including not only personal relationships but also antagonistic interactions such as competition for an appointment.

In such cases, the referee should notify the editor that an alternative referee should be sought. Where there is no alternative referee, the editor may request that the referee evaluate the work despite the conflict. In such cases, assuming that the conflict is not so severe as to prohibit objectivity, the referee may evaluate the paper but the referee's report should carry an appropriate caveat.

Referees should respect the confidential nature of the papers they referee. Such papers should not be shown to colleagues, except as part of the refereeing process; they should not be used as a basis for the referee's own research or for the referee's personal gain; and referees should not indicate whose papers they have been reviewing or the outcome of the review process. It will however sometimes be the case that the authors have already made the work publicly available, for example on Internet, in which case the publicly available version does not have to be treated in a confidential manner.

When a referee recommends acceptance of a paper, the referee is assuring the technical content, originality, and proper credit to previous work to the best of the referee's ability to judge these aspects; a referee should not recommend acceptance if the paper is not of adequate standard in some respect. The onus is on the referee to take sufficient care to fully evaluate the paper. Referees who are not able to assure the quality of the paper should not recommend acceptance without an appropriate caveat.

Referees should make every endeavour to complete reviews in a timely and professional manner. Reviews should be constructive. For example, in the review process it may be possible to correct a proof or generalise a result, and thus strengthen the paper anonymously on behalf of the author. Rejections should clearly explain, not only the faults of the paper, but a process that the author might use to produce a more acceptable outcome. Even in the case of a paper that a referee believes to be totally without contribution, it is helpful to explain how the author might verify for themselves whether this evaluation is correct. Every paper, no matter how weak, should have a careful and thorough review. Researchers should only decline to referee a paper with good reason.

Acknowledgements

Many people contributed to the content of this document. In particular I am grateful to Alistair Moffat, Lin Padgham, and Ross Wilkinson.

Justin Zobel