Computer Science 690AB, Winter '26
Course Diary:

Research Methods in Computer Science

---

Copyright 2026 by H.T. Wareham
All rights reserved

---

Week 1, Week 2, (Course Talk #1 Notes),
Week 3, Week 4, Week 5, Week 6, Week 7, Week 8, Week 9, (Course Talk #2 Notes),
Week 10, Week 11, Week 12, (end of diary)

Wednesday, January 7 (Lecture #1) (FS)

• Introduction to CS 690AB: Content and Logistics (Slides)

Friday, January 9 (Lecture #2) (FS)

• Giving Talks for Fun and Enlightenment (Slides)
  • Example talks (Talk #1 slides, Talk #2 slides)

Monday, January 12 (Lecture #3) (FS)

• Giving Talks for Fun and Enlightenment (Cont'd) (Slides)
• Example Talks:
  • Talk #0 (non-specialist audience / 50 minutes)
  • Talk #2 (non-specialist audience / 20 minutes)
  • Talk #3 (specialist audience / 25 minutes)
  • Talk #4 (specialist audience / 16 minutes)

Monday, January 12

• Course Talk #1 Notes
  I have done up a schedule for the first round of student course talks. These talks will run in our lecture slots starting on Monday, January 26, and concluding on Monday, February 2. The talks are ordered in a random fashion; if you need to switch your talk slot, please arrange to switch with other people yourself and, when this is done, tell me the changes so I can adjust the on-line schedule accordingly.

  Each talk is alloted 20 minutes with 5 minutes afterwards for questions. You may use the in-class blackboard and AV facilities as you please during your talks. You may also structure your talks as you please; all of these talks will get full marks if given, so just aim to give a fun talk about something you love (whose content you discuss with me beforehand so we do not get sued).

  May each of you please attend all talks and have fun with your own.

Wednesday, January 14 (Lecture #4) (FS)

• Typesetting Documents: Fun with LaTeX
  • Introduction: What is LaTeX?
    • TeX is a document typesetting system that was created by Donald Knuth in the late 1970s to "simplify" the creation of his multi-volume series The Art of Computer Programming.
    • LaTeX ("Lamport TeX") is a simplified document typesetting system layered on top of TeX that was created by Leslie Lamport in the early 1980s.
    • TeX and LaTeX do typesetting using text markup commands (unlike "What You See Is What You Get" (WYSIWYG) document typesetting systems like MS Word).
    • LaTeX is the standard system used in scientific publishing for conference and journal papers as well as books.
  • Basic LaTeX Document File Format (Sections 2, C.2, and C.5, LL)
    • By convention, all LaTeX document files have extension .tex, e.g., blah.tex.
    • LaTeX file structure:

      [document-start command]
      [preamble]
      \begin{document}
      [document body]
      \end{document}
      	      

    • The document-start command is a single line of the form \documentstyle[OPT]{TYP} or \documentclass[OPT]{TYP}, where OPT is a comma-separated list of typesetting options and TYP is a one-word document type. For the moment, we will focus on document-options list 12pt and document-type article.
    • The preamble contains various commands adjusting and/or augmenting the default typesetting settings defined by the document-type.
    • The document body contains all text, tables, figures, bibliography, etc that appear as the typeset document. All typesetting commands start with a left-slash (\); otherwise, all characters in the document body are treated as plain text to by typeset.
  • Sectioning Commands and Footnotes (Sections 2.2.1, 2.2.3, C.3.3, and C.4, LL)
    • There are four basic levels of section commands denoted by \section{TITLE}, \subsection{TITLE}, \subsubsection{TITLE}, and \subsubsubsection{TITLE}. These are automatically numbered. If section numbers are not wanted, place a star before the left curly brace in the command, e.g., \section*{TITLE}.
    • After command \appendix, all section-commands create appendices and their appendix-subsections.
    • A footnote with text FBODY that is marked by an automatically-numbered superscript is created using command \footnote{FBODY}.
  • Converting LaTeX Document Files to PDFs (Take I)
    • There are two ways to convert a LaTeX document file to a PDF file:
      • Convert the LaTeX file blah.tex to a DVI file via the command latex blah and convert the DVI file to a PDF via dvipdf blah.
      • Convert the LaTeX file blah.tex to a PDF directly via the command pdflatex blah.
  • Example: A sample Latex document v1.0 [example1.tex, example1.pdf, example1.script]
  • Text and Fonts (Sections 3.1, 6.7, and C.15, LL)
    • Each (possibly multi-line) block of text separated by one or more blank lines is treated as a paragraph. Don't worry about making paragraph-text margins even -- LaTeX does that for you automatically (though not always correctly; we'll talk more about how to fix such screwups later).
    • To override indenting at the beginning of a paragraph, use command \noindent.
    • To change the font or text-size of a block of text, enclose that block in curly braces and put the appropriate commands just inside the left curly brace.
      • There are seven font-sizes you can access easily -- \footnotesize, \small, \normalsize, \large, \Large, \LARGE, and \huge.
      • There are five font-types you can access easily -- \rm (Roman), \em (italics), \bf (bold face), \sc (small caps), and \tt (teletype).
      • To simultaneously change font-type and size, put the appropriate font-type command after the size command, e.g., {\large \bf ...}.
    • To insert blank horizontal or vertical space, use the commands \hspace*{AMT} and \vspace*{AMT}, respectively, where AMT is some number followed by a distance unit, e.g., 0.78in, 5mm. To go to the top of the next page, use \newpage.
  • Example: A sample Latex document v2.0 [example2.tex, example2.pdf, example2.script]
  • LaTeX is capable of many, MANY more things -- for an overview, please see Lectures #5 and 6 of COMP 690A (Fall 2023) available here.

Friday, January 16 (Lecture #5) (FS)

• Up until now, we've looked at how to present finished research -- let us now look at useful skills for doing research.
• Effective Literature Search: Dealing with Google Scholar et al
  • The central task in literature search is finding the "core" set of papers on a particular topic.
    • This establishes what has and has not been done on a topic research-wise, and is critical for establishing both the importance and the originality of proposed research.
    • It is thus useful in a number of research-related activities, e.g., finding and validating research topics, writing theses and surveys.
  • The core papers on a topic form one or more sets of papers, each of which has the following relationships:
    • They share a common terminology (keywords).
    • They are written by a particular set of researchers (authors).
    • Modulo the different times at which they are written, they refer to each other in their bibliographies (citation).

    To find a core set of papers, we exploit each and often all of these relationships.

  • Example: Relationships between the core set of papers on a topic:
    
    
    

    In this diagram, an arrow from X to Y means that X is cited by Y.

  • The first step in literature search is locating an initial relevant paper.
    • This is a two-step process: Finding candidate relevant papers and evaluating the relevance of found candidate papers.
    • There are several ways to find candidate relevant papers:
      1. Reading the literature
         • If you read widely in Computer Science, you may have run across relevant papers or know where relevant papers are published.
         • Though laudable, this takes a lot of time.
      2. Asking an expert
         • An unbiased expert on a topic is often the best source for references to relevant papers.
         • The main problem here is finding such an unbiased expert; asking around your own colleagues and department is a good start.
         • Experts need not be local; indeed, it is heartening how often well-known experts will respond to a sincere and well-prepared e-mail asking for help on their topic of expertise.
         • There are also some wonderful on-line forums, e.g., CS Stack Exchange; by evaluating obtained answers carefully, one may access a "composite expert".
         • Expert advice must still be handled carefully, as you have your own biases, e.g., Warnow / Meacham.
         • A recent variant on this technique is to ask a generative AI system for relevant papers. Any results obtained in this mannmer should be very carefully checked given possible bias introduced during system training as well as known tendencies of such systems to hallucinate and make things up.
      3. Doing keyword search
         • Keyword search is often the best available source for references to relevant papers.
         • Though general search engines are useful, where possible, use academic search engines such as Google Scholar.
         • Use the initial keywords you use yourself to describe your topic, e.g., "computationally difficult"; however, if nothing useful-looking turns up, use synonyms, e.g., "intractable", "NP-hard", and more general rather than more specific synonyms at that, e.g., "(computational) complexity".
         • Consider adding the keywords "survey", "review", and "overview" to your search terms -- surveys are not only very useful in giving you a handle on a new topic but are also often invaluable for later stages of literature search (see below).
         • Example: Keyword search for intractability results in robot motion planning. [Google Scholar]
         • Example: Keyword search for intractability results in component-based software design. [Google Scholar]
         • Example: Keyword search for intractability results in software modularization. [Google Scholar]
    • The best way to evaluate relevance of a found paper is to fully read that paper. However, if done carefully, a skim-read of the paper (focusing on the abstract, introduction, discussions, and conclusions) will often suffice.
      • When using Google Scholar, a paper's abstract and often the paper itself can be obtained by clicking on the name of the paper.
      • The full text of some papers may only be available for payment.
      • The best way to get around this is do searches through a university computer account, as university libraries have often paid for access already.
      • Failing that, try the All X versions link associated with the paper by Google Scholar, or look for a (possibly pre-publication) version of the paper on the personal websites of the paper's authors.
    • One must not forget that a relevant paper must also be reliable, i.e., you can trust that what it says is true.
      • This is particularly important given the quasi-chaotic nature of academic publishing in the Web Age, in which unvetted poreprint archives and predatory publishers that will print anything for money abould.
      • Where possible, rely on papers associated with reputable publishers, e.g., ACM, IEEE, Springer, and vetted preprint archives, e.g., arXiv.
      • When in doubt, consult a domain expert.

Monday, January 19 (Lecture #6) (FS)

• Effective Literature Search: Dealing with Google Scholar et al (Cont'd)
  • Once one has an initial relevant paper X, the next step is to exploit paper-relationships to find other relevant papers in the core.
    • There are four ways to do this:
      1. Doing refined keyword search (RKS)
         • Very useful if core consists of multiple author- and/or citation-disjoint paper-sets.
         • Most useful if all papers in the core share the same terminology.
         • Can be done nicely using the keyword box in Google Scholar.
           • Can customize years searched using the links on the left-hand size of the search page.
           • Beware leaving specific date-ranges in place unintentionally, as this may compromise later searches.
         • Though specific keyword searches are more appropriate at this point, you may also find general keyword searches useful.
         • Example: RKS for intractability results in component-based software design (""computational complexity" component selection"). [Google Scholar]
         • Example: RKS for intractability results in robot motion planning (""computational complexity" robot motion planning") [Google Scholar]
         • May be less useful if there are multiple disjoint sets of appropriate terminology.
      2. Looking for papers that cite X (Forward Citation Search (FCS))
         • Very useful if core consists of multiple author- and/or keyword-disjoint paper-sets.
         • Can be done nicely using the Cited by X link associated with a paper by Google Scholar.
           • Do such searching methodically and mindfully; it is all too tempting do recursive forward searches and inadvertently lose track of useful found references.
         • Example: FCS relative to Petty, Weisel, and Mielke (2003). [Google Scholar]
         • Example: FCS relative to Cesati and Wareham (1995) [Google Scholar]
         • Most useful if X is older (but not too old (see below)).
         • May be less useful if X is recent, in that there may be less time for people to be aware of X.
         • May be less useful if X is older and has been superceeded in importance by more recent papers.
      3. Looking for papers that X cites (Backward Citation Search (BCS))
         • Very useful if core consists of multiple author- and/or keyword-disjoint paper-sets.
         • Is typically done manually relative to a paper's bibliography; however, such bibliographies are occasionally available online with the initial reference, e.g., certain papers published by ACM and Elsevier, references through Citeseer.
         • Example: BCS relative to Petty, Weisel, and Mielke (2003). [Google Scholar]
         • Example: BCS relative to Cesati and Wareham (1995). [PDF] [Google Scholar]
         • Most useful when X is recent and a survey.
         • May be less useful if X is older and has had the references in its bibliography superceeded in importance by more recent papers.
      4. Looking for other papers by the same author(s) (Author Name Search (ANS))
         • Very useful if core consists of multiple keyword- and/or citation-disjoint paper-sets.
         • Can be done nicely using the keyword box in Google Scholar.
           • May be problematic if the author name is common, e.g., "Smith", "Chang".
           • Failing that, consider using area-specific author-oriented online services, e.g., DBLP (Computer Science), or author-maintained homepages.
           • As each method above often leaves out some papers by an author, may need to use all three.
         • Example: My papers as listed by Google Scholar, DBLP, and my homepage. [Google Scholar, DBLP, My Home Page]
         • Most useful if author works exclusively on the topic of interest.
         • May be less useful if author is prolific and works on multiple topics.
         • Beware of shifts in author-name due to marriage, e.g., "Keeling" => "Hartley-Keeling", "Hemachandra" => "Hemaspaandra", or indecision, e.g., "Harold Todd Wareham" => "H. Todd Wareham" => "Todd Wareham".
    • Newly-found papers must have their relevance assessed; once their relevance is established, these papers are then used to search for other relevant papers.
    • The techniques above should be applied to newly-found papers in an iterative fashion until no new papers are found. This is not a gaurantee that there aren't any other relevant papers out there, but it is a convenient search termination heuristic.
  • A secondary but nonetheless important task in literature search is finding the full reference details for a paper.
    • This task arises because (1) conference page limits and certain journal bibliography styles force authors to shorten references and (2) people are lazy (including some authors regarding their own papers (!)).
    • Though bibliographic standards have declined over the last several decades, it is still useful in your own work to have full bibliographic references because such references enhance the look of your work as well as your professional reputation.
    • The methods given above for finding papers by keyword and author name often work well here.
      • If the paper has been located using Google Scholar, the cite tab associated with the paper gives its bibitem and BibTeX information.
      • As some of these entries have been compiled by uninformed manual or automatic techniques, beware incomplete and/or misleading entries (see the Cygan et al example below).
    • Failing that, one can fall back on going back (via Google or manual library search) to the paper itself in the Tables-of-Contents of the conference, journal, or book in which the paper appeared.
    • Example: Finding the full reference for Mueller et al (2009) "Similarity as Tractable Transformation." [Google Scholar]
    • Example: Finding the full reference for Wareham, Evans, and van Rooij (2011) [Google Scholar]
    • Example: Finding the full reference for Cygan et al (2015) "Parameterized Algorithms". [Google Scholar]
  • More tools are becoming available on-line all the time to aid in literature search (some of undoubtably well-meaning but questionable legal status (*)), e.g.,
    • Science Citation Index (SCI) (Wikipedia entry)
    • Web of Science (Wikipedia entry)
    • Scholarpedia
    • Academia
    • List of academic databases and search engines (Wikipedia entry)
    • Sci-hub (*)

    Many such tools are based on technical and business models taken from other (possibly less academicly relevant) applications, e.g., generic search engines (Google Scholar), social media (Academia.edu), and hence individually incorporate only certain aspects of literature search. There are also a variety of business models, e.g., freely available, subscription.

  • There may be opportunities to create a literature-comprehensive freely-available tool that incorporates all forms of used literature search described above, e.g., a fusion of the keyword and forward citation search capabilities of Google Scholar with the backward citation search capabilities of Citeseer and the author name search capabilities of DBLP (with reference bookmarking / collection capabilities and the self-addition / maintenance capabilities of Wikipedia thrown in). Given the burgeoning size of the world's research literature and the utility of this research in academia, government, and industry (and possibly to the general public as well), such a tool could be the basis of a viable business, given the right business model.

Wednesday, January 21 (Lecture #7) (FS)

• We've now talked about various techniques underlying the research process -- what about research itself?
• Starting Research: Some Initial Thoughts
  • Recall the three necessary properties of good research:
    
  • Finding a research topic that is important is not so crucial at the BSch or MSc levels (where one is arguably learning techniques and how to apply them as well as the manner in which research is conducted), but it is crucial at the PhD level as well as in academic research in general. The latter is so because it is important research which allows publication in recognized venues and the formulation of quality funding proposals which are highly likely to be funded, both of which are important in building an academic career of your choosing (and not merely one that is available).
  • Assessing the originality and importance of a research topic is typically done by performing a thorough literature search and consulting with experts in the topic area; assessing whether a topic is one that you can work on productively requires (often brutal) honesty about one's own abilities.
  • Failure to do any of these assessments fully can result in much wasted time and effort.
  • Example: The 2001 SAC Incident (Failure wrt Originality)
  • Example: The 2007 ICDT Incident (Failure wrt Importance)
  • Example: Things I Didn't Do for My MSc and PhD I: the combinatorics of reticulate evolutionary trees and the parameterized complexity of evolutionary tree inference (Failure wrt Doability)
  • The Research Process: The Ideal vs. (Bitter) Reality
    
  • The reality of research is that many false starts may happen at any point in the process; sometimes, one may even have to backtrack to an earlier stage in the process and begin again.
  • Though one cannot ever eliminate such false starts, one can with forethought and appropriate effort minimize their occurrence and make the research process not only shorter but more enjoyable.
• Starting Research: Selecting and Validating Topics
  • This is a crucial part of both thesis and general research.
  • Four types of research topics:
    1. Verification of existing work (Old Bones, Old Graveyards (OBOG)), e.g., rerunning or redoing a simulation study (the latter if the original code is missing) to see if the same results are obtained.
       • Given that reproducibility (and hence trustworthiness) is a core aspect of science, this is sometimes necessary.
       • Given the importance of the original work being reproduced and the existence of at least a partial description of how this work was done, such a topic can be both important and doable; however, it is not original and hence often does not get the respect it deserves (and hence may lose importance).
       • These are low-risk research topics which, given their frequent though undeserved lack of respect, are relegated to BSch theses or "housekeeping" aspects of academic research in general.
    2. New application of existing methods (Old Bones, New Graveyards (OBNG)), e.g., applying standard genetic algorithm design techniques to develop a genetic algorithm for improving the prioritization of orthodontic hamster surgery cases.
       • Such topics are typical of basic ongoing scientific research (cf. Kuhnian Scientific Revolutions) and hence are the most common research topics.
       • By virtue of using existing methods, such work is doable and has importance and originality in proportion to the importance and originality of the application area.
       • These are low- to moderate-risk research topics, and hence typical of BSch and MSc theses; are also useful in academic research in general when results must be obtained reliably and quickly, e.g., conference / funding proposal deadline.
    3. Development and verification of new methods (New Bones, Old Graveyards (NBOG)), e.g., developing either a new type of genetic algorithm or a totally new type of algorithm period.
       • Depending on the degree of novelty of the new method, such topics are typical of advanced ongoing scientific research and minor Kuhnian Scientific Revolutions and hence are somewhat rare.
       • Such topics are by definition original and important in proportion to the advantages and applicability of the new methods; however, such work may not be doable or very difficult to do.
         • In order to verify the advantages and applicability of a new method, it is critical to apply this method to applications to which existing methods have already been applied and hence which have results against which the new method can be compared.
       • These are moderate- to high-risk research topics (in proportion to the novelty of the new methods) and are hence typical of MSc and PhD theses as well as academic research in general; are most productively tackled when there is no hurry to obtain results, e.g., when you have several years grace to work on a PhD or after you have tenure.
    4. Simultaneous development of new methods and new applications (New Bones, New Graveyards (NBNG)), e.g., atomic and relativistic physics.
       • Such topics are typical of major Kuhnian Scientific Revolutions based on previously unknown phenomena, e.g., radioactivity, and hence are very rare.
       • Such topics are by definition original and important; however, they may not be doable or very difficult to do.
       • These are very-high-risk research topics, i.e., nice work if you can get it.

Friday, January 23 (Lecture #8) (FS)

• Starting Research: Selecting and Validating Topics (Cont'd)
  • Two sources of thesis topics:
    1. Supervisor-selected
       • Risk inversely proportional to the degree to which the supervisor knows both the topic area (and hence the topic's originality and importance) and your ability to work productively on the topic.
       • Can be completed quickly and is very good for learning particular methods; however, may be a tad boring at times.
       • The most typical type of BSch and MSc topic.
    2. Self-selected
       • Much riskier, as you have to evaluate originality, importance, and doability by yourself.
       • Is typically time-consuming and a somewhat more terrifying experience; however, one learns a lot more, including how to be an independent researcher.
       • The most typical type of PhD topic.
  • Two sources of research topics in general:
    1. Open questions in existing work
       • Such questions may be published in the literature or circulating by word-of-mouth among the members of a research community; access the former by literature search / reading and the latter by talking to colleagues at your home university and at conferences.
       • Depending on the reliability of the source, topics based on such questions are important and original, and there may already be hints on how they might be doable.
       • Note that the effort involved in phrasing a good open question, i.e., one that is original, important, and doable, often makes people (IMHO rightly) reluctant to share them unless credit for the effort is acknowledged by co-authorship or some other means, e.g., Fellows and Berger 1995 conversation.
    2. Combination of elements of existing work
       • Such combinations arise from the fortuitous collision within your mind of disparate ideas that you have encountered, e.g., why hasn't anyone thought of cooking mushrooms with a blowtorch before?; as such, one can increase the probability of such combinations by creativity-boosting mental exercises as well as by increasing the number of ideas you encounter by reading and talking to people.
       • While combinations may indeed by original, their originality and importance as well as the extent to which they are doable must be very carefully assessed.
  • Example: $M_l$ consensus tree algorithms (my BSch dissertation) [OBNG / supervisor-selected; open questions]
  • Example: The computational complexity of evolutionary tree inference (my MSc project) [OBNG / self-selected; combination]
  • Example: The parameterized complexity of phonological processing (my PhD project) [OBNG / self-selected; combination + open questions]
  • Example: The parameterized complexity of of analogy derivation under Structure Mapping Theory (van Rooij et al (2008)) [OBNG / collaborator-suggested; combination + open questions]
  • Example: The parameterized complexity of environment design for distributed construction (Wareham and Vardy (2018)) [OBNG / collaborator-suggested (sort of); misunderstanding + combination]
  • Example: The structural approximability of cognitive processing (van Rooij and Wareham (2012)) [NBOG / collaborator-suggested; open questions]
  • Though a lot of work, the process of selecting and validating a research topic can become a seductive activity in itself. One must be careful to always be mindful and assess why you are doing what you are doing. Remember: your goal is to get a topic that is original, important, and doable, and if one possible topic is not working out, move on to another in a timely manner.
  • Example: Things I Didn't Do for My MSc and PhD II: chaos-theoretic aspects of evolutionary inference.
  • A final caveat is in order: A good research topic takes effort to develop and is valuable in itself -- beware of spreading it around before you've had a chance to work on it fully, as others may not respect this effort.
  • Example: The CWCB 1996 Incident.
  • Example: The 1999 PhD Defense Incident.
  • Example: The RECOMB 2001 Incident.
  • Example: The 2014 Cog Sci Appendix Query.

Monday, January 26

• Student Talks I
  • Dhulipala, Venkata Satya Naga Arun Kousik
  • Faruk, Syed Omar

Wednesday, January 28

• Student Talks I (Cont'd)
  • Joshi, Karan
  • Rathore, Nishant

Friday, January 30

• Student Talks I (Cont'd)
  • Molaei, Mohammad Amin
  • Asgari, Ali

Monday, February 2

• University closed; class cancelled.

Wednesday, February 4

• Student Talks I (Cont'd)
  • Pickett, Julian

Friday, February 6 (Lecture #9)

• Starting Research: Timelines and Proposals
  • Graduate Degree Research Timeline
    
    • Above is the ideal; may take longer, but should not take too much longer (cf. my MSc thesis [4 years]).
    • Does not include PhD comprehensive exams.
  • The timeline for funded research projects is similar in structure to that above though the specific durations involved vary with the project length, e.g., OPC (prep + 2 months evaluation + 1 year project), NSERC Discovery grant (prep + 5 months evaluation + 5 year project).
  • Non-funded research projects typically do not have timelines or proposals; however, it may still be worthwhile to do one or both of these for your own use.
    • A non-funded research project proposal can take several forms, e.g., working notes, annotated bibliography, scheduled seminar (like the departmental Graduate Research Forum!).
    • Such proposals serve also in spurring research along, as (unless you are eventually do something for a conference, journal special issue, or edited book) there are typically no deadlines.
  • The graduate thesis research proposal
    • Lays out what you want do, how you're going to do it, and how long you think it will take you to do it.
      • A graduate thesis proposal is not so much a contract for specific work as a demonstration that you know how to conduct research and have put in the requisite preparatory work, e.g., literature search.
      • As such, a graduate thesis research proposal is a promissory note that you're ready to start research -- whether it is eventually the research you said you'd do (and in the manner and time you said you'd do it in) is another matter.
      • A thesis proposal is at its heart a sales pitch whose purpose is to convince others to buy the idea that you should be allowed to do research -- hence, whatever makes the proposal more convincing, e.g., easy-to-understand stories, backup of claims by literature citations, believable timelines, is to the good.
  • The graduate thesis research proposal
    • General structure of proposal (single-spaced pages):
      • Title
        • Titles should be descriptive and relatively short, e.g., my BSc, MSc, and PhD thesis titles.
        • Save cutesy titles for later publications, e.g., my original BSch thesis title.
      • Abstract (between 150 and 250 words)
        • Describes the proposal in brief.
        • The above is meant to be taken literally: Consider a an abstract as consisting of motivation and previous work of 1 sentence each, followed by a sentence encapsulating motivation and research question, e.g.,
          X is very important because of Y.
          Though there is much previous work on Z, no one has yet investigated X.
          In our research, we will examine X using W.

          A longer abstract simply adds extra clauses and sentences to the organization sketched above. A very good example of how to do this is contained in the online description of a Nature Summary Paragraph.

      • Motivation [~2 pages]
        • Describes why this project is worth doing, i.e., what will be done in this project that has not been done before (originality) and why this is important.
        • As part of this, should describe previous and related work as context for proposed work in order to highlight the hole in previous work that the proposed research will fill. Do this in sketchier form here, with more details in the next section of the proposal.
        • A story provides a natural way of organizing this section (see the short description of an abstract above).
        • It is particularly important to back up statements with literature citations in this section. Such citations not only establish the truth of your statements (Don't Blame Me -- Blame Them) but also show that you've put in the requisite preparatory work (Trust Me -- I Know What I'm Talking About).
      • Previous and Related Work [~2-4 pages]
        • Describes in detail what has been done before that is either directly or indirectly related to the proposed project, cf. Motivation.
        • It is tempting to just list and briefly describe references, not unlike an annotated bibliography. However, all possible effort should be made to impose more organization and structure, such as a taxonomy of existing work (possibly organized chronologically to show how research has proceeded over time).
          • Such an imposed organization makes for an easier read of the proposal, and should also make it much easier for you to structure both your motivation and research question sections.
          • A well-done previous work section may also be a (possibly publishable) research contribution in its own right, e.g., my MSc thesis.
        • Ideally this section should be structured to highlight that which has not been done previously which you plan to do -- this gives a natural bridge to the following sections in the proposal. Where possible, consider making this structure a story or narrative, e.g., "Once upon a time there was a very important problem. Many people have worked on this problem so far, but they haven't done important thing X. However, I have a neat plan to do X -- would you like to hear about it? If so, please keep reading ..."
      • Research Question [~1 page]
        • Describes what specific question or set of questions you will be addressing in your thesis work.
        • A research question is intermediate in rigor between a formal scientific hypothesis and an everyday question.
        • The precise form of the research questions depends on the research topic; are much easier to define if the research was inspired by open questions in the literature.
        • Example: The two general research questions underlying all of my parameterized-complexity-based research relative to an problem X:
          1. Can X be solved optimally in polynomial time in general?
          2. If not, what restrictions do (and do not) make X solvable in effectively polynomial time?

        • Every question should be well-motivated by the preceding proposal sections and be answerable relative to the subsequently-described methods to be employed in the research; as such, these questions are the central pivot on which all else in the proposal depends and operates.
      • Methodology [~2-4 pages]
        • Describes how you will answer the research questions.
        • Methods are typically described in general terms, with appropriate literature citations for details; that being said, these descriptions must be specific enough to allow explanations of how these methods can be used to answer the posed research questions, i.e., no ``magic occurs here'' processes.
        • Note that this section and the following one are the ones where you convince others that the proposed research is doable by you.
      • Timeline [~1 page]
        • Describes your anticipated course of progress in the proposed research.
          • The emphasis here is on ``anticipated'' -- this is research, not engineering, and experienced researchers know things seldom go according to plan.
        • Can do sophisticated multi-tier graphics here; however, the simplest form (an itemized list consisting of month-specific dates with attached research activities and/or deliverables) is often sufficient.
        • The timeline must be believable -- to ensure this, be as specific as possible when describing research activities, e.g., no ``magic occurs here'' processes, and budget for some difficulties when establishing dates, e.g., The Scotty Principle.
      • Bibliography (15-30 references) [~2-3 pages]
        • Bibliography should be alphabetic by author name, and all references should be both complete and correct in details and consistent in format, e.g., do not describe conference papers in two or more ways.
        • Compiling a good bibliography is a lot of work, but it plays an important role in convincing others that you can be trusted to do research; it can also be a research contribution in itself, e.g., my MSc thesis.
    • Example: My PhD research proposal [3rd draft] (PDF)
    • Remember that you need not write up the sections in the order in which they appear in the proposal -- in order to establish originality and importance, Related Work, Research Questions, and Bibliography should be done first, followed by Methodology (to establish do-ability). Once you then know where you'll end up story-wise, one can then do your Motivation and Timeline.
      • Allow time to revise your proposal after the first draft of all sections are finally written -- at that point, you MUST adjust story and terminology to be consistent throughout the proposal.

Monday, February 9 (Lecture #10)

• Starting Research: Timelines and Proposals (Cont'd)
  • The general structure above is also characteristic of funded research proposals modulo differences in the length of particular sections and the presence of additional sections, e.g., personal record in previous funded research and the training of Highly Qualified Personnel (HQP), budgets, budget justifications.
    • Note that in the case of proposals involving funding, the extent to which eventual research must follow that described in the proposal varies with the funding agency and grant-type.
    • Example: OPC / NSERC equipment / Dutch PhD project proposals vs. NSERC Discovery Grant proposals.
  • Example: My 2010 NSERC Discovery Grant proposal (PDF)
  • Wrt non-funded research projects, it may well be worth doing up specific sections described above (in particular, previous work, research questions, and bibliography) to help both focus the research from the beginning and aid in creating subsequent publications describing the results of this research (which will require these materials anyway).
  • In all cases, be honest in describing all aspects of your proposal, particularly the extent to which the proposed research is original and important and your ability to do it.
    • Part of this can be including frank descriptions of potential difficulties and how you will deal with them.
    • Given the temptation and incentives to over-sell research by being less than honest, honesty can set one apart in a good way and be reputation-enhancing.
    • Example: Reviewer comment on my 2010 NSERC Discovery Grant proposal.
• Doing (and Surviving) Research
  • There is nothing like either the joy of research going well or the despair of research going badly. Though one can never be quite sure what research will bring result-wise, one can with care and forethought simultaneously maximize the former and minimize the latter.
  • There are many techniques for carrying out computer science research -- these techniques are what one picks up in courses and personal reading and from talking to your supervisor and other academics.
  • We will focus here not so much on the research process but on how to survive and enjoy this process.
  • General Research Etiquette
    • There are some basic common-sense rules that should always be followed when doing research:
      1. Be honest in performing research and reporting results.
      2. Acknowledge the contributions of others to your research, e.g., do not (excessively self-)plagiarize.
      3. Where possible and prudent, help others in their research.
      4. Be courteous and respectful to research support staff

      We are human, and lapses will occur (particularly wrt (3) and (4)); however, be on guard for lapses in (1) and (2) ((1) in particular), as much of research depends on the trustworthiness of researchers and such lapses can destroy research careers.

    • Though life in general may not be fair, aim to create a peaceable kingdom wrt your research and the people you deal with in doing that research.
  • Graduate Thesis Research
    • The Thesis Lifestyle
      • The Pace of Thesis Research
        • Work hard and work consistently, i.e., try wherever possible to avoid last-ditch 24/7 work binges.
        • Quite aside from keeping your supervisor happy (by allowing regular demonstrations of research progress), this is an excellent strategy for ensuring that you always have additional time to cope with unanticipated difficulties
      • The Necessity of Not Doing Thesis Research
        • Make sure (if you have to, by scheduling it) that you have regular breaks away from your thesis doing something that you find relaxing and enjoyable.
          • This not only includes daily and weekend breaks but vacations, e.g., visits home.
        • This helps give you both the energy to do good research and necessary perspective on what you are doing research-wise.
        • Oddly enough, the above applies even when research is going very well -- the endorphin rush of good research can be seductive and potentially damaging, e.g., death by joy.
        • Example: My 2015 Summer (Non-)Vacation
      • The Necessity of Setting Personal Priorities wrt Work
        • There are plenty of people out there who are more than happy to tell you how much you should work (and conversely how little you should be away from work).
        • Time spent working has both benefits, e.g., faster thesis finishing time, more publications, and drawbacks, e.g., broken relationships, personal depression / burnout (see below).
        • Example: Cases in point: Dan Brooks, Tao Jiang, and Bill Threlfall.
        • You need to decide what your priorities are and choose the tradeoffs between work and life that are right for you before others do it for you.
        • During graduate studies, this balance is particularly hard to strike given the unequal distribution of power between students and supervisors; this is yet another reason why it so important to maintain cordial and open relations with one's supervisor (see below).
      • All of the above can benefit by making sure that you are good at allocating and managing time both to research and life in general -- anything that helps in this is to the good, e.g., training in Time Management / Meditation / Mindfulness.

Wednesday, February 11 (Lecture #11)

• Doing (and Surviving) Research (Cont'd)
  • Graduate Thesis Research (Cont'd)
    • The Care and Feeding of Graduate Research Supervisors
      • Ideally, you should have a good intellectual and personal fit with your supervisor, i.e., they should know the things you need to do research and you should be able to talk to them.
        • The former helps your research go forward; the latter helps when your research has problems.
      • Ideally, you should have regular (at least weekly or bi-weekly) meetings at which you discuss research (and later thesis) progress.
        • Regularly-scheduled meetings are a great way to keep up the pace of research.
        • Try not to dread these meetings -- if there has been no progress, use the meeting to figure out why this occurred and what can be done about it.
      • Example: My MSc and PhD supervisor meetings.
      • As in any relationship, if problems arise between you and your supervisor, talk them out at the earliest possible opportunity when at all possible.
      • Student-supervisor relations are seldom perfect, and the causes of deviations from the ideal as well as the degree to which these deviations are problematic vary wildly (Tolstoy's Dictum)
        • If your supervisor is bureaucratically challenged wrt regulations and deadlines, make sure that you know what the relevant regulations and timelines are and make sure that you follow them.
        • If your supervisor is not making time to meet with you or to give you timely advice and feedback, do not let them off the hook -- their time is your right as their student, and patiently but firmly request that they make time for you.
        • If your supervisor is not a good intellectual or personal fit, get the help you need elsewhere, e.g., supervisory committee members (PhDs), other faculty members (departmental and otherwise), fellow students of the same supervisor, fellow students, friends and family.
      • If there are irreconcilable differences between you and your advisor or the relationship between you becomes abusive, talk to your graduate officer immediately for advice.
        • Though there may ultimately be little that can be done to compel a tenured supervisor to behave, not letting others know what is going on will only allow things to get worse; moreover, the assistance of these others will be necessary if you need to switch supervisors.
        • Switching supervisors is a common fantasy in times of trouble, and it can be done and survived -- however, it is even in the best situations disruptive and should only be invoked for real if there is no other solution.
    • Writing Up the Thesis
      • Theses take different lengths of time to write -- perhaps the only common feature is that all theses take longer to write than you think they will. Hence, start thesis writeup as early as possible and budget time for problems that can arise.
      • Decide on your typesetting system up front.
        • The main choices in Computer Science are MS Word and LaTeX, with research areas varying in their preference.
        • Make sure that the conventions you adopt, i.e., for tables and figures, citations, and bibliographic references, are consistent with those mandated by your university.
        • Example MUN Thesis Guidelines (webpage)
        • Thesis templates (when available) can be invaluable.
        • Example Unofficial MUN Thesis Templates (LaTeX)
      • The standard thesis format looks strangely like that for the thesis proposal (introduction, background (including previous work), methodology, results, discussion, conclusions, bibliography).
        • This standard format van vary according to research area, e.g., software engineering vs. evolutionary algorithms vs. computational complexity analysis vs. theoretical computer science.
        • Your supervisor is typically your best guide to which thesis format you should use.
      • Theses need not be written up in either the final order of sections or at the final level of detail.
        • Consider first doing brief background and methodology sections and only fill these in with details after you have written up your results and discussion (at which point you will know the details you need to put into the background and methodology). Conclusions and the introduction can be done last.
        • As terminology and story are especially prone to vary with time over long-duration writeups like theses, allocate time at the end to make both consistent throughout.
        • Never make the bibliography the last thing you do -- always add all references in full as you cite them in the thesis drafts.
          • Any time you spend adding full references that are later deleted is typically vastly less than the you spend trying to recover references that you forgot.
          • The above is also a good reason why you should never wait until the text is finalized to add literature citations.
      • Make sure you acknowledge text, tables, and/or figures in your thesis that are derived or taken from other sources.
        • Small pieces of text, e.g., 1-2 sentences, can be placed in thesis text by enclosing them in quotation marks and giving a literature citation with page numbers at the end. Such page numbers can be done in LaTeX using the optional square-bracket argument for the \cite command, e.g.,
          This approach is not commonly used, in large part because of the opinion of practitioners such as Wareham who claim ``... that the whole thing is nonsense'' \cite[page 23]{War01}.

          Larger pieces of text should be offset from the thesis text, again with a literature citation with page numbers at the end. Such offsetting can be done in LaTeX using the quotation or quote text environments.

        • If tables or figures are adapted from other sources, e.g., the table or figure is simplified or enhanced, it suffices to acknowledge this in the caption, e.g.,
          \caption{A simplified model of a neuron (adapted from \cite[Figure 4]{SNW98}).}

          If the table or figure in your thesis is exactly the same as it appears in the source, e.g., you've re-used the JPEG file from the source, remove "adapted from" in the caption acknowledgment above and get written permission from the author and/or publisher to re-use the table or figure. Failure to do this can result in the rejection of your thesis on copyright grounds.

        • Given the severity of the repercussions for not handling such matters correctly, if you are at all in doubt about how re-use should be acknowledged wrt thesis text and obtained permissions, consult your supervisor and the MUN copyright office (webpage).
    • Solicit feedback from your supervisor and/or supervisory committee on individual chapters (and, when they are critical, individual sections) as they are written, i.e., avoid Big Bang Writeup.
      • This ensures that you do not drift too far from standard formats and/or supervisor preferences, and can save a lot of work.
      • This also allows early detection and resolution of format disagreements between co-supervisors as well as simplifying final thesis checking before submission for examination (see below).
    • Theses at MUN have been rejected because of poor written English. Hence, if you need help with your written English, get it.
      • Some supervisors provide this; in other cases, use the Writing Center (webpage) or hire an editorial consultant.
      • The Writing Center is free but may take a while; editorial consultants are not free but the benefit of feedback that is speedy (and possibly better-suited to you and your thesis) may make it worthwhile.
      • Some people do this throughout the writeup process while others only do it for the final draft prior to thesis submission -- I prefer the former, if only because Englishification of a large document all at once can take a lot of time.
    • Be aware of your intellectual property rights as regards your thesis.
      • Example: MUN thesis intellectual property rights (Webpage)
      • Depending on circumstances, you may not want to put everything you know or derive in your thesis.
      • Example: My 1990 MUN M.Eng. Experience.

Friday, February 13 (Lecture #12)

• Doing (and Surviving) Research (Cont'd)
  • Graduate Thesis Research (Cont'd)
    • Thesis Submission, Examination, and Defence
      • The process of submitting, examining, and defending theses takes different lengths of time -- perhaps the only common feature is that this process always takes longer than you think it will. Hence, you need to start this process as early as possible and budget time for problems that can arise.
      • The Official Timeline (MSc):
        1. Examiners selected and appointed by SGS Dean (1/2 month)
        2. Formal submission and examination (1-2 months)
        3. Notification of results (acceptable, minor revisions, major revisions, fail)
        4. Final submission to MUN (6 months for minor revisions; 12 months for major revisions and re-examination)

        Remember also that a departmental seminar must be given immediately prior to (2).

      • The Official Timeline (PhD):
        1. Review of thesis by supervisory committee (1-2 months)
        2. Examiners selected and appointed by SGS Dean (1/2 month)
        3. Formal submission and examination (1-2 months)
        4. Notification of results (acceptable, minor revisions, major revisions, fail)
        5. Defence and final submission to MUN (oral defence (+1 month), then 6 months for minor revisions; 12 months for major revisions and re-examination)

      • The official bottom line is to plan on 1-2 semesters to complete the process. However, for various reasons, it can take longer, e.g.,
        • Your supervisory committee may be away and/or busy (the latter during term, the former over holidays (Xmas, summer))
        • Paperwork requiring multiple signatures among multiple offices takes time (especially if you are trying to get it done between semesters or during holidays (late summer, Xmas)).
        • Examiners may be busy or unresponsive.
        • It may be problematic scheduling a multi-person defence (though this is much better now with e-defences).
        • Your own university paperwork may not be in order, e.g., outdated program of study, unpaid fees.

        Given the above, the best way to avoid such additional delays is to first make sure that all your paperwork is in order and then initiate the submission process early in the Fall or Winter terms.

      • The co-operation of supervisors and your supervisory committee is crucial during this phase of your thesis; hence, all advice for dealing with supervisor problems given above goes double here.
        • Students of bureaucratically challenged supervisors need to take particular care and be prepared for the non-standard, e.g., helping to prepare your own list of examiners (!).
      • Though this is a complex and scary process, don't panic -- if you have put in the requisite work and care, things will be okay.
        • In particular, the PhD defence is often an anti-climax -- my own experience and that of many others I've talked to is that afterwards, one's second reaction (after the initial "Yahoo!") is "What was I so worried about?"
        • Use as much stress as you need to make sure that you are prepared -- any stress after that is superfluous.
    • The Eternal Question: To Teach or Not to Teach?
      • This is more an issue for PhD than MSc students; however, it is something everyone should consider if they are looking at a career in academia.
      • Benefits: Teaching a course not only helps you decide if you can and want to teach, but it is also an invaluable enhancement of your CV when you go to apply for academic jobs.
      • Drawbacks: Teaching well (especially for the first time) is very, very time-consuming, and can easily (and seriously) delay your thesis research and writeup.
      • Example: Patricia Evans and myself.
      • Given the above, whether or not you should teach during your graduate studies must be discussed and cleared with your supervisor.
      • If you do decide to teach, it would be well worth your time to build up some expertise by taking a a course on teaching if one is available, e.g., TSEP..
    • Dealing with Graduate Research Burnout
      • Not to confused with End-of-Thesis Psychosis (ETP), e.g., OCD + paranoia + manic/depression, which everyone gets in the high-stress situation that is the end of a thesis.
      • Burnout is much more severe and has a variety of symptoms; what is most characteristic is long-term depression and lack of research and/or thesis progress.
      • There are a variety of causes of burnout, e.g., bad choices (topic, scheduling), bad luck (research, accidents, family, relationships)
      • Example: My BAh thesis burnout.
      • Example: My PhD thesis burnout.
      • Tips for averting burnout
        • Work hard and consistently to cope with unanticipated difficulties.
        • Maintain good and regular interactions with your supervisor.
        • Schedule and religiously take downtime.
        • Keep a progress diary (and read it).
      • What to do (and not to do) when burnout happens.
        • DO recognize the signs as early as possible.
        • DO ask for the help you need (and use it).
          • Depending on the causes and severity of your burnout, this help may involve counseling, therapy, and/or drugs.
          • Depending on the causes of your burnout, talk to the appropriate people, be it your supervisor, friends, or family, or mental health professionals, e.g.,
            • MUN Student Wellness and Counseling Center (webpage): 1-709-864-8500
            • Mental Health Emergency Services - Eastern Health (webpage): Dial 811
        • DO NOT be ashamed.
        • DO NOT ignore it.

Monday, February 16 (Lecture #13)

• Doing (and Surviving) Research (Cont'd)
  • Non-thesis research.
    • You will typically have more choice about what you do and a longer time in which to do it after you complete your theses, but depending on your job, you may have much less time in which to do it than you did during your theses, e.g., my niece's evaluation of my academic working life in 1999.
      • Good habits wrt managing your time and setting priorities may be even more crucial after your theses than they were during your theses.
      • This is yet another reason why it is a good idea to develop good habits wrt such matters during your theses.
    • The Eternal Question: To Collaborate or Not to Collaborate?
      • After (and to a degree, even during) your theses, you have the option of collaborating with others in doing research. One's first experience is most often collaborating with your supervisor and your fellow supervised grad siblings; this broadens later to others.
      • There are a variety of types of collaboration, varying for example in the number, diversity of expertises and/or geographical locations of the people involved and the duration of the collaboration.
        • Larger (especially geographically distributed) collaborations require much more management and co-ordination than smaller collaborations.
        • Collaborations involving diverse expertises require initial and ongoing establishment of common ground that may not be required if all collaborators are in the same research area.
        • Long-term collaborations require a much better intellectual, work-habit, and personal fit between you and your collaborators than short-term collaborations.
      • Benefits: Research can go faster and, with the talents of others, more robustly and in far more directions then you could have done on your own.
      • Drawbacks: Relationships with collaborators must be carefully co-ordinated and nurtured to result in productive and enjoyable research; writeup must also be co-ordinated very carefully, e.g., who has The Writing Stick for which part and who sets the overall tone and style of the writeup?
      • Like many other types of relationships, collaborations do not always work and even if they do, they may not work for long.
      • Example: My own collaborative relationships with Marco Cesati, Gianluca Della Vedova, Patricia Evans, Yvan Rose, and Iris van Rooij.
      • Whatever the form of collaboration, establishing who will be authors on any produced work and what the author order will be is very important.
        • This is because author order means different things even within subareas of the same discipline let alone in different disciplines and these things can have an impact on co-authors getting tenure and/or grants,
        • Example: The 1992 Wood's Hole biologist conundrum.
        • Example: Theoretical vs. non-Theoretical Computer Science.
        • If you are a graduate student, be aware that there are no standard guidelines wrt co-authorship with your supervisor; hence, when you start producing papers, it is worth having a frank chat with your supervisor about this.
        • Example:: Mike Fellows, Ulrike Stege, and Ken Kent.
        • There are benefits and drawbacks to every co-authorship policy. and there will always be people who will tell you what you should and should not do. It is therefore important for you to (as with our previous discussion on establishing work / life priorities) decide early on what is right for you before other people do it for you.
      • Though collaboration (particularly between disciplines or with industrial partners) is very strongly encouraged these days, you must ultimately decide what is right for you and the research life you want to lead.
    • Unfortunately, burnout is not restricted to theses but can occur at any time during research; hence, all advice on preventing and dealing with thesis research burnout given above also applies here.

Wednesday, February 18

• University closed; class cancelled.

Friday, February 20

• Instructor ill; class cancelled.

Monday, February 23

• Midterm break; class cancelled.

Wednesday, February 25

• Midterm break; class cancelled.

Friday, February 27

• Midterm break; class cancelled.

Monday, March 2 (Lecture #14)

• So, at the end of all of this, you've completed some research and have neat results. What now?
• Presenting Research I: The Joy of Writing Papers
  • Published papers are important because they are a measure of research success in academia -- your publications help you get funding and PhD offers when you are a graduate student and job offers, funding, tenure, and promotions when you become a professor.
  • Papers may be solicited by an explicit invitation or contributed without an explicit invitation. The former is characteristic of journal special issues or edited books and the latter by general conference or journal publication.
  • Given a piece of research there are three initial questions that must be answered before starting a paper.:
    1. Is there one paper or several?
       • Given the rewards listed above, there is a great pressure to publish as many papers as possible.
       • One response to this pressure is to break your research into one or more Minimal Publishable Units (MPUs), each of which is a separate paper.
         • One way to do do this by first publishing research as one or more conference papers (usually omitting details such as proofs) and then publishing one or more journal papers (this time with details).
         • While the MPU approach is sometimes seen as necessary, MPUs do not get a lot of respect and if over-used my harm your academic reputation -- hence, their use should be balanced with taking longer to create papers that tell complete stories wrt a research problem (the Shannon approach).
       • What makes a piece of research one paper or a set of papers varies with the academic discipline involved, and is best figured out by chatting with your supervisor and other researchers in your area.
       • Regardless of how many papers there are. each paper must tell a story.
    2. Who is the paper for?
       • This means determining the paper's audience, i.e., intended readers, and venue, e.g., conference / journal / book chapter.
       • The former dictates the focus and level of detail in the paper; the latter dictates paper structure, e.g., overall page length, required sections, as well as other important constraints, e.g., submission deadline (if any).
    3. Who are the (initial) authors of the paper?
       • Ideally, each researcher who made a significant contribution to a piece of research should be an author, with the lessser contributions of others being listed in the paper's Acknowledgements section.
       • However, this varies in practice, e.g., Mike Fellows and Ken Kent. Moreover, as paper preparation proceeds, authors may drop out or be added.
  • Once you've answered these questions, preparation of the paper(s) can begin. This involves several more decisions:
    1. The typesetting systems and tools to be used.
       • As with graduate theses, the most common options in Computer Science are MS Word and LaTeX.
       • Most journals, conferences, and book publisher make MS Word and/or LaTeX paper templates available -- if so, use them to simplify both your writing and downstream tasks, e.g., making sure you are obeying any page limits (see below).
       • Overleaf is a great tool for co-ordinating multi-author papers in LaTeX.
    2. The structure of the paper.
       • This involves laying out the sections in the paper, what is in each section, and (at least estimates of) the number of pages for each section.
       • The last of these is very important if there are (as is often the case, e.g., conferences, book chapters, (some) journals) paper page length limits.
    3. If the paper is multi-author, which author(s) are responsible for which sections.
       • Ideally, this should leave each section with one author; if there is more than one, extreme care must be taken to ensure that authors do not accidedntally overwrite each other's work.
       • In any case, one author should be designated as the master author, who is responsible for co-ordinating the activities of all other authors (and, if there is a deadline, keeping everyone on schedule).
  • At this point, actual writing can proceed. If the paper is written over a period of time, care must be taken that both final terminology and story are consistent over the whole paper -- this is particularly important in multi-author papers, and may involve final rewriting and tweaking by the master author.
  • Example: The 2008 analogy conference paper (van Rooij et al (2008)).
  • Example: The 2011 problem solving journal paper (Wareham, Evans, and van Rooij (2011)).
  • Example: The 2018 swarm design journal special issue paper (Wareham and Vardy (2018)).

Wednesday, March 4

• Course Talk #2 Notes
  I have done up a schedule for the first round of student course talks. These talks will run in our lecture slots starting on Monday, March 16, and concluding on Monday, March 23. The talks are ordered in a random fashion; if you need to switch your talk slot, please arrange to switch with other people yourself and, when this is done, tell me the changes so I can adjust the on-line schedule accordingly.

  Your talk will be on your course project topic and 20 minutes long with 5 minutes afterwards for questions. The marking scheme for these talks is here. You may use the in-class blackboard and AV facilities as you please during your talks.

  May each of you please attend all talks and have fun with your own.

Wednesday, March 4 (Lecture #15)

• Presenting Research I: The Joy of Writing Papers (Cont'd)
  • Example: The 2019 cognition complexity invited book (van Rooij et al (2019)).
  • Example: The 2026 (hopefully) problem solving invited book chapter.
• Presenting Research II: The Joy of Peer Review
  • Peer review is the process by which research work submitted for academic publication is evaluated.
  • The peer review process:
    1. A piece of research work is submitted to some venue (e.g., conference, journal, edited book) for publication and possibly presentation (oral or poster). The submitted work may or may not be author-anonymized.
    2. The venue chair sends the submitted work out for evaluation to two or more reviewers, who produce both numerical scores assessing aspects of the quality of and appropriateness of the submission and (possibly separate) sets of comments for both the authors and the venue chair. These reviews are typically unsigned.
    3. The venue chair uses the reviews to determine whether or not the submitted work is accepted for publication in the venue.
    4. The venue chair's decision and the reviews are sent to the authors of the submitted work.
       1. If the decision is acceptance, the authors make any suggested revisions and submit the revised version of the work for publication.
       2. If the decision is rejection, depending on the venue, the authors may be able to give rebuttals to the reviewers and/or submit a revised version for re-evaluation.
  • Reviewer anonymity is a crucial part of the review process, in that it allows criticism without fear of reprisal, e.g., senior academics denying tenure and/or publication to junior academics who criticize them.
  • Author anonymity is desirable in that it allows criticism unbiased by the reputation (good or bad) of the authors; however, its implementation in peer review varies widely across research communities and venues.
  • Purposes of peer review:
    1. To help the venue chair decide whether or not the submitted work is worthy of publication in the venue.
    2. To provide constructive feedback to the authors of the submitted work.
    3. To give the reviewer the opportunity to (1) be of service to their research community and (2) give evidence of such service in their CV.

    The last of these is the reason why many supervisors encourage their graduate students to do peer reviews.

  • Given the first two of the purposes above, an ideal review should give a concise summary of the reviewer's opinion of the worthiness of the submitted work along with reasons for this opinion (to aid the venue editor in making their decision) as well as clear, constructive, and detailed comments and revision suggestions (to aid the authors in producing the best possible version of the work).
    • The editor-summary can be in the front of the review and consist of an initial paragraph briefly summarizing the work's contributions, one or more following paragraphs giving a concise assessment (with reasons) of the originality and worthiness of the work, followed by a brief recommendation of acceptance or rejection for the work.
      • Depending on the software system or text protocol used to do reviews, this summary may also include numerical scores provided by the reviewer (typically on a 1-5 scale) assessing aspects of the submitted work (e.g., originality, correctness, relevance to the venue, quality of writing) as well as the review process itself (e.g., familiarity with the research area of the submitted work).
    • The author-comments can follow after the editor-summary and consist of a list of general overall suggestions followed by a list of specific comments; the latter should always use a consistent and precise scheme to localize the part of the submitted work to which the comments apply, e.g., Page 5, Section 2.3, Paragraph 2, line 3.
  • Example: Conference paper reviews (COCOA 2008)
  • Example: Journal paper review (TCBB 2006)
  • Guidelines for doing reviews
    • Spend an appropriate amount of time reading the submitted work, e.g., 30 minutes -- 2 hours for a conference paper and a day for a journal paper.
    • Spend an appropriate amount of time writing up your review, e.g., 1 -- 3 hours for a conference paper and a day for a journal paper.
    • You do not have to understand and verify that absolutely everything in the submitted work is correct to give a useful evaluation. This is especially important to remember if you are reviewing a paper that is not in your area of expertise.
    • You are not responsible for providing an exhaustive and definitive list of suggestions that will produce a perfect version of the submitted work -- rather, the aim to help the authors as much as you can within the limits imposed by a reasonable investment of your time.
    • Always be specific and concise in and provide clear justifications for all points raised in your comments to the venue editor.
    • Always be constructive and specific in your comments to the authors.
    • Always phrase your review as if you signed it, i.e., don't say anything anonymously that you would not be prepared to say and defend to the author's face.
    • Always respect deadlines for returning reviews.

Friday, March 6 (Lecture #16)

• Presenting Research II: The Joy of Peer Review (Cont'd)
  • Guidelines for dealing with reviews
    • Read reviews in a calm and supportive environment, e.g., the University of Victoria Rabbit House, a cocktail bar.
    • After the initial reading, put the reviews aside for a day to give your emotions time to settle; when you come back to them, decide in consultation with your co-authors (if any) (1) what comments are sensible and constructive and (2) what you are going to do about them.
      • Venues vary in how closely they require that you take review suggestions into account when revising a submitted work for publication, e.g., EpiRob vs. CGAMES.
    • Whatever the decision, try to (eventually at least) be as calm as possible.
  • Example: The submission history of my game agent playability paper with Scott Watson:
    • CIG 2014 (Submission/ Reviews)
    • CCAI 2015 (Submission/ Reviews)
    • CGAMES 2015 (Submission/ Reviews (Review #1/ Review #2)/ Published Version)
  • Sometimes, one has the option of multiple rounds of paper revision with the same venue (be it a conference or journal).
    • In such cases, along with the revised paper, one must also submit a response letter detailing how each reviewer comment has been addressed (or why it has not been addressed).
    • Such response letters must be phrased very diplomatically.
    • Sometimes, this works; sometimes, it doesn't.
  • Example: Revision response letters for the three rounds of revision on Bozorgi et al (2017) for the journal Knowledge-based Systems.
    • Letter #1 (2016) (PDF)
    • Letter #2 (2017) (PDF)
    • Letter #3 (2017) (PDF)
  • Example: My 2017 JIRS Experience (Revision response letter text).
  • Whether you are dealing with one or multiple rounds of revision, one should have methods for highlighting in the re-submitted paper what has been changed as well as referencing these changes in the response letter.
    • Some venues have stated methods which they require you to use; other times, you need to develp your own.
    • If you are doing your own, make sure however you highlight and reference changes can be undone easily for the next (hopefully final and accepted) version of the paper.
  • Example: Revision response for the single round of revision on Wareham et al (2023) for the journal ACM Transactions on Human-Robot Interaction.
    • Reviewer comments (txt)
    • Response to reviewers (txt)
    • Revised paper (PDF)
  • After all is said and done, even if they disagree with what you've done, constructive reviews are invaluable and can (if acted on) greatly improve the presentation and technical content of your papers (indeed, I have frequently thanked anonymous reviewers in the Acknowledgements sections of my own papers). Given the work that goes into constructive reviews, they should be treasured and in turn inspire you to write constructive reviews of thee work of others.

Monday, March 9 (Lecture #17)

• Presenting Research III: The Joy of Conferences
  • Benefits: Publishing at conferences is not only a great way to get quick feedback on your research (especially if it is still in progress) but it also gives you a chance to meet and converse with other researchers (which can be useful in many ways, e.g., getting advice on research, meeting potential research collaborators, furthering research with existing collaborators, establishing or maintaining your presence in your research community).
  • Drawbacks: Conference travel costs money and can take time away from the research process and conference publications, while valid CV entries (even if the conference is refereed), are not valued as highly as journal papers or chapters in edited books.
  • There are typically a number of conferences -- how do you choose which one is best for you and your work? There are a variety of reasons for choosing a conference, e.g.,
    • conference topic (the conference topic is either the same as your work or a topic you want to know more about)
    • conference attendees (the conference is attended by people you know or want to know)
    • conference location (the conference is somewhere close by (and hence cheap to travel to) or somewhere you want to go (possibly for non-academic reasons))
    • conference date (the conference fits in with other travel you are doing (so you can get more bang for your travel dollar. e.g., Bill Day's Triple Bank Shot) or the conference proceedings will appear in time for you to cite any accepted paper(s) in an upcoming grant application)
    • conference acceptance standards (the conference accepts few or lots of papers)

    There is typically more than one reason for picking a particular conference, and it is quite acceptable for you to weight reasons according to your priorities when making a decision.

  • An excellent source for conference candidates is the list of conferences that published papers that you cite in your work; another is your graduate supervisor.
  • Where possible, publish in reputable conferences.
    • Conferences whose submissions are refereed are to be preferred over those that are not.
    • Conferences that publish proceedings in accepted series, e.g. Lecture Notes in Computer Science (Springer), are to be preferred over conferences that self-publish proceedings.
    • Conferences sponsored by established Computer Science organizations, e.g., ACM, IEEE, are to preferred over conferences that are not.

    Beware of predatory conferences that publish anything for money, as this can harm your reputation (e.g., my EAI and Carr experiences).

  • Once you have selected a conference, download its typesetting template immediately and use it to prepare your paper from as early as possible; this prevents last-minute formatting problems from derailing your submission.
  • Even if your paper is accepted, there may be varying grades of acceptance, e.g.,
    • Paper accepted for oral presentation with publication of full paper in proceedings.
    • Paper accepted for poster presentation with publication of full paper in proceedings.
    • Paper accepted for poster presentation with publication of abstract in proceedings.

    As posters and abstracts are not as valued as talks and full papers, think seriously about whether it is worthwhile to attend if you are offered the second or third alternatives.

  • You can list accepted conference papers in your CV (and some grant applications) before the conference proceedings is published -- mark them as "to appear", e.g.,
    Wareham, T. (2016) Some notes on conference submission. To appear, 16th International Conference Conference (ICC 2016).

    As soon as the proceedings are published, fill in the full details in your CV accordingly.

  • Book conference travel and hotels as early as possible to make sure you get what you want and need, e.g., travel dates and/or hotels close to the conference site, for a reasonable price.
    • If the conference is held in a hotel, you sometimes get a reasonable rate as a conference attendee; if not, consider staying at a reasonably-priced (and possibly better quality) hotel nearby.
    • Example: My Seattle Conference Hotel Experience.
    • Example: The Fellows LA (Non)Conference Hotel Experience.
    • Apply for student travel grants if the conference offers them -- these help defray costs (leaving money for other conference travel) and can also be nice CV entries in themselves.
  • Conference attendance guidelines
    • If you are a graduate student, travel if possible with your supervisor, especially if your supervisor knows people at the conference -- this is a great way to get introduced to people, e.g., my experiences with Bill Day and Bill Gasarch. If you cannot travel with your supervisor, see if they can arrange for you to meet some of their contacts at the conference, e.g., Mesam Timmar.
    • You do not have to attend everything scheduled at the conference -- look at the schedule before the conference and at the end of each conference-day to ensure that you attend the events that are most useful to you (and have backup events in place if something that looks interesting turns out to be a dud). This is particularly crucial at large conferences with parallel event-tracks.
      • This also allows you to find "dead spots" on the conference schedule that allow you to do sightseeing, e.g., AMIA Remembrance Day.
    • Lots of stuff gets done at conferences outside of scheduled events, e.g., around snack-break tables, at meals; stay alert for such opportunities and balance them up against the scheduled events, e.eg., my STRUCTURES dinner.
    • By all means network but don't be pushy about it -- some people, e.g., myself, do not react well to being obviously (even if gently) networked.
  • Finally, saving money on hotels and travel is a laudable goal but never compromise on safety.
  • Example: Yvan Rose's Cheap Paris Hotel Experience.
  • Example: My AAAI 1996 Vancouver Bus / Train Station Experience.

Wednesday, March 11` (Lecture #18)

• Presenting Research IV: The Joy of Journals
  • Benefits: Journal papers are highly-valued contributions to your research community and CV and are the accepted venue for distributing completed research.
  • Drawbacks: Journal publication can take a long time (especially if the paper is rejected by one or more journals) and there may be unanticipated monetary costs, e.g., open access fees.
  • There are typically a number of journals -- how do you choose which one is best for you and your work? There are a variety of reasons for choosing a journal, e.g.,
    • journal topic (the journal topic is the same as that of your paper)
    • journal readership (the journal is read by people who you want your work to be read by)
    • journal publication costs (if any)
    • journal acceptance standards (the journal accepts few or lots of papers)
    • journal review and publication timeline (the journal promises quick review and publication of papers)

    There is typically more than one reason for picking a particular journal, and it is quite acceptable for you to weight reasons according to your priorities when making a decision.

  • An excellent source for journal candidates is the list of journals that published papers that you cite in your work; another is your graduate supervisor.
  • Where possible, publish in reputable journals.
    • Journals sponsored by established Computer Science organizations, e.g., ACM, IEEE, and/or journal publishers are to preferred over journals that are not.
    • Measurements of journal importance like impact factors can also be used to judge journal reputability.
    Beware of predatory journal that publish anything for money, as this can harm your reputation (see Beall's List for a recognized list of potential predatory publishers).

  • Once you have selected a journal, download its typesetting template immediately and use it to prepare your paper from as early as possible; this prevents last-minute formatting problems that may hold up submission (or publication if the paper is accepted).
  • There are three possible responses from the journal:
    • Paper is accepted (possible subject to minor revisions)
    • Paper requires major revisions and resubmission
    • Paper is rejected

    In the first two cases, make the suggested revisions promptly and resubmit; this resubmission may include a letter to the editor giving details of changes you made as well as responses to the comments / questions posed by the referees. Be aware that in the second case, there may be several rounds of revision, and acceptance is never guaranteed. As the for the final case, if this happens, breathe deeply, count to ten, and (unless irrevocable flaws in your work have been pointed out by the referees) prepare to revise and resubmit that work elsewhere.

  • Example: The publication history of van Rooij, Wright, and Wareham (2012).
  • The advice to make revisions and resubmit promptly in the first two cases above is crucial -- to do otherwise can dramatically prolong the time to publication.
  • Example: The publication history of van Rooij and Wareham (2012).
  • You can list journal papers in your CV (and some grant applications) when the papers have been submitted, when they are being revised, and when they have been accepted but not yet published. There are a variety of ways of doing this format-wise, e.g.,
    Wareham, T. (2016) Some notes on conference submission. Submitted to The Journal of Conferences.

    Wareham, T. (2016) Some notes on conference submission. Under revision for The Journal of Conferences.

    Wareham, T. (to appear) Some notes on conference submission. The Journal of Conferences.

    As soon as the paper is officially published, fill in the full details in your CV accordingly.

Monday, March 16

• Student Talks II
  • Pickett, Julian
  • Rathore, Nishant

Wednesday, March 18

• Student Talks II (Cont'd)
  • Dhulipala, Venkata Satya Naga Arun Kousik
  • Molaei, Mohammad Amin

Friday, March 20

• Student Talks II (Cont'd)
  • Asgari, Ali
  • Joshi, Karan

Monday, March 23

• Student Talks II (Cont'd)
  • Faruk, Syed Omar

References

• Bozorgi, A., Samet, S., Kwisthout, J., and Wareham, T. (2017) "Community-based Influence Maximization in Social Networks under a Competitive Linear Threshold Model." Knowledge Based Systems, 134, 149-158.
• Barik, T., Gulwani, S., and Juarez, M. (2022). "Storytelling and science." Communications of the ACM, 65(10), 27-30. [PDF]
• Lamport, L. (1994) LaTeX: A Document Preparation System: User's Guide and Reference Manual (Second Edition). Addison-Wesley. (Abbreviated above as LL)
• Smith, A.J. (1990) "The Task of the Referee." IEEE Computer, 23(4), 65-71. (PDF)
• van Rooij, I., Blokpoel, M., Kwisthout, J., and Wareham, T. (2019) Cognition and Intractability: A Guide to Classical and Parameterized Complexity Analysis. Cambridge University Press.
• van Rooij, I., Evans, P., Muller, M., Gedge, J., and Wareham, T. (2008) "Identifying Sources of Intractability in Cognitive Models: An Illustration using Analogical Structure Mapping." In B.C. Love, K. McRae, and V.M. Sloutsky (eds.) Proceedings of the 30th Annual Meeting of the Cognitive Science Society. Cognitive Science Society; Austin, TX. 915-920.
• Wareham, T. (2012) "Flyby: life before, during, and after graduate studies with mike fellows. In The Multivariate Algorithmic Revolution and Beyond: Essays Dedicated to Michael R. Fellows on the Occasion of His 60th Birthday. Springer; Berlin. 51-55. [PDF]
• Wareham, T., Evans, P., and van Rooij, I. (2011) "What Does (and Doesn't) Make Analogical Problem Solving Easy? A Complexity-Theoretic Investigation." Journal of Problem Solving, 3(2), 30-71.
• Wareham, T., de Haan, R., Vardy, A., and van Rooij, I. (2023) "Swarm Control for Distributed Construction: A Computational Complexity Perspective." ACM Transactions on Human-Robot Interaction, 12(1), 6:1-6:45.
• Wareham, T. and Vardy, A. (2018) "Putting It Together: The Computational Complexity of Designing Robot Controllers and Environments for Distributed Construction." Swarm Intelligence, 12(2), 111-128.

Created: November 12, 2025
Last Modified: March 9, 2026