Computer Science 690B, Winter '24
Course Diary:

Research Methods in Computer Science II

---

Copyright 2024 by H.T. Wareham
All rights reserved

---

Week 1, Week 2, Week 3, Week 4, Week 5, Week 6,
Week 7, Week 8, Week 9, Week 10, Week 11, Week 12, (end of diary)

Thursday, January 4 (Lecture #1)

• Introduction to CS 690B: Content and Logistics (Slides)
• The Research Process: The Ideal vs. (Bitter) Reality
  
  • The reality of research is that many false starts may happen at any point in the process; sometimes, one may even have to backtrack to an earlier stage in the process and begin again.
  • Though one cannot ever eliminate such false starts, one can with forethought and appropriate effort minimize their occurrence and make the research process not only shorter but more enjoyable.
• Starting Research: Selecting and Validating Topics
  • This is a crucial part of both thesis and general research.
  • Four types of research topics:
    • Verification of existing work (Old Bones, Old Graveyards (OBOG)), e.g., rerunning or redoing a simulation study (the latter if the original code is missing) to see if the same results are obtained.
      • Given that reproducibility (and hence trustworthiness) is a core aspect of science, this is sometimes necessary.
      • Given the importance of the original work being reproduced and the existence of at least a partial description of how this work was done, such a topic can be both important and doable; however, it is not original and hence often does not get the respect it deserves (and hence may lose importance).
      • These are low-risk research topics which, given their frequent though undeserved lack of respect, are relegated to BSch theses or "housekeeping" aspects of academic research in general.
    • New application of existing methods (Old Bones, New Graveyards (OBNG)), e.g., applying standard genetic algorithm design techniques to develop a genetic algorithm for improving the prioritization of orthodontic hamster surgery cases.
      • Such topics are typical of basic ongoing scientific research (cf. Kuhnian Scientific Revolutions) and hence are the most common research topics.
      • By virtue of using existing methods, such work is doable and has importance and originality in proportion to the importance and originality of the application area.
      • These are low- to moderate-risk research topics, and hence typical of BSch and MSc theses; are also useful in academic research in general when results must be obtained reliably and quickly, e.g., conference / funding proposal deadline.
    • Development and verification of new methods (New Bones, Old Graveyards (NBOG)), e.g., developing either a new type of genetic algorithm or a totally new type of algorithm period.
      • Depending on the degree of novelty of the new method, such topics are typical of advanced ongoing scientific research and minor Kuhnian Scientific Revolutions and hence are somewhat rare.
      • Such topics are by definition original and important in proportion to the advantages and applicability of the new methods; however, such work may not be doable or very difficult to do.
        • In order to verify the advantages and applicability of a new method, it is critical to apply this method to applications to which existing methods have already been applied and hence which have results against which the new method can be compared.
      • These are moderate- to high-risk research topics (in proportion to the novelty of the new methods) and are hence typical of MSc and PhD theses as well as academic research in general; are most productively tackled when there is no hurry to obtain results, e.g., when you have several years grace to work on a PhD or after you have tenure.
    • Simultaneous development of new methods and new applications (New Bones, New Graveyards (NBNG)), e.g., atomic and relativistic physics.
      • Such topics are typical of major Kuhnian Scientific Revolutions based on previously unknown phenomena, e.g., radioactivity, and hence are very rare.
      • Such topics are by definition original and important; however, they may not be doable or very difficult to do.
      • These are very-high-risk research topics, i.e., nice work if you can get it.
  • Two sources of thesis topics:
    1. Supervisor-selected
       • Risk inversely proportional to the degree to which the supervisor knows both the topic area (and hence the topic's originality and importance) and your ability to work productively on the topic.
       • Can be completed quickly and is very good for learning particular methods; however, may be a tad boring at times.
       • The most typical type of BSch and MSc topic.
    2. Self-selected
       • Much riskier, as you have to evaluate originality, importance, and doability by yourself.
       • Is typically time-consuming and a somewhat more terrifying experience; however, one learns a lot more, including how to be an independent researcher.
       • The most typical type of PhD topic.
  • Two sources of research topics in general:
    1. Open questions in existing work
       • Such questions may be published in the literature or circulating by word-of-mouth among the members of a research community; access the former by literature search / reading and the latter by talking to colleagues at your home university and at conferences.
       • Depending on the reliability of the source, topics based on such questions are important and original, and there may already be hints on how they might be doable.
       • Note that the effort involved in phrasing a good open question, i.e., one that is original, important, and doable, often makes people (IMHO rightly) reluctant to share them unless credit for the effort is acknowledged by co-authorship or some other means, e.g., Fellows and Berger 1995 conversation.
    2. Combination of elements of existing work
       • Such combinations arise from the fortuitous collision within your mind of disparate ideas that you have encountered, e.g., why hasn't anyone thought of cooking mushrooms with a blowtorch before?; as such, one can increase the probability of such combinations by creativity-boosting mental exercises as well as by increasing the number of ideas you encounter by reading and talking to people.
       • While combinations may indeed by original, their originality and importance as well as the extent to which they are doable must be very carefully assessed.

Tuesday, January 9 (Lecture #2)

• Starting Research: Selecting and Validating Topics (Cont'd)
  • Example: $M_l$ consensus tree algorithms (my BSch dissertation) [OBNG / supervisor-selected; open questions]
  • Example: The computational complexity of evolutionary tree inference (my MSc project) [OBNG / self-selected; combination]
  • Example: The parameterized complexity of phonological processing (my PhD project) [OBNG / self-selected; combination + open questions]
  • Example: The parameterized complexity of of analogy derivation under Structure Mapping Theory (van Rooij et al (2008)) [OBNG / collaborator-suggested; combination + open questions]
  • Example: The parameterized complexity of environment design for distributed construction (Wareham and Vardy (2018)) [OBNG / collaborator-suggested (sort of); misunderstanding + combination]
  • Example: The structural approximability of cognitive processing (van Rooij and Wareham (2012)) [NBOG / collaborator-suggested; open questions]
  • Though a lot of work, the process of selecting and validating a research topic can become a seductive activity in itself. One must be careful to always be mindful and assess why you are doing what you are doing. Remember: your goal is to get a topic that is original, important, and doable, and if one possible topic is not working out, move on to another in a timely manner.
  • Example: Things I Didn't Do for My MSc and PhD II: chaos-theoretic aspects of evolutionary inference.
  • A final caveat is in order: A good research topic takes effort to develop and is valuable in itself -- beware of spreading it around before you've had a chance to work on it fully, as others may not respect this effort.
  • Example: The CWCB 1996 Incident.
  • Example: The 1999 PhD Defense Incident.
  • Example: The RECOMB 2001 Incident.
  • Example: The 2014 Cog Sci Appendix Query.
• Starting Research: Timelines and Proposals
  • Graduate Degree Research Timeline
    
    • Above is the ideal; may take longer, but should not take too much longer (cf. my MSc thesis [4 years]).
    • Does not include PhD comprehensive exams.
  • The timeline for funded research projects is similar in structure to that above though the specific durations involved vary with the project length, e.g., OPC (prep + 2 months evaluation + 1 year project), NSERC Discovery grant (prep + 5 months evaluation + 5 year project).
  • Non-funded research projects typically do not have timelines or proposals; however, it may still be worthwhile to do one or both of these for your own use.
    • A non-funded research project proposal can take several forms, e.g., working notes, annotated bibliography, scheduled seminar (like the departmental Graduate Research Forum!).
    • Such proposals serve also in spurring research along, as (unless you are eventually do something for a conference, journal special issue, or edited book) there are typically no deadlines.
  • The graduate thesis research proposal
    • Lays out what you want do, how you're going to do it, and how long you think it will take you to do it.
      • A graduate thesis proposal is not so much a contract for specific work as a demonstration that you know how to conduct research and have put in the requisite preparatory work, e.g., literature search.
      • As such, a graduate thesis research proposal is a promissory note that you're ready to start research -- whether it is eventually the research you said you'd do (and in the manner and time you said you'd do it in) is another matter.
      • A thesis proposal is at its heart a sales pitch whose purpose is to convince others to buy the idea that you should be allowed to do research -- hence, whatever makes the proposal more convincing, e.g., easy-to-understand stories, backup of claims by literature citations, believable timelines, is to the good.

Tuesday, January 16 (Lecture #3)

• Starting Research: Timelines and Proposals (Cont'd)
  • The graduate thesis research proposal
    • General structure of proposal (single-spaced pages):
      • Title
        • Titles should be descriptive and relatively short, e.g., my BSc, MSc, and PhD thesis titles.
        • Save cutesy titles for later publications, e.g., my original BSch thesis title.
      • Abstract (between 150 and 250 words)
        • Describes the proposal in brief.
        • The above is meant to be taken literally: Consider a an abstract as consisting of motivation and previous work of 1 sentence each, followed by a sentence encapsulating motivation and research question, e.g.,
          X is very important because of Y.
          Though there is much previous work on Z, no one has yet investigated X.
          In our research, we will examine X using W.

          A longer abstract simply adds extra clauses and sentences to the organization sketched above. A very good example of how to do this is contained in the online description of a Nature Summary Paragraph.

      • Motivation [~2 pages]
        • Describes why this project is worth doing, i.e., what will be done in this project that has not been done before (originality) and why this is important.
        • As part of this, should describe previous and related work as context for proposed work in order to highlight the hole in previous work that the proposed research will fill. Do this in sketchier form here, with more details in the next section of the proposal.
        • A story provides a natural way of organizing this section (see the short description of an abstract above).
        • It is particularly important to back up statements with literature citations in this section. Such citations not only establish the truth of your statements (Don't Blame Me -- Blame Them) but also show that you've put in the requisite preparatory work (Trust Me -- I Know What I'm Talking About).
      • Previous and Related Work [~2-4 pages]
        • Describes in detail what has been done before that is either directly or indirectly related to the proposed project, cf. Motivation.
        • It is tempting to just list and briefly describe references, not unlike an annotated bibliography. However, all possible effort should be made to impose more organization and structure, such as a taxonomy of existing work (possibly organized chronologically to show how research has proceeded over time).
          • Such an imposed organization makes for an easier read of the proposal, and should also make it much easier for you to structure both your motivation and research question sections.
          • A well-done previous work section may also be a (possibly publishable) research contribution in its own right, e.g., my MSc thesis.
        • Ideally this section should be structured to highlight that which has not been done previously which you plan to do -- this gives a natural bridge to the following sections in the proposal. Where possible, consider making this structure a story or narrative, e.g., "Once upon a time there was a very important problem. Many people have worked on this problem so far, but they haven't done important thing X. However, I have a neat plan to do X -- would you like to hear about it? If so, please keep reading ..."
      • Research Question [~1 page]
        • Describes what specific question or set of questions you will be addressing in your thesis work.
        • A research question is intermediate in rigor between a formal scientific hypothesis and an everyday question.
        • The precise form of the research questions depends on the research topic; are much easier to define if the research was inspired by open questions in the literature.
        • Example: The two general research questions underlying all of my parameterized-complexity-based research relative to an problem X:
          1. Can X be solved optimally in polynomial time in general?
          2. If not, what restrictions do (and do not) make X solvable in effectively polynomial time?

        • Every question should be well-motivated by the preceding proposal sections and be answerable relative to the subsequently-described methods to be employed in the research; as such, these questions are the central pivot on which all else in the proposal depends and operates.
      • Methodology [~2-4 pages]
        • Describes how you will answer the research questions.
        • Methods are typically described in general terms, with appropriate literature citations for details; that being said, these descriptions must be specific enough to allow explanations of how these methods can be used to answer the posed research questions, i.e., no ``magic occurs here'' processes.
        • Note that this section and the following one are the ones where you convince others that the proposed research is doable by you.
      • Timeline [~1 page]
        • Describes your anticipated course of progress in the proposed research.
          • The emphasis here is on ``anticipated'' -- this is research, not engineering, and experienced researchers know things seldom go according to plan.
        • Can do sophisticated multi-tier graphics here; however, the simplest form (an itemized list consisting of month-specific dates with attached research activities and/or deliverables) is often sufficient.
        • The timeline must be believable -- to ensure this, be as specific as possible when describing research activities, e.g., no ``magic occurs here'' processes, and budget for some difficulties when establishing dates, e.g., The Scotty Principle.
      • Bibliography (15-30 references) [~2-3 pages]
        • Bibliography should be alphabetic by author name, and all references should be both complete and correct in details and consistent in format, e.g., do not describe conference papers in two or more ways.
        • Compiling a good bibliography is a lot of work, but it plays an important role in convincing others that you can be trusted to do research; it can also be a research contribution in itself, e.g., my MSc thesis.
    • Example: My PhD research proposal [3rd draft] (PDF)
    • Remember that you need not write up the sections in the order in which they appear in the proposal -- in order to establish originality and importance, Related Work, Research Questions, and Bibliography should be done first, followed by Methodology (to establish do-ability). Once you then know where you'll end up story-wise, one can then do your Motivation and Timeline.
      • Allow time to revise your proposal after the first draft of all sections are finally written -- at that point, you MUST adjust story and terminology to be consistent throughout the proposal.

Tuesday, January 23 (Lecture #4)

• Starting Research: Timelines and Proposals (Cont'd)
  • The general structure above is also characteristic of funded research proposals modulo differences in the length of particular sections and the presence of additional sections, e.g., personal record in previous funded research and the training of Highly Qualified Personnel (HQP), budgets, budget justifications.
    • Note that in the case of proposals involving funding, the extent to which eventual research must follow that described in the proposal varies with the funding agency and grant-type.
    • Example: OPC / NSERC equipment / Dutch PhD project proposals vs. NSERC Discovery Grant proposals.
  • Example: My 2010 NSERC Discovery Grant proposal (PDF)
  • Wrt non-funded research projects, it may well be worth doing up specific sections described above (in particular, previous work, research questions, and bibliography) to help both focus the research from the beginning and aid in creating subsequent publications describing the results of this research (which will require these materials anyway).
  • In all cases, be honest in describing all aspects of your proposal, particularly the extent to which the proposed research is original and important and your ability to do it.
    • Part of this can be including frank descriptions of potential difficulties and how you will deal with them.
    • Given the temptation and incentives to over-sell research by being less than honest, honesty can set one apart in a good way and be reputation-enhancing.
    • Example: Reviewer comment on my 2010 NSERC Discovery Grant proposal.
• Doing (and Surviving) Research
  • There is nothing like either the joy of research going well or the despair of research going badly. Though one can never be quite sure what research will bring result-wise, one can with care and forethought simultaneously maximize the former and minimize the latter.
  • There are many techniques for carrying out computer science research -- these techniques are what one picks up in courses and personal reading and from talking to your supervisor and other academics.
  • We will focus here not so much on the research process but on how to survive and enjoy this process.
  • General Research Etiquette
    • There are some basic common-sense rules that should always be followed when doing research:
      1. Be honest in performing research and reporting results.
      2. Acknowledge the contributions of others to your research, e.g., do not (excessively self-)plagiarize.
      3. Where possible and prudent, help others in their research.
      4. Be courteous and respectful to research support staff

      We are human, and lapses will occur (particularly wrt (3) and (4)); however, be on guard for lapses in (1) and (2) ((1) in particular), as much of research depends on the trustworthiness of researchers and such lapses can destroy research careers.

    • Though life in general may not be fair, aim to create a peaceable kingdom wrt your research and the people you deal with in doing that research.
  • Graduate Thesis Research
    • The Thesis Lifestyle
      • The Pace of Thesis Research
        • Work hard and work consistently, i.e., try wherever possible to avoid last-ditch 24/7 work binges.
        • Quite aside from keeping your supervisor happy (by allowing regular demonstrations of research progress), this is an excellent strategy for ensuring that you always have additional time to cope with unanticipated difficulties
      • The Necessity of Not Doing Thesis Research
        • Make sure (if you have to, by scheduling it) that you have regular breaks away from your thesis doing something that you find relaxing and enjoyable.
          • This not only includes daily and weekend breaks but vacations, e.g., visits home.
        • This helps give you both the energy to do good research and necessary perspective on what you are doing research-wise.
        • Oddly enough, the above applies even when research is going very well -- the endorphin rush of good research can be seductive and potentially damaging, e.g., death by joy.
        • Example: My 2015 Summer (Non-)Vacation
      • The Necessity of Setting Personal Priorities wrt Work
        • There are plenty of people out there who are more than happy to tell you how much you should work (and conversely how little you should be away from work).
        • Time spent working has both benefits, e.g., faster thesis finishing time, more publications, and drawbacks, e.g., broken relationships, personal depression / burnout (see below).
        • Example: Cases in point: Dan Brooks, Tao Jiang, and Bill Threlfall.
        • You need to decide what your priorities are and choose the tradeoffs between work and life that are right for you before others do it for you.
        • During graduate studies, this balance is particularly hard to strike given the unequal distribution of power between students and supervisors; this is yet another reason why it so important to maintain cordial and open relations with one's supervisor (see below).
      • All of the above can benefit by making sure that you are good at allocating and managing time both to research and life in general -- anything that helps in this is to the good, e.g., training in Time Management / Meditation / Mindfulness.

Tuesday, January 30

• Instructor ill; no lecture.

Tuesday, February 6

• MUN closed; no lecture.

Tuesday, February 13 (Lecture #5)

• Doing (and Surviving) Research (Cont'd)
  • Graduate Thesis Research (Cont'd)
    • The Care and Feeding of Graduate Research Supervisors
      • Ideally, you should have a good intellectual and personal fit with your supervisor, i.e., they should know the things you need to do research and you should be able to talk to them.
        • The former helps your research go forward; the latter helps when your research has problems.
      • Ideally, you should have regular (at least weekly or bi-weekly) meetings at which you discuss research (and later thesis) progress.
        • Regularly-scheduled meetings are a great way to keep up the pace of research.
        • Try not to dread these meetings -- if there has been no progress, use the meeting to figure out why this occurred and what can be done about it.
      • As in any relationship, if problems arise between you and your supervisor, talk them out at the earliest possible opportunity when at all possible.
      • Student-supervisor relations are seldom perfect, and the causes of deviations from the ideal as well as the degree to which these deviations are problematic vary wildly (Tolstoy's Dictum)
        • If your supervisor is bureaucratically challenged wrt regulations and deadlines, make sure that you know what the relevant regulations and timelines are and make sure that you follow them.
        • If your supervisor is not making time to meet with you or to give you timely advice and feedback, do not let them off the hook -- their time is your right as their student, and patiently but firmly request that they make time for you.
        • If your supervisor is not a good intellectual or personal fit, get the help you need elsewhere, e.g., supervisory committee members (PhDs), other faculty members (departmental and otherwise), fellow students of the same supervisor, fellow students, friends and family.
      • If there are irreconcilable differences between you and your advisor or the relationship between you becomes abusive, talk to your graduate officer immediately for advice.
        • Though there may ultimately be little that can be done to compel a tenured supervisor to behave, not letting others know what is going on will only allow things to get worse; moreover, the assistance of these others will be necessary if you need to switch supervisors.
        • Switching supervisors is a common fantasy in times of trouble, and it can be done and survived -- however, it is even in the best situations disruptive and should only be invoked for real if there is no other solution.
    • Writing Up the Thesis
      • Theses take different lengths of time to write -- perhaps the only common feature is that all theses take longer to write than you think they will. Hence, start thesis writeup as early as possible and budget time for problems that can arise.
      • Decide on your typesetting system up front.
        • The main choices in Computer Science are MS Word and LaTeX, with research areas varying in their preference.
        • Make sure that the conventions you adopt, i.e., for tables and figures, citations, and bibliographic references, are consistent with those mandated by your university.
        • Example MUN Thesis Guidelines (webpage)
        • Thesis templates (when available) can be invaluable.
        • Example Unofficial MUN Thesis Templates (LaTeX)
      • The standard thesis format looks strangely like that for the thesis proposal (introduction, background (including previous work), methodology, results, discussion, conclusions, bibliography).
        • This standard format van vary according to research area, e.g., software engineering vs. evolutionary algorithms vs. computational complexity analysis vs. theoretical computer science.
        • Your supervisor is typically your best guide to which thesis format you should use.
      • Theses need not be written up in either the final order of sections or at the final level of detail.
        • Consider first doing brief background and methodology sections and only fill these in with details after you have written up your results and discussion (at which point you will know the details you need to put into the background and methodology). Conclusions and the introduction can be done last.
        • As terminology and story are especially prone to vary with time over long-duration writeups like theses, allocate time at the end to make both consistent throughout.
        • Never make the bibliography the last thing you do -- always add all references in full as you cite them in the thesis drafts.
          • Any time you spend adding full references that are later deleted is typically vastly less than the you spend trying to recover references that you forgot.
          • The above is also a good reason why you should never wait until the text is finalized to add literature citations.
      • Make sure you acknowledge text, tables, and/or figures in your thesis that are derived or taken from other sources.
        • Small pieces of text, e.g., 1-2 sentences, can be placed in thesis text by enclosing them in quotation marks and giving a literature citation with page numbers at the end. Such page numbers can be done in LaTeX using the optional square-bracket argument for the \cite command, e.g.,
          This approach is not commonly used, in large part because of the opinion of practitioners such as Wareham who claim ``... that the whole thing is nonsense'' \cite[page 23]{War01}.

          Larger pieces of text should be offset from the thesis text, again with a literature citation with page numbers at the end. Such offsetting can be done in LaTeX using the quotation or quote text environments.

        • If tables or figures are adapted from other sources, e.g., the table or figure is simplified or enhanced, it suffices to acknowledge this in the caption, e.g.,
          \caption{A simplified model of a neuron (adapted from \cite[Figure 4]{SNW98}).}

          If the table or figure in your thesis is exactly the same as it appears in the source, e.g., you've re-used the JPEG file from the source, remove "adapted from" in the caption acknowledgment above and get written permission from the author and/or publisher to re-use the table or figure. Failure to do this can result in the rejection of your thesis on copyright grounds.

        • Given the severity of the repercussions for not handling such matters correctly, if you are at all in doubt about how re-use should be acknowledged wrt thesis text and obtained permissions, consult your supervisor and the MUN copyright office (webpage).
    • Solicit feedback from your supervisor and/or supervisory committee on individual chapters (and, when they are critical, individual sections) as they are written, i.e., avoid Big Bang Writeup.
      • This ensures that you do not drift too far from standard formats and/or supervisor preferences, and can save a lot of work.
      • This also allows early detection and resolution of format disagreements between co-supervisors as well as simplifying final thesis checking before submission for examination (see below).
    • Theses at MUN have been rejected because of poor written English. Hence, if you need help with your written English, get it.
      • Some supervisors provide this; in other cases, use the Writing Center (webpage) or hire an editorial consultant.
      • The Writing Center is free but may take a while; editorial consultants are not free but the benefit of feedback that is speedy (and possibly better-suited to you and your thesis) may make it worthwhile.
      • Some people do this throughout the writeup process while others only do it for the final draft prior to thesis submission -- I prefer the former, if only because Englishification of a large document all at once can take a lot of time.
    • Be aware of your intellectual property rights as regards your thesis.
      • Example: MUN thesis intellectual property rights (Webpage)
      • Depending on circumstances, you may not want to put everything you know or derive in your thesis.
      • Example: My 1990 MUN M.Eng. Experience.

Thursday, February 15

• University closed; no lecture.

Tuesday, February 20

• Midterm break; no lecture.

Thursday, February 22

• Midterm break; no lecture.

Tuesday, February 27

• Instructor ill; no lecture.

Thursday, February 29 (Lecture #6)

• Doing (and Surviving) Research (Cont'd)
  • Graduate Thesis Research (Cont'd)
    • Thesis Submission, Examination, and Defence
      • The process of submitting, examining, and defending theses takes different lengths of time -- perhaps the only common feature is that this process always takes longer than you think it will. Hence, you need to start this process as early as possible and budget time for problems that can arise.
      • The Official Timeline (MSc):
        1. Examiners selected and appointed by SGS Dean (1/2 month)
        2. Formal submission and examination (1-2 months)
        3. Notification of results (acceptable, minor revisions, major revisions, fail)
        4. Final submission to MUN (6 months for minor revisions; 12 months for major revisions and re-examination)

        Remember also that a departmental seminar must be given immediately prior to (2).

      • The Official Timeline (PhD):
        1. Review of thesis by supervisory committee (1-2 months)
        2. Examiners selected and appointed by SGS Dean (1/2 month)
        3. Formal submission and examination (1-2 months)
        4. Notification of results (acceptable, minor revisions, major revisions, fail)
        5. Defence and final submission to MUN (oral defence (+1 month), then 6 months for minor revisions; 12 months for major revisions and re-examination)

      • The official bottom line is to plan on 1-2 semesters to complete the process. However, for various reasons, it can take longer, e.g.,
        • Your supervisory committee may be away and/or busy (the latter during term, the former over holidays (Xmas, summer))
        • Paperwork requiring multiple signatures among multiple offices takes time (especially if you are trying to get it done between semesters or during holidays (late summer, Xmas)).
        • Examiners may be busy or unresponsive.
        • It may be problematic scheduling a multi-person defence (though this is much better now with e-defences).
        • Your own university paperwork may not be in order, e.g., outdated program of study, unpaid fees.

        Given the above, the best way to avoid such additional delays is to first make sure that all your paperwork is in order and then initiate the submission process early in the Fall or Winter terms.

      • The co-operation of supervisors and your supervisory committee is crucial during this phase of your thesis; hence, all advice for dealing with supervisor problems given above goes double here.
        • Students of bureaucratically challenged supervisors need to take particular care and be prepared for the non-standard, e.g., helping to prepare your own list of examiners (!).
      • Though this is a complex and scary process, don't panic -- if you have put in the requisite work and care, things will be okay.
        • In particular, the PhD defence is often an anti-climax -- my own experience and that of many others I've talked to is that afterwards, one's second reaction (after the initial "Yahoo!") is "What was I so worried about?"
        • Use as much stress as you need to make sure that you are prepared -- any stress after that is superfluous.
    • The Eternal Question: To Teach or Not to Teach?
      • This is more an issue for PhD than MSc students; however, it is something everyone should consider if they are looking at a career in academia.
      • Benefits: Teaching a course not only helps you decide if you can and want to teach, but it is also an invaluable enhancement of your CV when you go to apply for academic jobs.
      • Drawbacks: Teaching well (especially for the first time) is very, very time-consuming, and can easily (and seriously) delay your thesis research and writeup.
      • Example: Patricia Evans and myself.
      • Given the above, whether or not you should teach during your graduate studies must be discussed and cleared with your supervisor.
      • If you do decide to teach, it would be well worth your time to build up some expertise by taking a a course on teaching if one is available, e.g., TSEP..
    • Dealing with Graduate Research Burnout
      • Not to confused with End-of-Thesis Psychosis (ETP), e.g., OCD + paranoia + manic/depression, which everyone gets in the high-stress situation that is the end of a thesis.
      • Burnout is much more severe and has a variety of symptoms; what is most characteristic is long-term depression and lack of research and/or thesis progress.
      • There are a variety of causes of burnout, e.g., bad choices (topic, scheduling), bad luck (research, accidents, family, relationships)
      • Example: My BAh thesis burnout.
      • Example: My PhD thesis burnout.
      • Tips for averting burnout
        • Work hard and consistently to cope with unanticipated difficulties.
        • Maintain good and regular interactions with your supervisor.
        • Schedule and religiously take downtime.
        • Keep a progress diary (and read it).
      • What to do (and not to do) when burnout happens.
        • DO recognize the signs as early as possible.
        • DO ask for the help you need (and use it).
          • Depending on the causes and severity of your burnout, this help may involve counseling, therapy, and/or drugs.
          • Depending on the causes of your burnout, talk to the appropriate people, be it your supervisor, friends, or family, or mental health professionals, e.g.,
            • MUN Student Wellness and Counseling Center (webpage): 1-844-451-9700
            • Mental Health Emergency Services - Eastern Health (webpage): 737-4668
        • DO NOT be ashamed.
        • DO NOT ignore it.
  • Non-thesis research.
    • You will typically have more choice about what you do and a longer time in which to do it after you complete your theses, but depending on your job, you may have much less time in which to do it than you did during your theses, e.g., my niece's evaluation of my academic working life in 1999.
      • Good habits wrt managing your time and setting priorities may be even more crucial after your theses than they were during your theses.
      • This is yet another reason why it is a good idea to develop good habits wrt such matters during your theses.
    • The Eternal Question: To Collaborate or Not to Collaborate?
      • After (and to a degree, even during) your theses, you have the option of collaborating with others in doing research. One's first experience is most often collaborating with your supervisor and your fellow supervised grad siblings; this broadens later to others.
      • There are a variety of types of collaboration, varying for example in the number, diversity of expertises and/or geographical locations of the people involved and the duration of the collaboration.
        • Larger (especially geographically distributed) collaborations require much more management and co-ordination than smaller collaborations.
        • Collaborations involving diverse expertises require initial and ongoing establishment of common ground that may not be required if all collaborators are in the same research area.
        • Long-term collaborations require a much better intellectual, work-habit, and personal fit between you and your collaborators than short-term collaborations.
      • Benefits: Research can go faster and, with the talents of others, more robustly and in far more directions then you could have done on your own.
      • Drawbacks: Relationships with collaborators must be carefully co-ordinated and nurtured to result in productive and enjoyable research; writeup must also be co-ordinated very carefully, e.g., who has The Writing Stick for which part and who sets the overall tone and style of the writeup?
      • Like many other types of relationships, collaborations do not always work and even if they do, they may not work for long.
      • Example: My own collaborative relationships with Marco Cesati, Gianluca Della Vedova, Patricia Evans, Yvan Rose, and Iris van Rooij.
      • Whatever the form of collaboration, establishing who will be authors on any produced work and what the author order will be is very important.
        • This is because author order means different things even within subareas of the same discipline let alone in different disciplines and these things can have an impact on co-authors getting tenure and/or grants,
        • Example: The 1992 Wood's Hole biologist conundrum.
        • Example: Theoretical vs. non-Theoretical Computer Science.
        • If you are a graduate student, be aware that there are no standard guidelines wrt co-authorship with your supervisor; hence, when you start producing papers, it is worth having a frank chat with your supervisor about this.
        • Example:: Mike Fellows, Ulrike Stege, and Ken Kent.
        • There are benefits and drawbacks to every co-authorship policy. and there will always be people who will tell you what you should and should not do. It is therefore important for you to (as with our previous discussion on establishing work / life priorities) decide early on what is right for you before other people do it for you.
      • Though collaboration (particularly between disciplines or with industrial partners) is very strongly encouraged these days, you must ultimately decide what is right for you and the research life you want to lead.
    • Unfortunately, burnout is not restricted to theses but can occur at any time during research; hence, all advice on preventing and dealing with thesis research burnout given above also applies here.

Tuesday, March 5 (Lecture #7)

• Once one has done research, it is a good idea to write it up for publication.
  • This is important because publications are a measure of research success in academia -- your publications help you get funding and PhD offers when you are a graduate student and job offers, funding, tenure, and promotions when you become a professor.
  • There is hence a great pressure to publish as many papers as possible, and to thus break your research into Minimal Publishable Units (MPUs), each of which is a separate paper. Some people do this by first publishing an MPU as a conference paper (usually omitting details such as proofs) and then publishing it again as a journal paper (this time with details).
  • While the MPU approach is sometimes seen as necessary, MPUs do not get a lot of respect and if over-used my harm your academic reputation -- hence, their use should be balanced with taking longer to create papers that tell complete stories wrt a research problem (the Shannon approach).
  • What makes an MPU varies with academic discipline and subdiscipline, and is best figured out by chatting with your supervisor and other researchers in your area.
• So, you have written a paper -- the natural question is, where and how do you publish it? This is the topic that will take up the remainder of our course lectures.
• Presenting Research I: The Joy of Peer Review
  • Peer review is the process by which research work submitted for academic publication is evaluated.
  • The peer review process:
    • A piece of research work is submitted to some venue (e.g., conference, journal, edited book) for publication and possibly presentation (oral or poster). The submitted work may or may not be author-anonymized.
    • The venue chair sends the submitted work out for evaluation to two or more reviewers, who produce both numerical scores assessing aspects of the quality of and appropriateness of the submission and (possibly separate) sets of comments for both the authors and the venue chair. These reviews are typically unsigned.
    • The venue chair uses the reviews to determine whether or not the submitted work is accepted for publication in the venue.
    • The venue chair's decision and the reviews are sent to the authors of the submitted work.
      • If the decision is acceptance, the authors make any suggested revisions and submit the revised version of the work for publication.
      • If the decision is rejection, depending on the venue, the authors may be able to give rebuttals to the reviewers and/or submit a revised version for re-evaluation.
  • Reviewer anonymity is a crucial part of the review process, in that it allows criticism without fear of reprisal, e.g., senior academics denying tenure and/or publication to junior academics who criticize them.
  • Author anonymity is desirable in that it allows criticism unbiased by the reputation (good or bad) of the authors; however, its implementation in peer review varies widely across research communities and venues.
  • Purposes of peer review:
    • To help the venue chair decide whether or not the submitted work is worthy of publication in the venue.
    • To provide constructive feedback to the authors of the submitted work.
    • To give the reviewer the opportunity to (1) be of service to their research community and (2) give evidence of such service in their CV.

    The last of these is the reason why many supervisors encourage their graduate students to do peer reviews.

  • Given the first two of the purposes above, an ideal review should give a concise summary of the reviewer's opinion of the worthiness of the submitted work along with reasons for this opinion (to aid the venue editor in making their decision) as well as clear, constructive, and detailed comments and revision suggestions (to aid the authors in producing the best possible version of the work).
    • The editor-summary can be in the front of the review and consist of an initial paragraph briefly summarizing the work's contributions, one or more following paragraphs giving a concise assessment (with reasons) of the originality and worthiness of the work, followed by a brief recommendation of acceptance or rejection for the work.
      • Depending on the software system or text protocol used to do reviews, this summary may also include numerical scores provided by the reviewer (typically on a 1-5 scale) assessing aspects of the submitted work (e.g., originality, correctness, relevance to the venue, quality of writing) as well as the review process itself (e.g., familiarity with the research area of the submitted work).
    • The author-comments can follow after the editor-summary and consist of a list of general overall suggestions followed by a list of specific comments; the latter should always use a consistent and precise scheme to localize the part of the submitted work to which the comments apply, e.g., Page 5, Section 2.3, Paragraph 2, line 3.
  • Example: Conference paper reviews (COCOA 2008)
  • Example: Journal paper review (TCBB 2006)
  • Guidelines for doing reviews
    • Spend an appropriate amount of time reading the submitted work, e.g., 30 minutes -- 2 hours for a conference paper and a day for a journal paper.
    • Spend an appropriate amount of time writing up your review, e.g., 1 -- 3 hours for a conference paper and a day for a journal paper.
    • You do not have to understand and verify that absolutely everything in the submitted work is correct to give a useful evaluation. This is especially important to remember if you are reviewing a paper that is not in your area of expertise.
    • You are not responsible for providing an exhaustive and definitive list of suggestions that will produce a perfect version of the submitted work -- rather, the aim to help the authors as much as you can within the limits imposed by a reasonable investment of your time.
    • Always be specific and concise in and provide clear justifications for all points raised in your comments to the venue editor.
    • Always be constructive and specific in your comments to the authors.
    • Always phrase your review as if you signed it, i.e., don't say anything anonymously that you would not be prepared to say and defend to the author's face.
    • Always respect deadlines for returning reviews.

Tuesday, March 12 (Lecture #8)

• Presenting Research I: The Joy of Peer Review (Cont'd)
  • Guidelines for dealing with reviews
    • Read reviews in a calm and supportive environment, e.g., the University of Victoria Rabbit House, a cocktail bar.
    • After the initial reading, put the reviews aside for a day to give your emotions time to settle; when you come back to them, decide in consultation with your co-authors (if any) (1) what comments are sensible and constructive and (2) what you are going to do about them.
      • Venues vary in how closely they require that you take review suggestions into account when revising a submitted work for publication, e.g., EpiRob vs. CGAMES.
    • Whatever the decision, try to (eventually at least) be as calm as possible.
  • Example: The submission history of my game agent playability paper with Scott Watson:
    • CIG 2014 (Submission/ Reviews)
    • CCAI 2015 (Submission/ Reviews)
    • CGAMES 2015 (Submission/ Reviews (Review #1/ Review #2)/ Published Version)
  • Sometimes, one has the option of multiple rounds of paper revision with the same venue (be it a conference or journal).
    • In such cases, along with the revised paper, one must also submit a response letter detailing how each reviewer comment has been addressed (or why it has not been addressed).
    • Such response letters must be phrased very diplomatically.
    • Sometimes, this works; sometimes, it doesn't.
  • Example: Revision response letters for the three rounds of revision on Bozorgi et al (2017) for the journal Knowledge-based Systems.
    • Letter #1 (2016) (PDF)
    • Letter #2 (2017) (PDF)
    • Letter #3 (2017) (PDF)
  • Example: My 2017 JIRS Experience (Revision response letter text).
  • Example: Revision response for the single round of revision on Wareham et al (2023) for the journal ACM Transactions on Human-Robot Interaction.
    • Reviewer comments (txt)
    • Response to reviewers (txt)
    • Revised paper (PDF)

Tuesday, March 19 (Lecture #9)

• Presenting Research II: The Joy of Conferences
  • Benefits: Publishing at conferences is not only a great way to get quick feedback on your research (especially if it is still in progress) but it also gives you a chance to meet and converse with other researchers (which can be useful in many ways, e.g., getting advice on research, meeting potential research collaborators, furthering research with existing collaborators, establishing or maintaining your presence in your research community).
  • Drawbacks: Conference travel costs money and can take time away from the research process and conference publications, while valid CV entries (even if the conference is refereed), are not valued as highly as journal papers or chapters in edited books.
  • There are typically a number of conferences -- how do you choose which one is best for you and your work? There are a variety of reasons for choosing a conference, e.g.,
    • conference topic (the conference topic is either the same as your work or a topic you want to know more about)
    • conference attendees (the conference is attended by people you know or want to know)
    • conference location (the conference is somewhere close by (and hence cheap to travel to) or somewhere you want to go (possibly for non-academic reasons))
    • conference date (the conference fits in with other travel you are doing (so you can get more bang for your travel dollar) or the conference proceedings will appear in time for you to cite any accepted paper(s) in an upcoming grant application)
    • conference acceptance standards (the conference accepts few or lots of papers)

    There is typically more than one reason for picking a particular conference, and it is quite acceptable for you to weight reasons according to your priorities when making a decision.

  • An excellent source for conference candidates is the list of conferences that published papers that you cite in your work; another is your graduate supervisor.
  • Where possible, publish in reputable conferences.
    • Conferences whose submissions are refereed are to be preferred over those that are not.
    • Conferences that publish proceedings in accepted series, e.g. Lecture Notes in Computer Science (Springer), are to be preferred over conferences that self-publish proceedings.
    • Conferences sponsored by established Computer Science organizations, e.g., ACM, IEEE, are to preferred over conferences that are not.
  • Once you have selected a conference, download its typesetting template immediately and use it to prepare your paper from as early as possible; this prevents last-minute formatting problems from derailing your submission.
  • Even if your paper is accepted, there may be varying grades of acceptance, e.g.,
    • Paper accepted for oral presentation with publication of full paper in proceedings.
    • Paper accepted for poster presentation with publication of full paper in proceedings.
    • Paper accepted for poster presentation with publication of abstract in proceedings.

    As posters and abstracts are not as valued as talks and full papers, think seriously about whether it is worthwhile to attend if you are offered the second or third alternatives.

  • You can list accepted conference papers in your CV (and some grant applications) before the conference proceedings is published -- mark them as "to appear", e.g.,
    Wareham, T. (2016) Some notes on conference submission. To appear, 16th International Conference Conference (ICC 2016).

    As soon as the proceedings are published, fill in the full details in your CV accordingly.

  • Book conference travel and hotels as early as possible to make sure you get what you want and need, e.g., travel dates and/or hotels close to the conference site, for a reasonable price.
    • If the conference is held in a hotel, you sometimes get a reasonable rate as a conference attendee; if not, consider staying at a reasonably-priced (and possibly better quality) hotel nearby.
    • Example: The Fellows LA (Non)Conference Hotel Experience.
    • Apply for student travel grants if the conference offers them -- these help defray costs (leaving money for other conference travel) and can also be nice CV entries in themselves.
  • Conference attendance guidelines
    • If you are a graduate student, travel if possible with your supervisor, especially if your supervisor knows people at the conference -- this is a great way to get introduced to people, e.g., my experiences with Bill Day and Bill Gasarch. If you cannot travel with your supervisor, see if they can arrange for you to meet some of their contacts at the conference.
    • You do not have to attend everything scheduled at the conference -- look at the schedule before the conference and at the end of each conference-day to ensure that you attend the events that are most useful to you (and have backup events in place if something that looks interesting turns out to be a dud). This is particularly crucial at large conferences with parallel event-tracks.
    • Lots of stuff gets done at conferences outside of scheduled events, e.g., around snack-break tables, at meals; stay alert for such opportunities and balance them up against the scheduled events.
    • By all means network but don't be pushy about it -- some people, e.g., myself, do not react well to being obviously (even if gently) networked.
  • Finally, saving monmey on hotels and travel is a laudable goal but never compromise on safety.
  • Example: My AAAI 1996 Vancouver Bus Station Experience.
• Presenting Research III: The Joy of Journals
  • Benefits: Journal papers are highly-valued contributions to your research community and CV and are the accepted venue for distributing completed research.
  • Drawbacks: Journal publication can take a long time (especially if the paper is rejected by one or more journals) and there may be unanticipated monetary costs, e.g., open access fees.
  • There are typically a number of journals -- how do you choose which one is best for you and your work? There are a variety of reasons for choosing a journal, e.g.,
    • journal topic (the journal topic is the same as that of your paper)
    • journal readership (the journal is read by people who you want your work to be read by)
    • journal publication costs (if any)
    • journal acceptance standards (the journal accepts few or lots of papers)
    • journal review and publication timeline (the journal promises quick review and publication of papers)

    There is typically more than one reason for picking a particular journal, and it is quite acceptable for you to weight reasons according to your priorities when making a decision.

  • An excellent source for journal candidates is the list of journals that published papers that you cite in your work; another is your graduate supervisor.
  • Where possible, publish in reputable journals.
    • Journals sponsored by established Computer Science organizations, e.g., ACM, IEEE, and/or journal publishers are to preferred over journals that are not.
    • Measurements of journal importance like impact factors can also be used to judge journal reputability.

  • Once you have selected a journal, download its typesetting template immediately and use it to prepare your paper from as early as possible; this prevents last-minute formatting problems that may hold up submission (or publication if the paper is accepted).
  • There are three possible responses from the journal:
    • Paper is accepted (possible subject to minor revisions)
    • Paper requires major revisions and resubmission
    • Paper is rejected

    In the first two cases, make the suggested revisions promptly and resubmit; this resubmission may include a letter to the editor giving details of changes you made as well as responses to the comments / questions posed by the referees. Be aware that in the second case, there may be several rounds of revision, and acceptance is never guaranteed. As the for the final case, if this happens, breathe deeply, count to ten, and (unless irrevocable flaws in your work have been pointed out by the referees) prepare to revise and resubmit that work elsewhere.

  • The advice to make revisions and resubmit promptly in the first two cases above is crucial -- to do otherwise can dramatically prolong the time to publication.
  • Example: The publication history of van Rooij and Wareham (2012).
  • You can list journal papers in your CV (and some grant applications) when the papers have been submitted, when they are being revised, and when they have been accepted but not yet published. There are a variety of ways of doing this format-wise, e.g.,
    Wareham, T. (2016) Some notes on conference submission. Submitted to The Journal of Conferences.

    Wareham, T. (2016) Some notes on conference submission. Under revision for The Journal of Conferences.

    Wareham, T. (to appear) Some notes on conference submission. The Journal of Conferences.

    As soon as the paper is officially published, fill in the full details in your CV accordingly.

References

• Bozorgi, A., Samet, S., Kwisthout, J., and Wareham, T. (2017) "Community-based Influence Maximization in Social Networks under a Competitive Linear Threshold Model." Knowledge Based Systems, 134, 149-158.
• Barik, T., Gulwani, S., and Juarez, M. (2022). "Storytelling and science." Communications of the ACM, 65(10), 27-30. [PDF]
• Wareham, T. (2012) "Flyby: life before, during, and after graduate studies with mike fellows. In The Multivariate Algorithmic Revolution and Beyond: Essays Dedicated to Michael R. Fellows on the Occasion of His 60th Birthday. Springer; Berlin. 51-55. [PDF]
• Wareham, T., de Haan, R., Vardy, A., and van Rooij, I. (2023) "Swarm Control for Distributed Construction: A Computational Complexity Perspective." ACM Transactions on Human-Robot Interaction, 12(1), 6:1-6:45.
• --- (2023) "How To Ethically Leverage Artificial Intelligence (AI) in Your Academic Research as an Undergraduate or Graduate Student". [Text].

Created: October 17, 2023
Last Modified: March 3, 2024