Week 1
Thursday, January 5 (Lecture #1)
- Introduction to CS 690B: Content and Logistics (Slides)
- The Research Process: The Ideal vs. (Bitter) Reality
- The reality of research is that many false starts may happen at any point in the process; sometimes, one may even have to backtrack to an earlier stage in the process and begin again.
- Though one cannot ever eliminate such false starts, one can with forethought and appropriate effort minimize their occurrence and make the research process not only shorter but more enjoyable.
- Starting Research: Selecting and Validating Topics
- This is a crucial part of both thesis and general research.
- Four types of research topics:
- Verification of existing work (Old Bones, Old Graveyards (OBOG)),
e.g., rerunning or redoing a simulation study (the latter
if the original code is missing) to see if the same results
are obtained.
- Given that reproducibility (and hence trustworthiness) is a core aspect of science, this is sometimes necessary.
- Given the importance of the original work being reproduced and the existence of at least a partial description of how this work was done, such a topic can be both important and doable; however, it is not original and hence often does not get the respect it deserves (and hence may lose importance).
- These are low-risk research topics which, given their frequent though undeserved lack of respect, are relegated to BSch theses or "housekeeping" aspects of academic research in general.
- New application of existing methods (Old Bones, New Graveyards
(OBNG)), e.g., applying standard genetic algorithm
design techniques to develop a genetic algorithm for improving
the prioritization of orthodontic hamster surgery cases.
- Such topics are typical of basic ongoing scientific research (cf. Kuhnian Scientific Revolutions) and hence are the most common research topics.
- By virtue of using existing methods, such work is doable and has importance and originality in proportion to the importance and originality of the application area.
- These are low- to moderate-risk research topics, and hence typical of BSch and MSc theses; are also useful in academic research in general when results must be obtained reliably and quickly, e.g., conference / funding proposal deadline.
- Development and verification of new methods (New Bones, Old
Graveyards (NBOG)), e.g., developing either a new
type of genetic algorithm or a totally new type of algorithm
period.
- Depending on the degree of novelty of the new method, such topics are typical of advanced ongoing scientific research and minor Kuhnian Scientific Revolutions and hence are somewhat rare.
- Such topics are by definition original and important in
proportion to the advantages and applicability of the
new methods; however, such work may not be doable or
very difficult to do.
- In order to verify the advantages and applicability of a new method, it is critical to apply this method to applications to which existing methods have already been applied and hence which have results against which the new method can be compared.
- These are moderate- to high-risk research topics (in proportion to the novelty of the new methods) and are hence typical of MSc and PhD theses as well as academic research in general; are most productively tackled when there is no hurry to obtain results, e.g., when you have several years grace to work on a PhD or after you have tenure.
- Simultaneous development of new methods and new applications
(New Bones, New Graveyards (NBNG)), e.g., atomic and
relativistic physics.
- Such topics are typical of major Kuhnian Scientific Revolutions based on previously unknown phenomena, e.g., radioactivity, and hence are very rare.
- Such topics are by definition original and important; however, they may not be doable or very difficult to do.
- These are very-high-risk research topics, i.e., nice work if you can get it.
- Verification of existing work (Old Bones, Old Graveyards (OBOG)),
e.g., rerunning or redoing a simulation study (the latter
if the original code is missing) to see if the same results
are obtained.
- Two sources of thesis topics:
- Supervisor-selected
- Risk inversely proportional to the degree to which the supervisor knows both the topic area (and hence the topic's originality and importance) and your ability to work productively on the topic.
- Can be completed quickly and is very good for learning particular methods; however, may be a tad boring at times.
- The most typical type of BSch and MSc topic.
- Self-selected
- Much riskier, as you have to evaluate originality, importance, and doability by yourself.
- Is typically time-consuming and a somewhat more terrifying experience; however, one learns a lot more, including how to be an independent researcher.
- The most typical type of PhD topic.
- Supervisor-selected
- Two sources of research topics in general:
- Open questions in existing work
- Such questions may be published in the literature or circulating by word-of-mouth among the members of a research community; access the former by literature search / reading and the latter by talking to colleagues at your home university and at conferences.
- Depending on the reliability of the source, topics based on such questions are important and original, and there may already be hints on how they might be doable.
- Note that the effort involved in phrasing a good open question, i.e., one that is original, important, and doable, often makes people (IMHO rightly) reluctant to share them unless credit for the effort is acknowledged by co-authorship or some other means, e.g., Fellows and Berger 1995 conversation.
- Combination of elements of existing work
- Such combinations arise from the fortuitous collision within your mind of disparate ideas that you have encountered, e.g., why hasn't anyone thought of cooking mushrooms with a blowtorch before?; as such, one can increase the probability of such combinations by creativity-boosting mental exercises as well as by increasing the number of ideas you encounter by reading and talking to people.
- While combinations may indeed by original, their originality and importance as well as the extent to which they are doable must be very carefully assessed.
- Open questions in existing work
Week 2
Thursday, January 12 (Lecture #2)
- Starting Research: Selecting and Validating Topics (Cont'd)
- Example: $M_l$ consensus tree algorithms (my BSch dissertation) [OBNG / supervisor-selected; open questions]
- Example: The computational complexity of evolutionary tree inference (my MSc project) [OBNG / self-selected; combination]
- Example: The parameterized complexity of phonological processing (my PhD project) [OBNG / self-selected; combination + open questions]
- Example: The parameterized complexity of of analogy derivation under Structure Mapping Theory (van Rooij et al (2008)) [OBNG / collaborator-suggested; combination + open questions]
- Example: The parameterized complexity of environment design for distributed construction (Wareham and Vardy (2018)) [OBNG / collaborator-suggested (sort of); misunderstanding + combination]
- Example: The structural approximability of cognitive processing (van Rooij and Wareham (2012)) [NBOG / collaborator-suggested; open questions]
- Though a lot of work, the process of selecting and validating a research topic can become a seductive activity in itself. One must be careful to always be mindful and assess why you are doing what you are doing. Remember: your goal is to get a topic that is original, important, and doable, and if one possible topic is not working out, move on to another in a timely manner.
- Example: Things I Didn't Do for My MSc and PhD II: chaos-theoretic aspects of evolutionary inference.
- A final caveat is in order: A good research topic takes effort to develop and is valuable in itself -- beware of spreading it around before you've had a chance to work on it fully, as others may not respect this effort.
- Example: The CWCB 1996 Incident.
- Example: The 1999 PhD Defense Incident.
- Example: The RECOMB 2001 Incident.
- Example: The 2014 Cog Sci Appendix Query.
- Starting Research: Timelines and Proposals
- Graduate Degree Research Timeline
- Above is the ideal; may take longer, but should not take too much longer (cf. my MSc thesis [4 years]).
- Does not include PhD comprehensive exams.
- The timeline for funded research projects is similar in structure to that above though the specific durations involved vary with the project length, e.g., OPC (prep + 2 months evaluation + 1 year project), NSERC Discovery grant (prep + 5 months evaluation + 5 year project).
- Non-funded research projects typically do not have timelines or
proposals; however, it may still be worthwhile to do one or both
of these for your own use.
- A non-funded research project proposal can take several forms, e.g., working notes, annotated bibliography, scheduled seminar (like the departmental Graduate Research Forum!).
- Such proposals serve also in spurring research along, as (unless you are eventually do something for a conference, journal special issue, or edited book) there are typically no deadlines.
- Graduate Degree Research Timeline
Week 3
Thursday, January 19 (Lecture #3)
- Starting Research: Timelines and Proposals (Cont'd)
- The graduate thesis research proposal
- Lays out what you want do, how you're going to do it, and
how long you think it will take you to do it.
- A graduate thesis proposal is not so much a contract for specific work as a demonstration that you know how to conduct research and have put in the requisite preparatory work, e.g., literature search.
- As such, a graduate thesis research proposal is a promissory note that you're ready to start research -- whether it is eventually the research you said you'd do (and in the manner and time you said you'd do it in) is another matter.
- A thesis proposal is at its heart a sales pitch whose purpose is to convince others to buy the idea that you should be allowed to do research -- hence, whatever makes the proposal more convincing, e.g., easy-to-understand stories, backup of claims by literature citations, believable timelines, is to the good.
- General structure of proposal (single-spaced pages):
- Title
- Titles should be descriptive and relatively short, e.g., my BSc, MSc, and PhD thesis titles.
- Save cutesy titles for later publications, e.g., my original BSch thesis title.
- Abstract (between 150 and 250 words)
- Describes the proposal in brief.
- The above is meant to be taken literally: Consider a
an abstract as consisting of motivation and
previous work of 1 sentence each, followed by a
sentence encapsulating motivation and research question,
e.g.,
-
X is very important because of Y.
Though there is much previous work on Z, no one has yet investigated X.
In our research, we will examine X using W.A longer abstract simply adds extra clauses and sentences to the organization sketched above. A very good example of how to do this is contained in the online description of a Nature Summary Paragraph.
- Motivation [~2 pages]
- Describes why this project is worth doing, i.e., what will be done in this project that has not been done before (originality) and why this is important.
- As part of this, should describe previous and related work as context for proposed work in order to highlight the hole in previous work that the proposed research will fill. Do this in sketchier form here, with more details in the next section of the proposal.
- A story provides a natural way of organizing this section (see the short description of an abstract above).
- It is particularly important to back up statements with literature citations in this section. Such citations not only establish the truth of your statements (Don't Blame Me -- Blame Them) but also show that you've put in the requisite preparatory work (Trust Me -- I Know What I'm Talking About).
- Previous and Related Work [~2-4 pages]
- Describes in detail what has been done before that is either directly or indirectly related to the proposed project, cf. Motivation.
- It is tempting to just list and briefly describe references,
not unlike an annotated bibliography. However, all
possible effort should be made to impose more
organization and structure, such as a taxonomy of
existing work (possibly organized chronologically to
show how research has proceeded over time).
- Such an imposed organization makes for an easier read of the proposal, and should also make it much easier for you to structure both your motivation and research question sections.
- A well-done previous work section may also be a (possibly publishable) research contribution in its own right, e.g., my MSc thesis.
- Ideally this section should be structured to highlight that which has not been done previously which you plan to do -- this gives a natural bridge to the following sections in the proposal. Where possible, consider making this structure a story or narrative, e.g., "Once upon a time there was a very important problem. Many people have worked on this problem so far, but they haven't done important thing X. However, I have a neat plan to do X -- would you like to hear about it? If so, please keep reading ..."
- Research Question [~1 page]
- Describes what specific question or set of questions you will be addressing in your thesis work.
- A research question is intermediate in rigor between a formal scientific hypothesis and an everyday question.
- The precise form of the research questions depends on the research topic; are much easier to define if the research was inspired by open questions in the literature.
- Example: The two general research questions
underlying all of my parameterized-complexity-based
research relative to an problem X:
- Can X be solved optimally in polynomial time in general?
- If not, what restrictions do (and do not) make X solvable in effectively polynomial time?
- Every question should be well-motivated by the preceding proposal sections and be answerable relative to the subsequently-described methods to be employed in the research; as such, these questions are the central pivot on which all else in the proposal depends and operates.
- Methodology [~2-4 pages]
- Describes how you will answer the research questions.
- Methods are typically described in general terms, with appropriate literature citations for details; that being said, these descriptions must be specific enough to allow explanations of how these methods can be used to answer the posed research questions, i.e., no ``magic occurs here'' processes.
- Note that this section and the following one are the ones where you convince others that the proposed research is doable by you.
- Timeline [~1 page]
- Describes your anticipated course of progress
in the proposed research.
- The emphasis here is on ``anticipated'' -- this is research, not engineering, and experienced researchers know things seldom go according to plan.
- Can do sophisticated multi-tier graphics here; however, the simplest form (an itemized list consisting of month-specific dates with attached research activities and/or deliverables) is often sufficient.
- The timeline must be believable -- to ensure this, be as specific as possible when describing research activities, e.g., no ``magic occurs here'' processes, and budget for some difficulties when establishing dates, e.g., The Scotty Principle.
- Describes your anticipated course of progress
in the proposed research.
- Bibliography (15-30 references) [~2-3 pages]
- Bibliography should be alphabetic by author name, and all references should be both complete and correct in details and consistent in format, e.g., do not describe conference papers in two or more ways.
- Compiling a good bibliography is a lot of work, but it plays an important role in convincing others that you can be trusted to do research; it can also be a research contribution in itself, e.g., my MSc thesis.
- Title
- Example: My PhD research proposal [3rd draft] (PDF)
- Remember that you need not write up the sections in the order in which
they appear in the proposal -- in order to establish originality and
importance, Related Work, Research Questions, and Bibliography should
be done first, followed by Methodology (to establish do-ability). Once you
then know where you'll
end up story-wise, one can then do your Motivation and Timeline.
- Allow time to revise your proposal after the first draft of all sections are finally written -- at that point, you MUST adjust story and terminology to be consistent throughout the proposal.
- Lays out what you want do, how you're going to do it, and
how long you think it will take you to do it.
- The graduate thesis research proposal
Week 4
Thursday, January 26 (Lecture #4)
- Starting Research: Timelines and Proposals (Cont'd)
- The general structure above is also characteristic of funded research
proposals modulo differences in the length of particular sections
and the presence of additional sections, e.g., personal
record in previous funded research and the training of Highly
Qualified Personnel (HQP), budgets, budget justifications.
- Note that in the case of proposals involving funding, the extent to which eventual research must follow that described in the proposal varies with the funding agency and grant-type.
- Example: OPC / NSERC equipment / Dutch PhD project proposals vs. NSERC Discovery Grant proposals.
- Example: My 2010 NSERC Discovery Grant proposal (PDF)
- Wrt non-funded research projects, it may well be worth doing up specific sections described above (in particular, previous work, research questions, and bibliography) to help both focus the research from the beginning and aid in creating subsequent publications describing the results of this research (which will require these materials anyway).
- In all cases, be honest in describing all aspects of your proposal,
particularly the extent to which the proposed research is original
and important and your ability to do it.
- Part of this can be including frank descriptions of potential difficulties and how you will deal with them.
- Given the temptation and incentives to over-sell research by being less than honest, honesty can set one apart in a good way and be reputation-enhancing.
- Example: Reviewer comment on my 2010 NSERC Discovery Grant proposal.
- The general structure above is also characteristic of funded research
proposals modulo differences in the length of particular sections
and the presence of additional sections, e.g., personal
record in previous funded research and the training of Highly
Qualified Personnel (HQP), budgets, budget justifications.
- Doing (and Surviving) Research
- There is nothing like either the joy of research going well or the despair of research going badly. Though one can never be quite sure what research will bring result-wise, one can with care and forethought simultaneously maximize the former and minimize the latter.
- There are many techniques for carrying out computer science research -- these techniques are what one picks up in courses and personal reading and from talking to your supervisor and other academics.
- We will focus here not so much on the research process but on how to survive and enjoy this process.
- General Research Etiquette
- There are some basic common-sense rules that should always be
followed when doing research:
- Be honest in performing research and reporting results.
- Acknowledge the contributions of others to your research, e.g., do not (excessively self-)plagiarize.
- Where possible and prudent, help others in their research.
- Be courteous and respectful to research support staff
We are human, and lapses will occur (particularly wrt (3) and (4)); however, be on guard for lapses in (1) and (2) ((1) in particular), as much of research depends on the trustworthiness of researchers and such lapses can destroy research careers.
- Though life in general may not be fair, aim to create a peaceable kingdom wrt your research and the people you deal with in doing that research.
- There are some basic common-sense rules that should always be
followed when doing research:
- Graduate Thesis Research
- The Thesis Lifestyle
- The Pace of Thesis Research
- Work hard and work consistently, i.e., try wherever possible to avoid last-ditch 24/7 work binges.
- Quite aside from keeping your supervisor happy (by allowing regular demonstrations of research progress), this is an excellent strategy for ensuring that you always have additional time to cope with unanticipated difficulties
- The Necessity of Not Doing Thesis Research
- Make sure (if you have to, by scheduling it)
that you have regular breaks away from your
thesis doing something that you find relaxing and
enjoyable.
- This not only includes daily and weekend breaks but vacations, e.g., visits home.
- This helps give you both the energy to do good research and necessary perspective on what you are doing research-wise.
- Oddly enough, the above applies even when research is going very well -- the endorphin rush of good research can be seductive and potentially damaging, e.g., death by joy.
- Example: My 2015 Summer (Non-)Vacation
- Make sure (if you have to, by scheduling it)
that you have regular breaks away from your
thesis doing something that you find relaxing and
enjoyable.
- The Necessity of Setting Personal Priorities wrt Work
- There are plenty of people out there who are more than happy to tell you how much you should work (and conversely how little you should be away from work).
- Time spent working has both benefits, e.g., faster thesis finishing time, more publications, and drawbacks, e.g., broken relationships, personal depression / burnout (see below).
- Example: Cases in point: Dan Brooks, Tao Jiang, and Bill Threlfall.
- You need to decide what your priorities are and choose the tradeoffs between work and life that are right for you before others do it for you.
- During graduate studies, this balance is particularly hard to strike given the unequal distribution of power between students and supervisors; this is yet another reason why it so important to maintain cordial and open relations with one's supervisor (see below).
- All of the above can benefit by making sure that you are good at allocating and managing time both to research and life in general -- anything that helps in this is to the good, e.g., training in Time Management / Meditation / Mindfulness.
- The Pace of Thesis Research
- The Thesis Lifestyle
Monday, January 30 -- Friday, February 10
- MUNFA on strike; no lectures
Week 5
Thursday, February 16 (Lecture #5)
- Doing (and Surviving) Research (Cont'd)
- Graduate Thesis Research (Cont'd)
- The Care and Feeding of Graduate Research Supervisors
- Ideally, you should have a good intellectual and
personal fit with your supervisor, i.e.,
they should know the things you need to do research
and you should be able to talk to them.
- The former helps your research go forward; the latter helps when your research has problems.
- Ideally, you should have regular (at least weekly or
bi-weekly) meetings at which you discuss research (and later
thesis) progress.
- Regularly-scheduled meetings are a great way to keep up the pace of research.
- Try not to dread these meetings -- if there has been no progress, use the meeting to figure out why this occurred and what can be done about it.
- As in any relationship, if problems arise between you and your supervisor, talk them out at the earliest possible opportunity when at all possible.
- Student-supervisor relations are seldom perfect, and
the causes of deviations from the ideal as well as the
degree to which these deviations are problematic
vary wildly (Tolstoy's Dictum)
- If your supervisor is bureaucratically challenged wrt regulations and deadlines, make sure that you know what the relevant regulations and timelines are and make sure that you follow them.
- If your supervisor is not making time to meet with you or to give you timely advice and feedback, do not let them off the hook -- their time is your right as their student, and patiently but firmly request that they make time for you.
- If your supervisor is not a good intellectual or personal fit, get the help you need elsewhere, e.g., supervisory committee members (PhDs), other faculty members (departmental and otherwise), fellow students of the same supervisor, fellow students, friends and family.
- If there are irreconcilable differences between you and
your advisor or the relationship between you becomes
abusive, talk to your graduate officer immediately
for advice.
- Though there may ultimately be little that can be done to compel a tenured supervisor to behave, not letting others know what is going on will only allow things to get worse; moreover, the assistance of these others will be necessary if you need to switch supervisors.
- Switching supervisors is a common fantasy in times of trouble, and it can be done and survived -- however, it is even in the best situations disruptive and should only be invoked for real if there is no other solution.
- Ideally, you should have a good intellectual and
personal fit with your supervisor, i.e.,
they should know the things you need to do research
and you should be able to talk to them.
- Writing Up the Thesis
- Theses take different lengths of time to write -- perhaps the only common feature is that all theses take longer to write than you think they will. Hence, start thesis writeup as early as possible and budget time for problems that can arise.
- Decide on your typesetting system up front.
- The main choices in Computer Science are MS Word and LaTeX, with research areas varying in their preference.
- Make sure that the conventions you adopt, i.e., for tables and figures, citations, and bibliographic references, are consistent with those mandated by your university.
- Example MUN Thesis Guidelines (webpage)
- Thesis templates (when available) can be invaluable.
- Example Unofficial MUN Thesis Templates (LaTeX)
- The standard thesis format looks strangely like
that for the thesis proposal (introduction,
background (including previous work), methodology,
results, discussion, conclusions, bibliography).
- This standard format van vary according to research area, e.g., software engineering vs. evolutionary algorithms vs. computational complexity analysis vs. theoretical computer science.
- Your supervisor is typically your best guide to which thesis format you should use.
- Theses need not be written up in either the final order of
sections or at the final level of detail.
- Consider first doing brief background and methodology sections and only fill these in with details after you have written up your results and discussion (at which point you will know the details you need to put into the background and methodology). Conclusions and the introduction can be done last.
- As terminology and story are especially prone to vary with time over long-duration writeups like theses, allocate time at the end to make both consistent throughout.
- Never make the bibliography the last thing you
do -- always add all references in full as you cite
them in the thesis drafts.
- Any time you spend adding full references that are later deleted is typically vastly less than the you spend trying to recover references that you forgot.
- The above is also a good reason why you should never wait until the text is finalized to add literature citations.
- Make sure you acknowledge text, tables, and/or figures
in your thesis that are derived or taken from other sources.
- Small pieces of text, e.g., 1-2 sentences,
can be placed in thesis text by enclosing them
in quotation marks and giving a literature
citation with page numbers at the end. Such page
numbers can be done in LaTeX using the optional
square-bracket argument for the \cite command, e.g.,
-
This approach is not commonly used, in large
part because of the opinion of practitioners
such as Wareham who claim ``... that the whole
thing is nonsense'' \cite[page 23]{War01}.
Larger pieces of text should be offset from the thesis text, again with a literature citation with page numbers at the end. Such offsetting can be done in LaTeX using the quotation or quote text environments.
- If tables or figures are adapted from other sources,
e.g., the table or figure is simplified
or enhanced, it suffices to acknowledge this in
the caption, e.g.,
-
\caption{A simplified model of a neuron (adapted
from \cite[Figure 4]{SNW98}).}
If the table or figure in your thesis is exactly the same as it appears in the source, e.g., you've re-used the JPEG file from the source, remove "adapted from" in the caption acknowledgment above and get written permission from the author and/or publisher to re-use the table or figure. Failure to do this can result in the rejection of your thesis on copyright grounds.
- Given the severity of the repercussions for not handling such matters correctly, if you are at all in doubt about how re-use should be acknowledged wrt thesis text and obtained permissions, consult your supervisor and the MUN copyright office (webpage).
- Small pieces of text, e.g., 1-2 sentences,
can be placed in thesis text by enclosing them
in quotation marks and giving a literature
citation with page numbers at the end. Such page
numbers can be done in LaTeX using the optional
square-bracket argument for the \cite command, e.g.,
- Solicit feedback from your supervisor and/or supervisory
committee on individual chapters (and, when they are
critical, individual sections) as they are written,
i.e., avoid Big Bang Writeup.
- This ensures that you do not drift too far from standard formats and/or supervisor preferences, and can save a lot of work.
- This also allows early detection and resolution of format disagreements between co-supervisors as well as simplifying final thesis checking before submission for examination (see below).
- Theses at MUN have been rejected because of poor written
English. Hence, if you need help with your written English,
get it.
- Some supervisors provide this; in other cases, use the Writing Center (webpage) or hire an editorial consultant.
- The Writing Center is free but may take a while; editorial consultants are not free but the benefit of feedback that is speedy (and possibly better-suited to you and your thesis) may make it worthwhile.
- Some people do this throughout the writeup process while others only do it for the final draft prior to thesis submission -- I prefer the former, if only because Englishification of a large document all at once can take a lot of time.
- Be aware of your intellectual property rights as regards
your thesis.
- Example: MUN thesis intellectual property rights (Webpage)
- Depending on circumstances, you may not want to put everything you know or derive in your thesis.
- Example: My 1990 MUN M.Eng. Experience.
- The Care and Feeding of Graduate Research Supervisors
- Graduate Thesis Research (Cont'd)