This page is an orientation for junior researchers in NUOMI Lab who are developing their research skills and careers. It covers what research is (and isn’t), the four fundamental skills of research and what we expect for each, research best practices for meetings and code, how to engage with technical ideas and write a machine learning research paper, and a map of the skills researchers keep developing throughout their careers.

No one arrives with all of these skills. Treat this page as a reference to come back to as you grow, and as a starting point for conversations with your mentors. Further reading is on the Resources page.

On this page:

1. What Research Is (and Isn’t)
2. Four Fundamental Skills of Research
3. Research Best Practices
4. Working with Technical Ideas
5. Skills for a Research Career

What Research Is (and Isn’t)

Most of us are exposed to research through depictions in popular culture and media, which often default to tropes (e.g. the mad scientist, the wunderkind, the lone tortured genius) that encode misconceptions about the real practices of scientific research. In reality, research is much more mundane, and much more accessible and rewarding, than common misconceptions suggest.

1. “Research is done by a lone genius, usually by deriving lots of math.” In reality, research is almost always done in a team of collaborators who bring different skills, interests, and perspectives (and personalities). Most questions can be tackled fruitfully in many different ways, not just by doing lots of fancy math.

2. “Research is done by making monumental breakthroughs from epiphanies.” In reality, research is done by iterating: asking very concrete, narrowly scoped questions each week, designing experiments to answer these questions, then generating new questions.

3. “The value of research is in the eyes of reviewers at prestigious conferences and journals.” In reality, the value of research is what it means to you. You need to find an independent reason for doing research.

4. “Research skills are inborn; some people are just better at it.” In reality, no one is born with research skills, and everyone has to work hard to gain them.

5. “Research is always fun, and driven by burning intellectual fire and moral passion.” In reality, research can sometimes be exciting and fun, but it can also be very frustrating. You need to find a sustainable reason for doing research.

Why You Might Want to Do Research

You might want to do research for many different reasons; there is no single right reason to do research. For example:

• You are considering a PhD and want to know if you like research.
• You like working on open-ended problems.
• You like working with a team of friends on a challenge.
• You want to solve a real-world problem.
• You want to present your results at a machine learning workshop so you can meet other researchers in machine learning.
• You want to submit your results to a machine learning conference and connect with the wider machine learning community.

Four Fundamental Skills of Research

Research rests on four fundamental skills:

1. Defining and scoping a research problem
2. Surveying related work
3. Sustaining incremental progress on your research question
4. Communicating results

1. Defining and Scoping a Research Problem

Turn a vague or general real-life question into actionable technical questions that are valuable to answer. This involves learning:

• what makes a question useful to answer, i.e. of interest to some community;
• how to narrow down a vague or general question into concrete, actionable questions;
• how to translate goals in the application or task setting into machine learning formalism.

Questions to ask yourself:

• What application domains or research subfields does your research question impact? How does your research question advance these domains or subfields?
• What aspects of your research question can be formulated as a machine learning problem, and what framework of formulation should you choose?
• What aspects of your research question are solvable given your background and timeline?

What we expect: It will take many years of experience for you to be able to do the above independently. In the meantime, observe with care how your mentors define and scope research problems, and learn from their examples.

2. Surveying Related Work

Understand how your proposed approach to a problem relates to existing work. This involves learning to:

• collect representative works that use different methods to tackle the same problem, works that use similar methods for different problems, and works that demonstrate the significance of the problem;
• synthesise related works into conceptual categories, exposing different frameworks or schools of thought;
• situate your approach to the problem within the literature: what do you intend to contribute, and why is this contribution novel and significant?

How do you build up a representative collection of related works?

• Read other people’s literature surveys, and explore the works referenced there.
• Start with a set of “seed works”, and explore their references as well as the works that cite them.
• Explore works in adjacent fields that solve similar problems or use similar methods.

How do you summarise related works?

• Group related works by conceptual framework, modelling assumptions, inference techniques, types of data, space-time complexity, limitations, etc.
• Identify how your work is different from each group of related works, and why that difference means your approach is better.

What we expect: You will maintain a succinct summary of related works, and be able to clearly articulate your contribution in relation to these works, as well as build on existing knowledge.

3. Sustaining Progress

Maintain piecewise-small, concrete, consistent progress towards your research goal. This involves learning to:

• design experiments to test specific hypotheses;
• analyse and synthesise experimental results to inform new hypotheses;
• align short-term hypotheses with long-term goals.

In practice:

• Keep detailed research notes, and keep these notes updated and organised.
• For exploratory experiments, keep track of the parameters that vary versus those that are fixed.
• For targeted experiments:
  • State the hypothesis the experiment tests (there should be only one per experiment).
  • State how you will summarise the results (e.g. a table or a graph).
  • State how the result would give support for or against the hypothesis: describe how the graph of a positive result would look versus a negative result.
  • Consider alternative explanations for your results (e.g. are there confounding variables?).
• Generate new hypotheses. Every experiment, regardless of the result, yields insight.
• Periodically summarise your results, and ask what remains to be shown with respect to your research goal.

What we expect: Every team keeps a single, well-organised document, updated weekly, containing its research notes as well as feedback from meetings.

4. Communicating Results

A sequence of results is not a paper! Communicating research is the act of creating a single, scientifically honest narrative to fit your results. This involves learning to:

• create a single coherent narrative from your experimental results;
• curate experimental results that support the claims in your narrative;
• evaluate to what degree you’ve solved the problem, and identify limitations and future steps.

Questions to ask yourself:

• What does the body of your experiments say about your research question as a whole?
• Are your experiments sufficient to support your claims?
• Do the results from different experiments support the same conclusion or different conclusions?
• What are the limitations of your experiments (what can you not conclude from the results)?
• What are some interesting directions for future research suggested by your experiments?

What we expect: Every team produces a conference-style paper summarising its work. The process of writing is what turns facts into knowledge. See How to Write a Machine Learning Research Paper below.

Research Best Practices

Research Meetings

How to make the most of your research meetings:

• Prepare: plan out what you are going to present and how you will present it (e.g. make slides).
• Review: succinctly summarise the goals for the week.
• Summarise: succinctly summarise, at a high level, what you did to meet these goals (e.g. which experiments you did, and why you did them).
• Conclude: succinctly summarise, at a high level, the results of the experiments and which hypotheses they support or invalidate.
• Ask: state what you’d like feedback or help on, invite others to ask you questions, and have your results on hand for detailed discussion.

Your Research Week

How to make the most of your research week:

• Have a clear sense of next steps and goals before leaving each meeting, and check that your understanding of these goals matches your mentors’.
• Practise checking in and asking for feedback: have a mentor check your experimental design before executing it, and practise presenting your results to a peer or mentor to get suggestions for improvement.

Code

How to create usable scientific code bases:

• Use Jupyter notebooks for quick sanity checks or exploratory experimentation. Then write up all usable code as Python scripts, and recreate the experiments done in notebooks.
• Use good programming practices:
  • Keep your code modular.
  • Unit test each piece.
  • Test against SciPy, scikit-learn, etc. implementations whenever comparisons are available.
  • Document your code, and use pedagogical naming conventions.
• Use version control:
  • Start a private GitHub repository.
  • Keep the main branch clean and up to date, and experiment on separate branches.
  • Develop a good workflow and conventions for merging.

Working with Technical Ideas

Engaging Meaningfully with Technical Ideas

Literature

• Identify intellectual and research trends.
• Identify differences and similarities.
• Identify trade-offs between different approaches.
• Identify gaps, open questions, and opportunities.

Logic and argumentation

• Develop fluency in the rules of logical deduction (e.g. first-order logic).
• Extract arguments from writing.
• Evaluate empirical or analytic evidentiary support by applying the rules of logical deduction.
• Write clearly and precisely.

Symbolic formalism

• Losslessly translate English statements into mathematical formalism, and vice versa.
• Identify ambiguities in translation (when the translated result is not unique).
• Map formalism to high-level intuition (a level higher than saying in English what the symbols literally mean).

Impact

• Identify methodological design choices.
• Unfold the potential impact of methodological design choices.
• Identify appropriate metrics for downstream tasks.
• Identify design choices in metrics.
• Unfold the potential impact of metric choices.

How to Write a Machine Learning Research Paper

Start with the introduction. Write one sentence for each of the following points. Each comes with a check for whether you got it right.

1. The problem motivation. What is the real-life (or other) motivation for your problem? What is the setting for your problem?

   Check: Does the reader care about your problem (i.e. feel that your problem is important) after reading your motivation?

2. The problem statement. What exactly is the problem you are solving in this paper?

   Check: Does your method completely and thoroughly solve the problem?

3. The existing work and gaps. What are the existing approaches to solving your problem, and why are they insufficient? Where do they fall short?

   Check: Does your method outperform all existing approaches in the areas where they fall short?

4. The method. What is your solution to the problem?

   Check: Does your reader have a clear sense of how your method is put together (e.g. its major technical pieces, the major components in your pipeline, major innovative changes to existing methods)?

5. The evidence. What are the empirical and theoretical evaluations or validations of your method?

   Check: Do your experiments and theoretical analyses show that your method outperforms all existing approaches in the areas where they fall short? Do they show that your method works for the right reasons?

Turn your introduction into a paper.

1. Lay out a skeleton. Start with bullet points for an introduction, laying out a skeleton for your argument. Make sure your logic is sound. Check with other people that your argument is convincing.

2. Outline the body. Using the skeleton in your introduction, create an outline for the body of your paper by naming your sections and subsections. Section and subsection names should be informative: tell the reader what the section is about.

3. Synthesise as results come in. As your literature survey, theoretical analyses, and experimental results come in:
   • Synthesise information by outlining each subsection: write down the title and the key take-away (last sentence) of each paragraph. Paragraph titles should be informative, telling the reader what the paragraph is about. The key take-away should be informative too: if the reader skipped the entire paragraph and only read the last sentence, they should get a sense of the main insight highlighted in that paragraph.
   • Summarise results by adding tables and figures with informative captions. Each caption should state which methods, datasets, and metrics are visualised by which aspects of the table or figure, and should end with the main take-away of the table or figure.
4. Flesh it out. Once you have added most of the main content, start fleshing out your paper with full sentences, without losing the tight logical structure of the bullet-point version:
   • Put in structure to guide the reader and highlight the logic: at the beginning of each section or subsection, outline its key points or results and its logic; at the end, summarise its key points or results.
   • Make your paper skimmable: make sure each paragraph has a bolded title and a bolded take-away; bold important insights and conclusions; make sure important information gets its own paragraph rather than being buried in a long discussion.

   At this point, don’t worry too much about page limits or about being repetitive. Write for clarity, accuracy, and ease of understanding.

5. Polish. Once your paper is mostly fleshed out, begin polishing:
   • Observe the elements of good technical writing as you edit (refer to the book on writing we discuss in group meetings).
   • Always favour clarity of logic over literary style: err on the side of short, clear sentences, and spell out conclusions explicitly rather than implying them.
   • Read for ambiguity: is every definition of jargon or a concept, and every description of a method, concrete and detailed enough to rule out multiple interpretations? Test this adversarially: what else could satisfy your definition or description, other than the intended concept or process?
   • Add connectors and transitions that highlight the relationships between ideas and insights:
     • informative transitions and connectors between consecutive paragraphs and blocks of text (e.g. “because of the concern we raised in Section 1.2.1, we now do the following analysis…”);
     • useful references to terms, concepts, and insights in other parts of the paper (e.g. “surprisingly, this result contradicts what we predicted in Theorem 3 in Section 3.1…”);
     • signposts that anticipate and prepare the reader for what is to come (e.g. “this approximate inference method works well only if our choice of approximate posteriors is expressive; we study the relationship between the choice of approximate posteriors and performance in Section 5…”).
6. Shorten last. Once your paper has been polished for clarity, you may shorten it to improve concision or to meet page limits. However, you must do this without losing:
   • technical clarity in the definitions and descriptions of concepts and processes;
   • structure that guides the reader and highlights the logic;
   • skimmability.

Skills for a Research Career

This is a superset of generally helpful skills for machine learning PhD students and researchers. No one is expected to have all of them, and researchers keep developing these skills throughout their careers. Use it as a map to reflect on where you are and what you’d like to work on next.

Managing Research

• Time management: how to be efficient
• How to switch between different projects, or manage your time across different projects
• How to critically evaluate research proposals and questions
• How to read research papers (quickly)
• How to find relevant resources on Google (how to identify keywords, and know that certain keywords are related to your questions)
• How to design toy datasets to test hypotheses
• How to come up with hypotheses given results
• How to synthesise insight from a bunch of random results
• How to evaluate the novelty of your technical contribution to a body of literature
• How to pivot when a project seemingly reaches a dead end
• How to come up with small project ideas (extending existing work)
• How to come up with big project ideas (asking more big-picture or philosophical questions)
• How to formalise a vague question into a concrete, solvable problem statement
• How to break up a big research programme into concrete questions for students you mentor

Communicating Research

• How to communicate results during 10-, 20-, and 30-minute meetings
• How to ask for the feedback you need during research meetings
• How to communicate your research programme as an elevator pitch
• How to listen for insight during technical talks
• How to phrase questions for Q&A during and after technical talks
• How to give talks to, and write for, different types of audiences
• How to write and speak in a mathematically and technically precise fashion
• How to review papers
• How to solicit feedback

Managing Collaborations

• How to network
• How to set expectations for collaborations (authorship, workflow, roles, etc.)
• How to avoid or resolve potential conflicts during collaborations: e.g. making your collaborators feel appreciated and their contributions valued, and making sure that you feel your own role is well defined and your contribution significant and valued
• How to work effectively with people who are more senior, or more experienced or skilled in an area you’re working in
• How to mentor younger researchers and students

Being a Faculty Member

• How to identify what kind of institution you want to work at
• How to prepare a research statement
• How to prepare a teaching statement (if relevant)
• How to prepare a job talk
• What questions to ask during interviews to find out whether you want to work at that place
• How to answer interview questions
• Writing grants
• Balancing research and teaching
• Balancing research, teaching, and service

Managing Yourself

• How to manage stress from lots of work
• How to manage work-life balance
• How to manage stress from not having a strong identity as a researcher, or from having that identity challenged
• How to manage stress from uncertainty about your progress
• How to manage stress from second-guessing feedback or your progress
• How to realistically evaluate your growth and progress
• How to motivate yourself to work (or to work effectively)
• How to set reasonable goals for yourself