Research design and proposal writing [TU059/TU060]
[A high-order thinking course]
Timetable
Tuesdays 3 pm (sharp) to 5pm
Venue
- CQ-409: small tiered lecture theater (Central Quad, Grangegorman campus)
Slack
Submission deadlines:
- First assignment: 25th of February, 2024 (20%)
- Second assignment: 7th of April, 2024 (20%)
- Third assignment (final proposal): 15th of May, 2024 (60%)
All assignments must be submitted to
luca(dot)longo(@)tudublin.ie in pdf format only, by the established
deadline, 11.59pm (irish time)
No marks: assignments submitted after each deadline will be marked zero
Disclaimer:
assignments from previous students cannot be shared due to the GDPR and
other regulations about privacy. Additionally, each assignment should be based on the student's own reasoning, and not influenced by others' previous assignments.
The electronic notes are my guide for running lectures and are not intended to fully contain everything covered during classes.
It is the student’s responsibility to take notes during classes.
- Intro to the module, learning outcomes, indicative
syllabus and activities
(pdf) - Research, Science & Computers: Computer Science Research
(pdf) (video)
Essential notions
Precision, focus, rigor
TODO:
- Read and review the following notes on 'developing a research topic' (
pdf,video) - identify the area of research you want to work in, with 4 keywords
- identify potential data (for DS student) in the area of research of interest, or think how to write a piece of SW (for ASD) to deliver performance measures. Note that data can be of any shape/form (numerical, categorical, textual, images, signals etc.) and it is needed within the context of the scientific method.
- start contacting lecturers, as they might have research problems for you, as well as data already available
- here some
research idea proposed by lecturers
Hints for developing a research idea FOR DATA SCIENCE (DS) students
-
here a collection of datasetsthat you might want to use for creating data-driven models (eg. using machine/deep learning) -
A searching engine of datasets
Hints for developing a research idea FOR Advanced Software Development (ADS) students
- Think of an existing piece of SW that you can download and improve/extend, and then compare (A/B testing)
- Think of an existing framework that you can run multiple time, under different configurations, and then compare (against performance measures such as memory, time, resource consumption etc.)
- Think of designing/developing a new algorithm, and compare its performance with state of the art algorithms (eg. a genetic algorithm, a sorting algorithm, etc).
- Think of designing/developing a new system, and compare its performance with state of the art systems (eg. a recommendation system, a computational trust system, etc.)
- Think of developing an interactive interface, under different configurations, and compare their usability/user experience/engagement/cognitive load of users
MOTTO If you can not describe what you are
doing as a process, you do not know what you are doing. (W.
Edwards Deming)
TODO:
- Read the following notes on 'Literature review' (
pdf,video) - identify 2 peer-reviewed research articles in the area of research you have selected
- identify preliminary gaps from the research articles you have read and identify a research problem
- write 3 preliminary research questions on the area of research you have picked (they will be discussed in the next class)
- Developing a research hypothesis (pdf)
- Q/A on research hypothesis + in-class examples and discussion
The C5 complex You need be, clear, concise,
compelling, committed and consistent to achieve any sort of
success (Chris Gardner)
TODO:
- if you haven`t done yet, identify appropriate methods/techniques/algorithms/approaches for your research project
- write a research hypothesis on the research problem you have identified
- identify 1 other peer-reviewed research articles (so far 3) in the area of research you have selected
- lecture notes on assumptions, scope, limitations and delimitations of research design
- extra notes on assumptions, scope, limitations and delimitations of research design
TODO:
- define the assumptions, limitations, delimitations and scope your research study
- identify 2 other peer-reviewed articles articles (so far 4) in the area of research you have selected
- Questions on first assignment + Q/A
- In-class activity on research hypothesis
TODO:
- carefully read the instructions of the first assignment
- identify 1 other peer-reviewed research articles (so far 5) in the research you have selected
TODO:
- select the research methods relevant to your research design (by type, form, objective, reasoning)
- keep amending your research question and research hypothesis
- identify 2 others peer-reviewed research articles (so far 7) in the area of research you have selected
- peer-review marking of first assignments
- research aims and objectives (pdf)
Research is replicable when a researcher, or an independent group of researchers,
can copy the same process and arrive at the same results as the original study supporting and testing its validity.
TODO:
- define the general and specific research objectives for your research experiment, to eventually provide an answer to test your research hypothesis
- keep refining your research question and research hypothesis
- identify 2 others peer-reviewed research articles (so far 9) in the area of research you have selected
Videos
Correlation analysis
Distributions comparison
- Independent T-test
- independent Mann-whitney-test
- Dependent T-test
- Dependent Wilcoxon-test
- ANOVA (Analysis of Variance)
- Kruskal-walling-ANOVA (Analysis of Variance)
TODO:
- identify which statistics are appropriate for the performance measure you have selected for your research experiments; these will need to provide an answer to test your research hypothesis
- refine your research hypothesis by incorporating the concept of significance
- identify 1 other peer-reviewed research articles (so far 10) in the area of research you have selected
- Intro to the second assignment + Q/A
- In-class activity on refining your research hypothesis incorporating the concept of significance
TODO:
- keep refining your research hypothesis according to the concept of significance
- work on clearly refine your general and specific research objectives to allow replicability
- identify 1 other peer-reviewed research articles (so far 11) in the area of research you have selected
- Dissemination of research (pdf)
TODO:
- keep refining your research question + research hypothesis + general/specific research objectives
- identify 1 other peer-reviewed research articles (so far 12) in the area of research you have selected
- Intro to the third assignment + Q/A
- In-class activity on refining the research hypothesis and defining research objectives
TODO:
- keep aligning your research question to your research hypothesis and the research objectives designed to test it
- Wrapping up the semester
- Q/A + specific feedback to individual questions
- Prepare a set of slides -
Exactly 8 slides (studentnumber_first_assignment.pdf)(marks will be given according to how focused, precise, rigouros and relevant the presentation is, as well as the replicability of what is proposed)
Note: use all the space in each slide in a way that the content is readable when printed (use white background).- slide 1
Title: 'Research question, hypothesis and preliminary design (assignment 1)'
content: name, surname, programme number, stream, student number (change accordingly)
slide 2
- a focused sentence (only one) for best describing the SCOPE of your research
- a focused sentence (only one) for best describing the ASSUMPTIONS of your research (if you have multiple assumptions, delimitate them with a semicolon ';')
- a focused sentence (only one) for best describing the LIMITATIONS of your research (if you have multiple limitations, delimitate them with a semicolon ';')
- a focused sentence (only one) for best describing the DELIMITATIONS of your research (if you have multiple delimitations, delimitate them with a semicolon ';')
- DOMAIN:
- path 1 + inText citations in APA7 style
- path 2 + inText citations in APA7 style
- path 3 + inText citations in APA7 style
- path 4 + inText citations in APA7 style
- ASSUMPTIONS:
- SCOPE:
- LIMITATIONS:
- DELIMITATIONS:
- slide 3
Title: "Gaps in the literature and research question"
content: Write about:- precise and concise gaps extracted from literature you have identified by reading 5+ articles. You can write multiple sentences/paragraphs, organise them as you wish (by theme, gap, discipline, etc.) and justify your claims by citing (inText citation) all the relevant articles you have found.
- precise research question (one 'question' ending with a question mark)
Marks (total 6%):
0-> gaps are unknown or very vague; it is impossible to anticipate a research question from them;
1> gaps are somehow vague AND not all the identified relevant articles have been cited or contextualised; it is difficult to anticipate the research question from the gaps;
2> gaps are not vague but can be more precise OR not all the relevant articles have been cited or contextualised; a research question can be imagined, but with some degree of uncertainty;
3-> gaps are not vague and precise, backed up by ALL the relevant articles identified from the literature review, and this can lead the reader to imagine a possible coherent research question from them;
+
0-> research question is unknown or very vague; it is impossible to imagine how to design a research experiment to answer it;
1-> research question is valid BUT it is vague; it is very difficult to imagine how to design a research experiment from it;
2-> research question is not vague but can be more precise; it is in somehow sufficient to imagine how to design some component of a research experiment to answer it, BUT not all;
3-> research question is very precise AND complete AND valid; it is straightforward to imagine how to design a research experiment from it;
- slide 4
Title: "hypothesis "
content: Define the null and alternate hypotheses for tackling the research question (max 1 sentence each). Note that the null hypothesis does not need to be in the form of 'IF...THEN...', but as a statement reflecting the commonly accepted fact. - slide 5
Title: "Feasibility of the study"
content: Briefly describe the activities/tasks you have planned towards testing the research hypothesis and a rough time required for implementing each of them (BE REALISTIC) - slide 6 + 7
Title: "Bibliography"
content: list the articles you have read so far (this should be always an ongoing activity). Overall you must have 5+ bibliographic peer-reviewed entries. items)STRICTLY using theAPA7 style - Slide 8
Title: "Performance metrics of your experiment"
Content: Provide details about the performance metrics you are going to use in your experiment (names, ranges, type, etc.). Overall, provide as many details as possible to inform the reader on the performance metrics you will be using in your experiment and its evaluation. Examples - for Data Science students (DS), these metrics include (among many more) precision, recall, f1-score, ROC curves, AUC curves, etc. For Advanced Sofware Development (ASD) students, these metrics include (among others) time, resource consumption, memory, Signal-to-noise ratio, usability metrics, user experience metrics, etc.
Title: "Domain, scope, assumptions, limitations and delimitations of research - ACM 2012"
Content: A bullet point list of 5 items (and only 5) are required (DOMAIN, ASSUMPTIONS, SCOPE, LIMITATIONS and DELIMITATIONS). For the DOMAIN of your research usi the categories and concepts from theACM 2012 classification systemand produce 4 paths, from root to leaf (example of one path:A: Human-centered computing → Human computer interaction (HCI) → HCI design and evaluation methods → Usability testing (citation 1, citation 2, etc..)). At the end of each path, cite the relevant articles (at least 1 from the 5+ you have read so far). Note that you can cite the same article also in different branches. Please use theAPA7 style(inText). The other 4 elements of the bullet poing list are:
Marks (total 3%):
0-> domain AND scope AND limitations, AND assumptions, AND delimitation are vague;
0.5->four of the above are vague and ONLY one is well-defined and clear;
1->three of the above are vague AND ONLY two are well-defined and clear;
2->two of the above are vague AND ONLY three are well-defined and clear;
2.5->only 1 is vague and four are well-defined and clear;
3-> domain, scope, assumptions, limitations and delimitations are all well-defined and clear;Marks (total 5%):
0-> it is not a valid hypothesis;it is impossible to imagine which research tasks need to be implemented to test it;
1-> it is valid but VERY vague; it is VERY difficult to imagine which research tasks need to be implemented to test it;
2-> it is valid but vague; it is difficult to imagine which research tasks need to be implemented to test it;
3-> it is valid AND not vague; it is ONLY sufficient to anticipate SOME of the research tasks that need to be implemented to test it;
4-> it is valid AND not vague; it is sufficient to anticipate ALMOST all the research tasks that need to be implemented to test it;
5-> it is valid AND not vague AND precise. ALL the research tasks that need to be implemented to test it can be clearly anticipated;
Marks (total 3%):
0-> undefined plan to test hypothesis;
1-> vague plan to test hypothesis with minimal description of the research activities, leaving doubts on how to implement them;
2-> fair plan to test hypothesis with research activities sufficiently described, but NOT fully complete, leaving the reader with some doubts on how to implement some of them;;
3-> good plan to test hypothesis with complete description for EACH research activity, allowing the reader to FULLY implement them, without doubts;Mark (total 2%):
0-> the APA7 style has not been used;
1-> the APA7 style has been partially used; some errors OR incompleteness exist in in-text citations OR in the bibliographic section;
2-> correct AND complete use of the APA7 style across in-text citations AND in the bibliographic section;
Mark (total 1%):
0-> no performance metrics have been identified;
0.5->some performance metrics have been identified, BUT within minimal description, leaving readers with doubts;
1-> performance metrics have been clear identified AND clearly described AND all the details of each relevant metric have been provided; the description FULLY help readers to understand how the experiments, aimed at testing the research hypothesis, can be evaluated/validated with performance metrics;
Note - While building your slides, take into account the following questions- Is the scope of the research clear? Is the motivation convincing? Are the gaps well defined? Is the research question focused or still ambiguous, too narrow, or too broad?
- Is the hypothesis clear and convincing? How can it be better expressed?
- Do you believe the study is feasible given the time frame provided? (master or phd?)
- slide 1
- Prepare a set of slides -
Exactly 8 slides (studentnumbner_second_assignment.pdf)(marks will be given according to how focused, precise, rigourous and relevant the presentation is)
Note: use all the space in each slide in a way that the content is readable when printed (use white background).- slide 1
Title: "Research question, hypothesis and refined design (2nd assignment)"
content: name, surname, programme number, stream, student number
slide 2
- a focused sentence (only one) for best describing the SCOPE of your research
- a focused sentence (only one) for best describing the ASSUMPTIONS of your research (if you have multiple assumptions, delimitate them with a semicolon ';')
- a focused sentence (only one) for best describing the LIMITATIONS of your research (if you have multiple limitations, delimitate them with a semicolon ';')
- a focused sentence (only one) for best describing the DELIMITATIONS of your research (if you have multiple delimitations, delimitate them with a semicolon ';')
- slide 3
Title: "Gaps in the literature and research question"
content: Write about:- precise and concise gaps extracted from literature you have identified by reading 10+ articles. You can write multiple sentences/paragraphs, organise them as you wish (by theme, gap, discipline, etc.) and justify your claims by using all the relevant articles you have found.
- precise research question (one 'question' ending with a question mark)
Marks (total 4%):
0-> gaps are unknown or very vague. It is difficult to understand the gaps in the literature.
1-> gaps are not vague but can be more precise and informative;
2-> gaps are not vague and precise;
+
0-> research question is unknown or very vague
1-> research question is not vague but can be more precise and informative;
2-> research question is very precise, leading the reader to image a potential research experiment
- slide 4
Title: "hypothesis + research methods "
content: Define the null and alternate hypothesis for tackling the research question (max 1 sentence each) incorporating the concept of statistical significance. List the research methods you are going to use to test your hypothesis (by type, by objective, by form and by reasoning) and justify why. - slide 5
Title: "General and specific research objectives for experimental purposes towards hypothesis testing using statistical tools"
Content: Define a list of general and specific research objectives for describing what needs to be done in your experiment and for testing your research hypothesis as a bullet list of the form: O1, O2, ..., On.
General and specific research objectives can be multiple and the goal is to guide the reader to implement and execute your experiment and generate the empirical evidence to accept/reject your research hypothesis using statistical tests. You have to mention which statistical tests you will be using and why. Marks will be awarded based on the granularity of explanations and on the capability to inform the readers. - slide 6 + 7
Title: "Bibliography"
content: list the articles you have read so far (this should be always an ongoing activity). Overall you must have 10+ bibliographic peer-reviewed entries. items)STRICTLY using theAPA7 style - Slide 8
Title: "Performance metrics of your experiment"
Content: Provide details about the performance metrics you are going to use in your experiment (names, ranges, type, etc.). Overall, provide as many details as possible to inform the reader on the performance metrics you will be using in your experiment and its evaluation. Examples - for Data Science students (DS), these metrics include (among many more) precision, recall, f1-score, ROC curves, AUC curves, etc. For Advanced Sofware Development (ASD) students, these metrics include (among others) time, resource consumption, memory, Signal-to-noise ratio, usability metrics, user experience metrics, etc.
Title: "Domain and scope of research - ACM 2012"
Content: Describe the domain/s and the scope of the research using the categories and concepts from the ACM 2012 classification system (example of one line:A: Human-centered computing → Human computer interaction (HCI) → HCI design and evaluation methods → Usability testing). You must describe your research with 4 (and only 4) lines as a bullet point list (A, B, C, D). Beside each line cite the relevant articles that you have read (10+) according to theAPA7 style. Additionally, add below this list the following:
Marks (total 2.5%):
0-> domain AND scope AND limitations, AND assumptions, AND delimitation are vague;
0.5->four of the above are vague and ONLY one is well-defined and clear;
1->three of the above are vague AND ONLY two are well-defined and clear;
1.5->two of the above are vague AND ONLY three are well-defined and clear;
2.0->only 1 is vague AND four are well-defined and clear;
2.5-> domain, scope, assumptions, limitations and delimitations are all well-defined and clear;Marks (total 4.5%):
0-> It is not a valid hypothesis;
1-> It is valid (predictive, testable) but very vague;
2-> It is valid, not vague, but it is difficult to anticipate how to test it;
3.5-> It is valid, not vague, precise, correctly incorporating the concent of statistical significance, and clearly anticipating a method to test it. It leads the reader to imagine the research activities to be designed to accept/reject the alternate hypothesis.
+ 1-> convincing research methodsMarks (total 5%):
0-> research objectives AND statistical tests are unclear OR absent;
1-> NOT ALL the research objectives have been defined AND they are fragmented AND not convincing. It is EXTREMELY difficult to understand how the empirical evidence will be gathered to test the hypothesis AND/OR which statistical tests will be used to accept/reject it;
2-> MOST of the research objectives have partially been defined BUT they are fragmented AND not convincing. It is VERY difficult to understand how the empirical evidence will be gathered to test the hypothesis ANR/OR which statistical tests will be used to accept/reject it;
3-> SOME of the research objectives have partially been defined AND SOME fully designed BUT they are fragmented AND not fully convincing. Understanding how the empirical evidence will be gathered to test the hypothesis and/or which statistical tests will be used to accept/reject it requires SOME effort;
4-> ALL the research objectives have been defined, AND they are clear BUT not fully convincing. It is NOT difficult to understand how the empirical evidence will be gathered to test the hypothesis ANR which statistical tests will be used to accept/reject it;
5-> research objectives have been defined AND they are clear AND convincing. It is straighforward to understand how empirical evidence can be gathered to test hypothesis AND which statistical tests will be used AND why to accept/reject it.Mark (total 2%):
0->the APA7 style has not been used;
1-> the APA7 style has been partially used across the slides;
2-> correct use of APA7 style across the cited articles that are also fully complete;
Mark (total 1%):
0-> no performance metrics have been identified;
1->some performance metrics have been identified, BUT within minimal description, leaving readers with doubts;
2-> performance metrics have been clear identified AND clearly described AND all the details of each relevant metric have been provided; the description FULLY help readers to understand how the experiments, aimed at testing the research hypothesis, can be evaluated/validated with performance metrics;
Note - While building your slides, take into account the following questions- Is the scope of the research clear? Is the motivation convincing? Are the gaps well defined? Is the research question focused or still ambiguous, too narrow, or too broad?
- Is the hypothesis clear and convincing? How can it be better expressed?
- Do you believe the study is feasible given the time frame provided? (master or phd?)
- slide 1
Structure of the final proposal (2-column article) to be submitted as
studentnumber_research_design.pdf
to lecturer by established deadline
[you can use the Latex template
with the APA7 for latex , or word before converting to pdf.]
- Front-page
- Student name and surname
- Student number
- Stream (eg. ASD, DA) or PhD structured program
- DIT programme (eg. DT228A / DT228B, )
- Title of proposal
(max words 20 words) - List of people and companies associated with the
project (including those you might need input from or data
from)
(max 20 words) - Sources of data needed for the project, if any
(max 30 words)
- The research background
(max 300 words).
Describe the context of application and provide the reader with some background and main notions/concepts related to the domain and scope of your research. - Informal description of the research problem
(max 300 words)
Describe the problem you aim to tackle informally. Also define the assumptions, limitations and delimitations of your research proposal. - Literature review and its gaps + state-of-the art approaches
to solve the identified research problem
(max 1000 words)
Describe the relevant peer-reviewed articles you have read (15+) in the selected domain of research and identify the gaps. Additionally, identify and describe the state-of-the-art approaches to solve the identified research problem.
Gaps and state-of-the-art will inform the proposal of your research question. - Research question
(max 70 words) - Hypothesis
(max 300 words)
Formally define your alternate and null hypothesis as well as provide a textual description of your alternative hypothesis.
- Research objectives and experimental activities
(max 1000 words)
Define your general objectives and their specific research objectives. (you can use a combination of textual and visuals). Provide precise details about how you plan to achieve each specific research objective (eg. programming languages, technologies employed, execution of surveys, baseline methods/approaches etc.).In some of the specific research objectives, you might need to clearly specify all the details of dataset (eg. dependent/independent variables, scales and ranges, sample size etc.) as well as specific formulas. The overall goal is to allow your reader to implement your experimental design and independently achieve each research objectives. - Evaluation of designed solution with performance metrics (and statistical tests)
(max 300 words)
Describe carefully how you are going to evaluate the outcomes of your experiment statistically, the performance metrics you have planned to use, considering the concept of significance, and how you are going to accept/reject your hypothesis
Describe how findings will be related to the research question - Bibliography
References - using theAPA7 style(max 1500 words) - Activities
Timeframe for each research activity planned in section 7 (1 line per activity) + a gantt chart(max 300 words) - Appendix
Embedd the slides and material (pdfs) developed during the semester, including your previous assignments. If these are not present, marks achieved during the semester cannot be confirmed. DO NOT SUBMIT other files: embed everything in the appendix of the proposal.
Structure and marking scheme of final proposal here- If you do not receive an acknowledgement (within a couple of days) by email from lecturer it means: the proposal has not been received, or the proposal is not in the right format thus it has been discarded. If you think your proposal is correct, then it is your responsibility to contact the lecturer and be sure he has received your final submission.
An example of a correct proposal can be found here