Today Maxim Mozgovoy, from the University of Joensuu, defended his dissertation titled “Enhancing Computer-Aided Plagiarism Detection” against his opponent Prof. Kinshuk, from Athabasca University. After responding to 18 slides-worth
of pointed and well-thought through questions, Maxim had a look of relief on his face as Prof. Kinshuk announced that he would recommend Maxim and his dissertation work worthy of the position of a doctorate.
The floor was then opened for questions from the audience, and as is typical in Finland no questions were asked. Had it been part of Finnish etiquette for people to ask questions and had I known more about the computer science discipline, here is what I would have asked:
“Your opponent questioned how you would distinguish your work as research instead of simply as software development, and I have three related questions to this. Your research questions on page 9 can be answered a hundred different ways. It is typical in most fields to document your particular approach to solving the problems/answering the questions, making explicit some of the strengths, limitations, and assumptions and some justification regarding the framework you chose for as the way that you wanted to go about answering those questions. (1) Did you have a framework (sometimes called a methodology) for answering your research questions? (2) If so, what was it? (3) If so, why did you not write a section about it in any of your papers or in the dissertation as a whole?”
Afterward I talked with Erkki and Kinshuk about this, and they indicated that it is much less likely for those in computer science (specifically when dealing with algorithm development) or mathematics to have such a section. Although it is more typical to have a methodology section in hypothesis-based research, I still wonder if it would be good in any dissertation to at least make some effort to understand and explicitly state what assumptions (along with their underlying strengths and weaknesses) went into the approach chosen for the research.
Any thoughts?
For certain reasons I expect seeing not many comments here saying ‘oh yes, you hit it straight on’, although many agree with you, I think.
I think, it is not that customary to talk about methodology in physics/maths/cs, since we usually assume the use of “classic” approach to prove something. In physics, you do experiments, in maths you prove the theorems, in algorithm development you experiment and prove you results — every time in the same way. Since these sciences are precise, there is no room for “alternative” methodologies, I think.
Thanks for commenting Maxim. Even after your formal defense, you are still helping us understand your work 🙂 And helping me understand more about my questions regarding the different disciplines.
You say that sciences are precise and that there is no room for “alternative” methodologies, and I wonder about that. I think the word “methodology” is a bit problematic, because of the baggage it carries with it, and so perhaps I should not have mentioned it.
Regardless, I think there are definitely assumptions made in any discipline, and even in scientific disciplines, which are often invisible to us unless we talk about them more (e.g. positivism, determinism, etc). Perhaps I will attempt to go a little deeper.
Two professors I had as a graduate student publishes a paper that helped me see how assumptions are embedded in any methodologies and/or scientific paradigms. Yanchar and Williams (2006) argue that the “metaphysics” – or assumptions underlying our science, methods, and theories – are too often held uncritically, because they are unconscious, and are thus transferred to others more easily than what is directly argued. In his discussion of the “metaphysical” assumptions underlying science, Burtt (1954) said:
Yanchar and Williams (2006) propose that although we cannot escape the theoretical assumptions we make, we can at least take efforts to make them more explicit and coherent.
Now exactly what that means – I am still struggling to understand myself? And how each discipline chooses to handle this issue seems to be a little different. I know those are complex ideas – but any other thoughts?
Yes, we are talking about tricky things. Right now, I cannot express a consistent viewpont on the subject, just some random comments…
1) In case of my thesis, it was a bit hard to explain methodology, since it is paper-based work. Each paper had its own methodology, since they sometimes were devoted to subjects of a VERY different nature.
2) In case of educational science research, I understand what “methodology” is. Since you’re dealing with people, there is no 100% method to be sure about the results. There are only methods of different degree of reliability. So you can explicitly state that you’ve used questionnaires, formal numeric results or even lie detectors — this information might be extremely important to the people who will either believe or not believe into your results.
3) At least, in our tradition (in Russia), in natural sciences people are encouraged to avoid metaphysics. It is considered as something not scientific 😉 For example: “our experiments show that molecules behave like balls”. This is a wrong approach. You should say: “if we treat molecules as balls, then we can use technique X and Y to calculate Z, and the analytic results match the experimental results”. So you explicitly state that you build a model that helps to get simpler and reliable calculations, but you avoid making conclusions on how the real world is actually operating.
4) And I have no idea what is methodology in maths and algorithms analysis. To prove the compexity of the algorithm I either use maths (complexity analysis) or experiment by running the program watching the clock. There are definitely some initlal assumptions on the input data, used hardware, etc. But I wouldn’t call them “methodology”…
What Yanchar and Williams refer to as “theoretical assumptions,” I call “epistemological awareness.” All ways of knowing included limitations and are inherently compressed versions of “truth.”
The only difference between the softer and harder sciences is that the softer ones are forced to consider these factors because their fields are too complex to be addressed by purely experimental approaches or reasoned proofs.
I somehow disagree with the statement that the would not be methodology in research of Maths/Physics/CS. Let me explain.
For me methodology is the approach how you’re trying to solve the problem and what set of tools and techniques you’re using while doing it. It is seems to be true that this “toolbox” is not documented in most of maths/physics and more mathematical studies in CS due to the fact that it is somehow obvious. When you read e.g. a mathematical proof, you will know how it has been done. E.g. if it is induction, it is clear that the author has used induction to prove it.
Maths is perhaps the most “precise” of these sciences. Physics is wide area and I do believe they sometimes really have to explain how the research has been done and how they have get to the results. CS is perhaps the most problematic of all these, since there is so many type sof CS. Just see what the dept. of CS at Joensuu is doing. Thus, I believe some effort should be put on documenting the “methodologY” what ever it is. For example there are several ways to study algorithms; you can prove it mathematically or you can test it with datasets. Thus, two approaches, two methodologies, a need to explain why and how. Again, this might be obvious while reading it, but still I think it be worth to document it explicitly.
Maybe a little reading of the philosophy of science might help here. I have started this journey and am discovering just how big an area it is.
However, Thomas Kuhn in his talk of “Normal Science” and “Scientific Revolutions” possibly gives some indication of the reasoning. Kuhn would contend that in periods between changes in the underlying paradigm (revolutions), the methods are taken for granted. They fit within the particular operating paradigm of the scientific discipline. It is only when there is disputes over the underlying paradigm that debate on methodology becomes relevant.
In Maxim’s case, he is operating within the defined parameters of the field of algorithm development. I haven’t read the thesis although I think Marilyn did but I would imagine that Maxim applied the standard techniques for evaluating algorithms such as performance, effectiveness, … In that setting, I wouldn’t expect any discussion of methodology unless he believed that there was a better way to assess the suitability of his algorithms for the domain that he was operating in.
The difficulty in the social sciences and education is that for any particular question or hypothesis, there are a range of possible ways of attempting to prove the validity of the hypothesis. Maybe, it could be said that there is no claim to normal science within these disciplines. I see computer science education research as also lacking a normal science paradigm.
Regards, Errol