Writing, submitting and publishing a paper contains the best of times and the worst of times. There is great pleasure in getting your thoughts down on paper, hammering on them with earlier research and the critical eyes of all the available authors, editors and peers and putting them to test of scientific method. We’ve gone through this process twice now with Bloom’s revised taxonomy and five psychotherapeutic games.
One of the thoughts at the centre of my research for a Philosophical doctorate, was that greater transference of game content into life beyond, might be achieved when a game would address a meta-cognitive level. This is an interesting higher level of cognition where your actions are decided – possibly by you. All very well, but first we needed a solid determination of this meta-cognition and when it was happening during gaming before we can start testing it’s power of transference. I found the Revised Bloom’s Taxonomy which included a category for meta-cognition and wondered if this might be suitable to task.
While discussing such considerations with a fellow Horizon2020 reviewer and game researcher, who seemed interested in this concept of meta-cognition, we came up with a plan: Let us try this taxonomy on five different psychotherapeutic games and see what we get. We did this work and shaped our results into the paper Looking for Metacognition subsequently send to GALA 2016 (International Conference on Games and Learning Alliance) where it was reviewed, accepted and won 3rd place in the best paper competition.
Rejoice! This distinction also meant that we were invited to publish an extended version of our paper in the IJSG (International Journal of Serious Games) special edition. I saw fit to use this opportunity to confront a limitation of our work that had been bothering me – if I wanted to use this Revised Bloom’s Taxonomy across different games to identify meta-cognition for further analysis, I needed to prove that it had intercoder reliability. We decided that adding the process of TESTING our application of the taxonomy on the five games would provide enough of an extension – and that we were in need of more researchers to perform the coding with us. Luck was with us, as two more befriended researchers considered this work to be interesting enough to pursue, and on we continued with the four of us.
Headlong into trouble. Our process of applying the taxonomy proved itself as highly subjective and we concluded that this taxonomy was entirely unsuitable for the kind of work we had had in mind for it. This was not what we had expected – but a question once asked must content itself with the answer it receives. We described our process in detail and sent out our findings to IJSG to be peer-reviewed. Our work was returned to us with many comments. Most of which greatly added to the structure and clarity of the paper. Most of the critique of our peers was much appreciated and the paper was changed accordingly. Some comments threw out our process under ‘bad methodology’ due to the results we obtained. One of our peers felt that if the taxonomy did not meet the criteria of intercoder reliability, we should work on a protocol until it did meet the criteria and that that paper might be a work worth publishing. We disagreed.
Now, our paper wasn’t the greatest of works. No great academic achievement, no awards were awaited by any of us. Certainly, there was much to improve and with the help of another round of reviewing, improve it we did. And yet comments remained that we could not bend to without disavowing our results. We disagreed. We were notified that our work would not be published.
So our combined efforts lay in vain. Restructured, debated and refined into a simple point being made in great detail. No great revelatory work, but I felt none the less that it was work that would contribute in a small way to a greater understanding in our discipline. I happened upon a journal (International Journal of Computer Games Technology) that matched the interests of our work, practiced publishing open to everyone and willing to waive the payment it usually required. Quickly, I adapted our work one more time, to the style demands of the publication, and notified my fellow researchers as we send out hope once more. The editor returned that they were interested in our work and once more it moved onwards to be critiqued by our peers. We received the comments of just one reviewer. Most of the comments were small and adhering to them improved the work further, some of the comments questioned the method due to the results. We disagreed.
We were notified that that our work would be published: Using the Revised Bloom Taxonomy to Analyze Psychotherapeutic Games.
So here we stand – at the end of a road that took two years. A road that has seen much debate and effort of all kinds, for which we are grateful. At the end we can conclude simply that the thoughts we had did not pass the act of being tested – which is helpful, as it prompts us to look elsewhere. If we are to find what is efficient we also need to distil what is unsuitable.
Both papers – and other work – might also be found here
The (un)proving of such an hypothesis, leads to the question of measurement. Currently, there is no decent measurement for transference of in-game behaviour and experiences to ‘real life’ behaviour and experiences which takes into account the interactive nature of a game environment and the more abstract knowledge levels that might be the key to behavioural success.
