feedback Archives : Those Sharp Words

So in this post I’ve decided to berate the academic community for doing it wrong in peer review. Which is a large statement for someone who is not an academic.

But the thing is, while contemplating politeness in the academic sphere because reasons, I came across a study of the discourse of peer review by the excellent Professor Brian Paltridge and was horrified.

The Mathematical Bridge in Cambridge (a wooden bridge with a geometrical design), a lawn, an old brick building and a punt proceeding down the river Cam — Photo by Chris Boland on Unsplash

My constructive criticism credentials

Some background about me is relevant at this point. I have been a CELTA (or CELTA-analogue) trainer for the last twenty years.* I’ve run well over 100 courses at this point**, and I worked out a while back that I have watched and – and this is the important bit – given feedback on over 500 online lessons (and climbing), and therefore far far far FAR more offline ones.

CELTA***, for the uninitiated, is a teaching boot camp originally designed to take people who have never taught before, let alone taught a language, and get them classroom ready as English teachers in four weeks.

Because teaching is a lot harder than that basic premise suggests, this is not necessarily a straightforward thing to do. Especially when many of the people on the course have an excellent practical grasp but an almost non-existent theoretical grasp of the subject they are teaching (aka, they are native speakers of English).

To complicate matters, I have also worked in a context where the people doing the course are generally not, in fact, newbies. To learning a language, to learning English to an excellent level, to understanding how language works, or to teaching.

Why do they do this course then? They are generally there for the opportunity to have the car of their teaching stripped down and put back together with the latest parts, cleaner, with no rust, and working even more efficiently than before.

And the internationally recognised certificate. Which is important because on CELTA there are grades. Guess which grades experienced teachers would prefer to get?

The importance of feedback

Add in the fact that this all costs a reasonable chunk of money, and that the main assessment is done not via marking essays or an exam but by watching and grading lessons, being a qualification that is designed to show you know how to teach not just about teaching, and hopefully you can see that my main role in course participants’ progress and ultimate success is trying to give clear but constructive feedback in a high stakes and often pressurized environment to somewhat stressed out people who need it so they can continue to advance and / or meet the necessary requirements of the course.****

Now, am I good at giving constructive feedback? Not for me to say. Ask the graduates of the courses I have run.

But I certainly have opinions about it, which brings me back to peer review for academic journals, where it seems to me, if Brian Paltridge’s research is to be believed, that some academics are very bad at it.

What is peer review?

Just in case, let’s review what we, or rather I, know about peer review to check that we, or rather I, have the basics down.

Academic journals, which is where academics publish a lot of their research findings, do not just take an academic’s word for it that everything they say in their paper is gold, and that their experiments and other studies have been exhaustively and properly conducted. No, they ask other academics from the same field to review and give feedback on what has been submitted before releasing said article on the world. I think the standard is to have at least two such reviewers, and that the reviews are usually anonymous, to better allow the peers to say what they really think.

Academia is a small world really.

Types of review

I daresay different journals might have different categories of grading, but let’s say that a typical overall response might be:

accept
accept with minor revisions
accept with major revisons
reject

Add to this the idea that there is a lot of pressure to publish in the academic world, and so a lot of submissions, and journals can afford to be picky and, to be fair, the whole of the academic world wants the publicly displayed research results to be correct and robust.

Peer review is, therefore, famously fierce. Negative comments far, far outweigh positive ones. Particularly, for some reason, if you are reviewer 2.

A number of memes where reviewer 2 in academic peer review is painted as being wrong and someone the meme maker hates

Now, I do not have a problem with this as such. Academia should be rigorous. It is about putting your ideas to the test, to the most stringent tests, and coming out the other side with something worth saying. And even if it is already pretty good, why not strive to make it as good as possible? Academic excellence rests on this.

What I do have a problem with is the wording of some of the comments.

The problem – too much hedging is unclear and / or rude

I appreciate that peer review is a different context to giving feedback to teachers about their lessons on a CELTA course, which is built around the idea of continuous professional development, with lessons and the lessons learned from delivering them building on one another. The teachers I work with and I are not trying to get one lesson to the highest standard possible, making people revise it, repeat it and polish it until it is.

For example, I would prefer a better balance of criticism to positive comments than the statistics provided by Professor Paltridge suggest, but I do appreciate that that is perhaps driven by this difference*****. So I am going to take what I would normally see as a demotivating and overwhelming imbalance and allow it.

Or I would if clearly peer reviewers didn’t seem to be at least as aware as I am of the likely effect of 700 negative comments and one vague positive. Sadly, what seems to happen more often than it should is that they retreat into polite obsteusification in order to try to put their suggestions in the least offensive way possible.

Except for the ones who are just using politeness as to be breathtakingly rude. Sometimes it’s hard to tell, because the point about politeness in an English speaking world is that it needs to match the context. Overdoing the politeness levels is generally a sign that you are really very cross, and using politeness as a weapon.

Examples of infelicitous wording

Here are some of the things peer reviewers said, as quoted in Brian Partridge’s book. I have slightly edited them because I wanted them to stand alone for one of the exercises for academics Paltridge suggests in his book, which is to ask potentially bemused paper writers if they can identify what the reviewer wants, which items are urgent, and which can be ignored.

See what you think. Also, can you spot the politeness strategies being used (or possibly abused)?

If one claims to present an overview of the main approaches to genre analysis, one should not reduce existing approaches to genre analysis to justify these approaches.
Is it really necessary to present the methodology section in such detail?
The author may wish to pay heed to some of the suggestions found in Lewis (1999) concerning the content of certain courses.
Some proofreading to remove infelicitous wording is probably needed.
I wondered if the author could expand a little on the benefits of students engaging in simulations.
It would be worth citing a more recent volume.
Research question 1 has not been dealt with in the paper.
Below are some references to intercultural communication I suggest the author familiarizes themselves with.
Consider making a brief reference somewhere in the paper to David Russell’s Activity theory.
I would recommend that the authors revise this to focus on using a slightly more formal register.

Rudeness

That first one was received by Professor Brian Paltridge himself. I can see two possible explanations for it.

Either the reviewer was actively being as nasty as they possibly could and using the anonymity of peer review to really let fly, or they were so intimidated by feeling they needed to criticise such an illustrious figure as Brian Paltridge, Professor of TESOL, University of Sydney, who has (I quote his Sydney Uni bio) “published extensively in peer-reviewed journals, and written books and book chapters for leading academic publishers” and is ” included in Stanford University list of the world’s top 2% most influential researchers”****** that they freaked out and overdid it. It’s just too hedged.

Some of the others are iffy too. Looking at you ‘infelicitous wording’ and ‘slightly more formal’.

Honestly. </eyeroll>

I would suggest that what is actually rude in academia to assume that your reader is such a bad academic that they cannot take a bit of robust dialogue. Sugar coating it implies that have something to be embarrassed about, which in my non academic opinion anyone who has spent all that time putting together something they offer for publication, decidedly does not.

Knowledge is queen.

Lack of clarity

However, regardless of whether or not these are examples of academics trying to start a fight with each other, when I gave this list to some non-native but very competent speakers of English they did not realise that in fact all the suggestions were not suggestions, but something the writer expected to be fixed.

Now, I will admit that giving people a list and asking them to say which are important does rather push people into assuming some of them are not. Welcome to my training seminars. Look, it gets people’s attention.

But this is mirrored in Paltridge’s book by what I assume is a statistically significant sample of reviewers expressing surprise when they got the revised versions of the paper where an insufficient number of the comments they made had been addressed.

Mmmmmmmmm. How did that come about, I wonder?

Even more than the potential rudeness in the more hedged comments, I am offended by the lack of clarity.

Improving peer review comments

So I came up with a set of suggestions, because you can take the teacher out of the classroom, but you absolutely cannot stop her from trying to fix bad communication strategies.

I am open to being told why this is not appropriate, but I will absolutely be mentally firing into the sun anyone using the impersonal ‘one’ to make their criticism, saying what is wrong without telling me how to fix it, asking a pointed question instead of anything more helpful, or presenting something as a choice when it is not. Or using the word infelicitous.

Bearing in mind there might be online forms to fill in, of course. I can’t control the forms.*******

Strengths

Start with some positives:

An overall summarising comment (This was overall an interesting read)
Include specific praise (The idea about …. was genuinely original, and the whole paper was well written and easy to follow).

To work on

Introduce the constructive criticism section clearly:

Here are my suggestions for improvement
The following areas will all need to be addressed

Tell the author directly what to do using imperatives.

Explaining why in addition to this is useful. Even if you think it is obvious, it is one of those interesting psychological quirks that giving a reason (any reason really) is helpful in getting people to do what you want.

It also still allows you to say what the problem was in a way that does not distract from the main point (how to fix it) or annoy and / or upset people (because it is negatively commenting on their darling). Avoid just telling people what was wrong.

Expand the section on the benefits of simulations by… in order to strengthen the argument in this section
Include citations to more recent research such as… so as to avoid the appearance of having missed something
Cut down on the amount of detail in the methodology section by removing… because it’s a standard procedure and the reader can then get to the meat of the paper more quickly

Suggestions

Only be more indirect in the suggestion if it is truly a suggestion not an expectation. It’s still a good idea to say why, and bury any direct criticism in the explanation. Putting this in a whole different section would help too.

Consider reworking the title to the methodology section to remove the allusion to Pterry in case it confuses rather than amuses

End on a positive note

Finish with encouragement if you cannot really add more praise (or have both):

I hope the author is not daunted by the amount of work needed here, and I look forward to reading the revised paper
I wish to reiterate my opinion that this is very much a worthwhile topic and that the paper is along the right lines, and I look forward to reading the revised version

I await feedback.

Footnotes. Yes I do know there is a proper footnote function on WordPress. This is funnier. We all still miss Terry Pratchett, right? Look, I’m not giving feedback in peer review here. I can have my little joke.

*Good grief.

**Probably. I’ve lost count. Close to that though. Might be more!

*** Or analogues to CELTA, as othercoursesdoexist, but only if they include at least 6 hours of assessed teaching, and preferably at least three times that of peer observation plus a significant amount of collaborative planning.

**** Depending on whether you see the course as useful or a hoop jumping exercise. Both. Can’t it be both?

***** To be honest, I do not believe this at all. Positive feedback is necessary to balance out any kind of constructive criticism. See this post about politeness for why. Plus, pointing out what people do well, as well as being psychologically helpful, is actually part of the usefulness of feedback – it’s not always obvious to the recipient what they have done well or why, and if you want the good things to keep happening, or in the context of peer review, for this to stay when something is revised, this is important flag up. However, baby steps, baby steps.

****** Although this may be a comment he received earlier in his career. It stuck with him enough for him to highlight it in his book though.

******* Do these need revising? I do enjoy fiddling with forms to try to make them produce better results. This is not a joke. I really do.

EFL (English as a foreign language) teaching is driven by feedback loops.

Quite how we should structure lessons is up for debate – is it about presenting some language and then providing more to less supported practice, or is it about setting up a task you want students to get better at and then working with them until they do? Who should do more of the work of analysing performance or language, the teacher or the students? Even, what language do students need to be able to handle? Grammar structures? Words? Set phrases? Combinations of words? Combinations of words which have some kind of pattern underlying them?

But generally speaking, however lessons are frameworked, they are going to be organised around an activity, exercise, task or question the teacher asks, and students will get fairly immediate feedback on their responses. And then there will be another activity, exercise, task or question. With more feedback. It may vary in type or content, but it’s there, over and over throughout the lesson. The idea is that each round of feedback refines understanding or skills, and that students can keep putting into practice the lessons learnt when they do the next step.

So one of the things I found quite hard to get my head round when teaching teenagers in a state school context in the UK was this isn’t how it’s done there.

Now, I can see why. School is really quite tiring. The concentration needed to focus intensely on something individually or in pairs and then snap yourself back to work as a class, get your head down, drag your focus to the teacher, thinking, then talking, then listening, multiple times back and forth in one lesson… Well, doing that for a couple of hours twice a week is very different to sustaining it over what is quite a lengthy school day, every day, for five days, for months at a time.

Plus, given that school groups are really quite wide ranging and large, being able to manage the lesson so that each student is ready to have their work checked at more or less the same time as everyone else for a series of short exercises is probably wildly optimistic.

So lessons tend to work in thirds. There’s a whole class presentation type stage. Then comes a stage where students are working on things on their own, which if you are really feeling your oats as a teacher will be chosen according to each individual student’s level, needs, or preferences. And then there’s a plenary, where whatever they are supposed to have learned is checked, but in a sort of broad ‘what overarching principle have we learned this lesson’ sort of way. Specific outcomes for the exercises students have been working on will only be looked at when the teacher has time to take in books and mark them, assuming they were that kind of task in the first place.

And this will not be every lesson.

As you can imagine, this gives a very different pace to lessons, and a very different way of learning., and a very different idea of what makes a lesson work. It should, in theory, make a teacher think a lot more about what outcome for the lesson they want to achieve, rather than measuring success by a series of correctly answered exercises for a start.

I’ve been thinking about this because teaching online, it turns out, needs to be run a bit differently to teaching face to face.

A teachers stands in a forest surrounded by her pupils. The affordances of this environment will shape how she teaches. — Image by Sasin Tipchai from Pixabay

The main issue is that everything takes longer. Particularly pairwork. Mainly pairwork, even. I mean, don’t get me wrong I love how intimate the space of breakout rooms can be. No longer are you trapped in a large echoing room with the buzz of voices all around – there you are, just you and your partner, together, preferably on a sofa with a cup of coffee.

Different sofas, I’ll grant you. But a sofa none the less.

And as a teacher I can hear the students (in that room) a lot better than when my attention is fighting to tune out everybody else. Online pairwork is great! But every time you use pairwork online it comes at a time cost. And so you cannot use it for all the stages of the lesson that a face to face teacher might.

Now this in itself is a good way of figuring out when pairwork is truly important, not just for maximising students’ opportunity to speak, but for the purposes of collaboration, peer teaching and so on. It cannot just be the default.

Then the challenge is deciding how to monitor. Because if you cannot listen in to everybody at the same time, or you cannot listen in to anyone because you have not got the time for breakout rooms, then you have to figure out ways to get feedback on what students are doing or what outcomes they have reached that is inclusive of as many of the class as possible.

Which brings me to the point that online teaching may be a bit more teacher led in places, but it can also be more inclusive. It’s very easy to throw out a general question to a face to face class and let yourself fly with the fastest because you think you have some idea of how everyone got on.

This is not something you can let yourself do when you are working with a group online.

Luckily, not only are there online tools that help with remote monitoring, like Google Docs, but features on the teaching platforms themselves, like the chatbox in Zoom, allow you to do this. You just need to figure out what to use and when.

(Just. Ahahahahahahaha).

What has all this to do with discourse analysis and online communication, you may be asking yourself?

Not much. ‘S my blog. I can write what I like.

Oh go on then. It’s a nice example of what is known as affordance, a term coined back on the 60s* to describe what the environment allows someone to do, the way that humans shape the world around them to facilitate their lives, and that learning to use what is around them appropriately, natural or not, is a crucial aspect of learning how to fit in.

Affordances don’t need to be physical objects. They can also be someone’s talent, skills or a desire for something to happen.

But affordance became associated, in the fullness of the 80s, with product development and programming and similar, connected with designing things to be used in a particular way, preferably so that people would understand and be able to use them in the way intended without intensive instructions. The form would fit the function, sort of thing, that reading the manual would be superfluous.

Why yes, in case you are wondering, a man most certainly did come up with that idea**.

The term affordances also comes up when people talk about online communication and the ways that different platforms shape language and language use in ways that are different to we are used to, or think are normal. Some of which is an intended design feature, and some of which is users adapting in novel and interesting ways to their environment. More of a bug, in fact. Think of the rise of the emoji as a solution to the problem of not being able to use a quirk of an eyebrow or intonation and so on to indicate when you were trying to be a bit tongue in cheek.

To be honest I think ‘allowances’ is a better word, but then a) its inventor wanted a totally new word for his concept and b) it does connect to the phrase ‘afford someone an opportunity’ which puts the whole idea on a resolutely positive footing.

Which definitely brings us squarely back to the topic of this blog. Online communication’s affordances are different, sure, but that just changes the way you go about it, and what you might get out of it. It isn’t necessarily a debased form of real communication, just as online teaching isn’t necessarily an imperfect copy of ‘real’ face to face teaching.

Now excuse me while I just get back to stripping out all the unnecessary bits from my next lesson plan in order to focus on the essentials.

*By, if you must know, a psychologist called James J Gibson.

**Donald Norman.

Tag: feedback

Reviewing Peer Review Comments – Am I Reviewer 2?