Wednesday, October 23, 2013

And Then Some More Fun Gender Research! On Textual Analysis of Gender in Emails. Or on Water Rats.


To explain why I write about this particular study, I have to mention the sites which picked it up hot from the oven.  They are a physics site and a tech site (you might not want to read the comments there).  Someone then brought the study to me the way my mom's cat used to bring her water rats:  Like a prize but not really.  In my case it was more like:  See?  Women and men are really very different and it's physics and tech guys who are interested in this matter, possibly because they want to tell us that women aren't in the STEM careers because of biological gender differences.

The study (pdf), however, is not about biological gender differences, or at least cannot prove that the differences it argues it found are biological.  It isn't even about mathematics or tech!  It has to do with a textual analysis of Enron management e-mails when it comes to emotions!  A girly topic, really, but whatever. 

The Enron managers'  e-mails are publicly available, and that's probably why they were picked for the analysis of gender differences in the language people use in e-mails.  The here-relevant conclusions of the study:
 We show that there are marked differences across genders in how they use emotion words in work-place email.  For example, women use many words from the joy-sadness axis, whereas men prefer terms from the fear-trust axis.
The e-mails used in the research came from the Enron corpus which contains more than 200,000 e-mails.  Sounds very impressive, right?  One can do a lot of statistical analysis with that amount of data.

Except that the study doesn't do any statistical inference.  What this means is that the results cannot be generalized to anything outside the studied groups of male and female Enron managers, and the number of those, after various prunings, consists of 41 women and 89 men, and their e-mails (a total of 32,045, 12,125 of those sent by women and the rest by men).

Likewise, we are not told what positions the men and women had in management.  In theory, emails by a marketing manager might well differ in tone from emails from an accounting manager, and if men and women are not equally dispersed among all the jobs, the analyses may not be about gender as much as about the position a particular manager has.

Never mind.  The point of what I'm doing here is this:  A possibly interesting but statistically a bit simplistic study creates ways to group words into categories such as "joy," "trust," "fear," "surprise," "sadness," "disgust," "anger" and "anticipation," and then compares the created groups to love letters and suicide letters in an attempt at partial validation. 

The categories are then used to analyze the emotional content of emails sent to and by 41 women and emails sent to and by 89 men, and conclusions are shown as bar graphs of differences in the percentages of various words by gender.

It's not that the study is terrible.  It reads like a pilot study.  But the attention it received from certain groups seems interesting, especially given that the summary I reproduce above should be written like this:

We show that there are marked differences across the male and female Enron managers in how they use emotion words in work-place email.  For example, the 41 women use many words from the joy-sadness axis, whereas the 89 men prefer terms from the fear-trust axis.
 
Those corrections are because the reader is given no information which would allow her or him to construct confidence intervals or test hypotheses, and that means that nothing in the study can be reliably generalized to groups outside it, and we can tell nothing about the statistical significance of the differences the tables in the article summarize, which, by the way, makes the adjective "marked" also pretty meaningless.

The other problem I came across while reading the study has to do with the percentage differences given in the tables.  However hard I tried, I couldn't get the same differences from the  data the researchers gave.  Because I can really be quite stupid (not often, but those female genes, you know), I wrote to both the researchers asking for clarification on how the figures were obtained.

That was two weeks ago.  Usually researchers are very nice to me and respond when I ask them questions which I often do, so I'm beginning to think that this particular question will remain unanswered.

To repeat, the point of this post is not the particular study as such, but the people who found it interesting, even considering its lack of statistical analyses and its pilot-study nature.

So that's how I baked the water rats for you.