Izabela Korwel

Data transformation in R

izabela — Sat, 23 Aug 2008 19:28:16 +0000

I just dug out from the bottom of my drawer a very old set of data. The story is long, but in an essence what I am trying to do is to make the data look normal (normally distributed). Let’s see how it would progress in R.
Step 1- I drew Q-Q plots for all my 73 variables only to discover, that 5 look decently close to normal. Most of them look long-tail skewed, long-tail symmetric at best:

For anybody interested, two simple commands in R:

qqnorm(variable1, main=’text you want above plot’)
qqline(variable1)

Step 2- basic thing to do, log transformation:

log.data <- log(data)

and re-check the Q-Q plots. Seemed that the transformation helped the long-tailed skewed data:

but not the symmetric. It also seemed to mess up previously looking normal data :(. Anyway, 17 variables still needed to be taken care of.

Step 3- when I was at it, I tried square root transformation, with effect similar as above. It mostly helped the long-tailed skewed, especially the ones with outliers on this end. By the way, to do it in R (I know,it’s basic):

sqrt(data)

Step 4 - looking through some literature, I found a paper with idea of hyperbolic sinus (sinh) or inverse hyperbolic sinus (asinh) transformations for symmetric non-normal data. The idea is to do it on median adjusted data.

Step 5- I got ready for some power transformations.
Boxcox in MASS library works only with lm and aov models, so I dismissed it quickly.
I Googled bctrans command in alr3 package. Error quickly stopped me:
Error in optim(start, neg.kernel.profile.logL, hessian = TRUE, method = “L-BFGS-B”, :
L-BFGS-B needs finite values of ‘fn’
Looks like maybe two variables are too similar, as suggested here?

Before I figured out Box-Cox in R, the project went back into the drawer. Anyway, some thoughts on data transformations by a chemist, not statistician….
What I find interesting is that in case of multivariate data set, like mine, you have all kind of distributions, and some are closer and some are further from normality, almost each one is of its kind. By trying to apply one transformation to all of them, you also affect them all.
Just to compare how different transformations affected my “good” and “bad” data:

And I am not convinced that this is a good thing. I also realize that statisticians used it for years and probably proved it superior to not transformed data. But I would love to see some comments from people working with “real life” problems.

Technorati Tags: normally distributed, Q-Q plots, long-tail skewed, long-tail symmetric, log transformation, power transformations

Terrible NA’s

izabela — Thu, 03 Jul 2008 20:32:18 +0000

In R project, most of stuff is simple unless you have NA values (missing data). I struggled with them already in PCA. This time, I am not talking about any complicated programming, models or what not. No. All I want to do in this moment is to calculate a median!
I discovered on more than one page on the Net that summary functions like mean(data), median(data), var(data) or range(data) when data set contains NA, require na.rm=T to work:

median(data, na.rm=T)

Surprisingly, summary(data) will work without NA argument. Who would guess.
But, to make my life more miserable, mean(data, na.rm=T) on my data set works, while median(data, na.rm=T) produces only an error :

Error in median.default(set) : need numeric data

(Problem unsolved so far. E-mailed my R-guru, but he is on vacation till next week)

After few e-mail exchange with my R guru, there are following conclusions:

median(as.matrix(data), na.rm=T)

gives you median for whole data set, not variable by variable. Useless.

median(data[,3],na.rm=T)

gives you median for variable in column 3. If you have 120 variables, I guess you need to do it one by one. Not more helpful.
Bottom line? No solution at this time…

Technorati Tags: R project, NA values, missing data, summary functions, mean, median, na.rm

R for Dummies

izabela — Thu, 03 Jul 2008 19:57:39 +0000

Just a short note on what I just discovered on the Internet, looking for some R stuff. It turns out, there is something like R-Commander. In simple words- R with pull-down menus and basic features, something like R for Dummies :). And by basic features I mean data summaries and graphs, t-test, ANOVA, clusters, PCA, distributions and some more!
I am installing it on my computer right now. All you need to do is to install package Rcmdr, and load it. If you are missing any packages (quite many, in my case), you get a message and R takes care of it by itself.
R-Commander itself seems to be rather intuitive and easy to use.

Technorati Tags: R-Commander, R

Hotelling T2 test

izabela — Wed, 02 Jul 2008 16:06:38 +0000

This time an easy assignment for R project. I needed Hotelling T-test, to compare multivariate means between two data sets. It turns out, all you need to do is to load the package ICSNP, and use function HotellingsT2. OK, with R there always needs to be a trick, and you always have to learn something else meantime. And here, you need to have you data divided into two sets, by group, with no group labels. So, if you have everything in one data frame, function subset comes in handy. Nicely described in Verzani book, the function allows you to restrict rows using logical phrase in subset= or columns, using numbers in select= as in example below:

X <- subset(data, subset = group == 'group1', select = 2:10)
Y <- subset(data, subset = group == 'group2', select = 2:10)

and then it is simple as that:

HotellingsT2 ( X, Y, test = ‘f’ )

The other option for test is ‘chi’, if your data is not-so-perfectly-normal (you can check package manual for more details).

Short note- this needs to be told what to do with outliers. na.action=na.omit or similar solves the problem.

Technorati Tags: R project, Hotelling T-test, ICSNP, HotellingsT2, subset

PCBs in Miami

izabela — Tue, 10 Jun 2008 21:23:03 +0000

As much as I love my job, sometimes I wonder how really important is what I do. Well, I am starting almost each scientific paper from “Polychlorinated biphenyls are important environmental contaminants” or something of this sort. But are they really? To keep me going believing in it, I really need from time to time to read the articles like this about PCBs being found in whales or like that, featuring PCBs and other currently monitored pollutants in human blood, both published by National Geographic. But in fact I never seen PCBs being present in more “contemporary” literature before. Until today. I am currently “reading out” Edna Buchanan crime stories, and here it is, on page 43 of “Suitable for framing”. They are talking about PCBs which were found in copier ink and are contaminating air in the office. Can be as well true! I guess it means that somebody, not just us scientist, not only knows, but also cares about those PCBs around them. And it reminds me as well about the interesting PCB-related web page somebody mentioned on last meeting. It was started by a parent who found out that PCBs may be present in caulk, and contaminate soil around schools, where children play. Really, when I think how little research translation most of us, scientist do, I wonder where people really find all this information! I am sure they don’t read our scientific papers and journals, I don’t read them myself! I will print out and read articles related to my work, and that’s it. They are just plain boring. But that’s good that they do, and that’s good that I am working on something that matters. After all.

Technorati Tags: Polychlorinated biphenyls, National Geographic, PCBs, Edna Buchanan, research translation

IKEA bike

izabela — Thu, 05 Jun 2008 21:32:56 +0000

I guess most of you know what IKEA is? The self-assembly Swedish style furniture, in Europe wide-known and ubiquitous chain? And I know they don’t make bikes. However….
My husband bought me a bike. One just like this one. It arrived in a nice box:

And with the know-how knowledge of my husband, when I came back from work, the bike was ready to ride. What a disappointment for me! I am getting bike for assembly, and don’t even get to see the assembling process? Luckily, the frame was not exactly for my height, so I needed to get a new frame. This time, instructed step by step, I disassembled the bike with my own hands into pieces, and when the new frame arrived, mostly assembled it myself. The final result:

I guess most fun was from using the torque wrench :).

Technorati Tags: IKEA, self-assembly, bike

Magazines subscritpion

izabela — Fri, 30 May 2008 17:48:49 +0000

The logic of America still fascinates me. Yeah, I know this is money driven country, but..
What I am talking about now, is the magazine subscription. For example, I enjoy reading National Geographic, leaving alone the fact that their photos and photographers are considered the best. The easiest way not to worry about not missing next issue is of course to subscribe to it. That’s what I do. First year, it was OK- I used the piece of paper in the magazine I bought, got special price, was happy. Next year, I got the renewal suggestion for … 30$. Guess what? If instead of renewing I would buy a new subscription, my price would be what? Whole 19$. I was enraged and e-mailed them about it. I succeeded in getting the price of 19$. This year, situation is less ridiculous, but still… My offer? 19$, doesn’t sound bad at all. But, if I decide to buy a new one, this time is 15$. So I am being punished for being loyal? Doesn’t make sense to me at all.
I am not even mentioning the fact, that the minute I click and buy a new subscription, their Customer Service will send me a subscription renewal offer. For another year. Already. Rigth away, in case I forgot I just did it. And then for another year. I think I am getting up to 6 offers per year. If I react to all of them, I would quickly have a subscription till the end of my life!
Anyway, I am renewing 3 months in advance only. What a waste of money for sending out this notices. My money, I should add.

Technorati Tags: magazine subscription, National Geographic, subscription renewal

Talk at the conference

izabela — Fri, 16 May 2008 15:32:04 +0000

Next week, it is a big day in my scientific carrier. I am going to give a talk at the international conference. I am very exited. I find it as a great chance to kind of force people to notice my research ;). It just seems to be more direct interaction than poster, at which people may stop or may not, depending on how interesting graphical explanation of your results you prepare. The problem is that my research is rather inter-disciplinary. It is too biological for chemists to notice, and too chemical for biologist to care.
At the same time, I am stressed like hell. Not only my boss already twice removed some slides from my posters, making me to invent my story from the very beginning. Also, telling the presentation is one thing, answering all the questions people may ask is another. They can ask me about anything they like, most probably- anything they are experts in. Which not necessarily is something connected to my research…
Anyway, I know a lot of people preparing for a presentation like that would prepare slides in Power Point, then sit down and write their whole presentation. And then I guess try to learn it by heart?
I don’t do that. I prepare slides, thinking how I am going to talk about them. And then I just run the presentation and just talk about them. It seem like harder way of doing that. But at least, I will find myself in front of 150 people in big room and if I freeze, I won’t need to remind myself what the heck was the first word of the first sentence of my introduction to my talk….
My boss already gave me one the advice I really like. Focus on your plot- explain axis and what is presented on the plot, and the story will come to you…
Anyway, I wonder how many people really write the presentation down, and how many prefer to go live?

Technorati Tags: talk, international conference, poster, presentation

On Open access publishing

izabela — Thu, 01 May 2008 17:35:42 +0000

The big issue this year is the new NIH policy requiring to provide the copy of accepted manuscript resulting from NIH sponsored grant to open access repository in biological sciences database PubMed. As much as I was always thrilled by open access publishing, I am starting to seriously rethink this opinion. New policy requires that every manuscript accepted after April 7, 2007, describing research financed by NIH, needs to be submitted to PubMed. You have some options though. You may send the “dirty manuscript”- your file as of the day the manuscript was accepted - or, in some cases - the final published paper (after proofing, journal style re-wording etc.).

The “case” depends on the journal in which final manuscript is published, or really what copyrights you retained when submitting. The article, depending on your choice of way to submit it, may appear in PubMed right away, or after 6 or 12 months. It may be exactly the same file as published paper, or it may differ quite considerably (but not in the scientific content!).

Of course, the publishers of non-open-access journals worry about their profits, and it looks that they have the basis for that. Some, like American Chemical Society, suggest that you pay $1000 (per manuscript!) to retain the right to open access distribution of that article, and thus it may be published on PubMed (or your own web page for that matter) right away. Seems to be a bit costly to me. As somebody commented today, it is a week or two worth of supplies, not a spare change really.

Now, in the beginning the whole open access was invented as the way to provide to everybody - be it the US taxpayers or whole scientific community even in poor countries - non limited access to all the results, papers and ideas. The question is, if it is not going to block or at least seriously limit the possibility for the scientists in poorer countries to publish in popular, and thus probably expensive journals? They currently have an option, because only NIH requires posting papers, and if you don’t want to buy open access, you don’t have to. The questions is, if paying large amounts of money to have your paper published (which to some extent is already present as per page fee in some journals, but not in many) is not the direction in which the open access movement is going? And this is my impression it may be the case.

Technorati Tags: NIH, open access, PubMed

On scientific review

izabela — Wed, 26 Mar 2008 18:54:05 +0000

If you are a scientist, from time to time you hear a discussion about Open Access or Impact Factors… You for sure have your opinion on those “hot topics”. Now, is anybody ever thinking about how unfair the peer review procedure is? The reviewer always knows who wrote the paper he is reviewing, and I am sure quite often people who should do not refuse. Isn’t reviewing somebody’s paper the best way to get to them? And you will never know, because the reviewer remains anonymous. I am leaving alone the fact how often reviewers know little about the subject and ask really….. trivial questions.
Now, there are two possible ways to change the peer review process. My favorite would be to make it double blind- reviewer doesn’t know whose paper he is reviewing and the author doesn’t know who reviewed his manuscript. Sounds fair to me. But recently discussing it with my husband, he came up with another idea- the double-open peer review. Scientists prepare manuscript and simply put it out to the open discussion by everybody who cares. Of course if you ever read all the comments to news stories on any news web page, you realize that the discussion to be constructive needs to be somehow moderated. Moderated, not censored! I cannot think of simply making that type of open review happen, but I like the idea.

Technorati Tags: peer review