Estimating correlation with missing data

Wed Oct 27, 2010 by brett

There’s a great article here on estimating correlations and means when you have missing data in your datasets. It uses the Expectation Maximisation algorithm to calculate the missing values, and what is interesting is how much the implied correlation changes after applying EM.

Slinging Wood

Mon Sep 20, 2010 by brett

We finished our four week archery course with 2020 Archery this weekend. We had a great instructor, and we both enjoyed the lessons.

How to get Maplin's USB Microscope working on a Mac

Sun Sep 19, 2010 by brett

I just brought Maplin’s USB Digital Microsoft with 400x magnification, which is advertised as having Windows-only drivers. System Profiler identified the chipset as being from Vimicro Corporation. I went to the Driver Download section of their website and downloaded (and installed) the DRV_ZC0301PLus_070305 driver. I then opened up Photo Booth, and was able to select the Vimicro camera as the video source. What I should have done was just try and access the Microscope using Photo Booth before I installed the driver, as I half suspect that there was no actual need to install that driver.

Close-up of a new razor blade

Close-up of a new razor blade

Please try to use the microscope with Photo Booth first before downloading the driver, and let me know in the comments if that works.

How Disqus does scaling

Wed Sep 8, 2010 by brett

Here is a great presentation given by Jason Yan and David Cramer of Disqus fame about how their site was architected in order to scale to handle 75 million comments.

"Nevermore" quoth the Paperclip

Sat Aug 21, 2010 by brett

I sit at my keyboard tapping, many formats o’er lapping, searching for a template for a letter to Lanore.

Lanore, my Facebook friend long-standing, but my distance she’s demanding, a restraining order landing, with the full force of the law.

So I’m at the keyboard thinking, with my broken heart a-sinking, trying to find the words to reconnect with lost Lenore.

“I see you’re trying to write a letter, can I help you make it better?”

Quoth the paperclip.

But I click him off dismissive and continue with my missive, giving him no more permissive, to disturb me evermore.

As I sit there thinking, my scotch quietly drinking, my hope it is a-shrinking, needing words to lead me back to lost Lenore.

My broken heart almost stopping, I try to stop the pop-up popping, but the paperclip comes back a-hopping, “Let me help you woo Lenore”.

Again I click him off dismissive and continue with my missive, giving him no more permissive, to disrupt me evermore.

And then the blue screen does appear, my heart is burning with the fear, that I’ve lost my missive dear, whose ending is as yet unclear, my missive to my love Lenore.

“Your letter is nevermore” quoth the paperclip.

My knowledge of computing, all evidence refuting, makes me try rebooting, restarting that single core, to reclaim my lost lettor, my final missive to Lenore.

But the letter has been deleted, replaced instead by one image repeated, a paperclip looking conceiting, completely filling my hard drive store.

And the paperclip whispers “Nevermore”.

by Helen and Brett

Squid Wars

Sat Aug 21, 2010 by brett

Another flash story, this one written in about 30 minutes. Any resemblence to real life characters is purely coincidental (no shotgun blasts to the face for me, thank you very much).

“Red Leader, you are cleared for attack”. How did I find myself here; a marine biologist attacking an American space station armed with a squid?

I blame the World Cup. It was during 2010 that people realised that an octopus named Paul was correctly predicting the outcome of the games. It didn’t take Dick Cheney long to see the implications of this and found the “Future Crime” division of the FBI. He kidnapped the prescient octopus from Germany, and started arresting trouble-makers and future insurgents BEFORE they had committed any crimes. Of course, the crime families decided they needed some means of retaliation, and they soon started kidnapping marine biologists and experimenting with their own cephalopod recruits. Soon the word “army” was a literal term. I was a prominent marine biologist specialising in octopi, so I was one of the first to be forcibly recruited to the Rebel Alliance.

By 2012, Dick Cheney had declared the UN obsolete and was running world affairs from a space station orbiting the planet. By this time he had had his third heart attack and was breathing through a respirator. Although old, he was still vigorous, and would stride menacingly down the corridors of his space station, dressed in his gleaming black space suit.

The Rebel Alliance decide to attack the space station, but the only way to avoid the attention of Cheney and the prescient Paul is to be armed with your own cephalopod. My companion is a squid, the 2nd squid recruit of Delta team 2, who we call R2D2 for short. He is strapped in behind me as we start our attack run. I am sweaty and feeling slightly nauseous. R2D2 has been fitted with a synthesiser and he chirps behind me, an instruction to veer left. A missile from the space station flashes past, just missing our starboard wing. “Nice one, R2” I say. “Almost there… Almost there…” I mutter to myself, and with a slap of the launch button, the missile is away and the station is destroyed.

Of course, it is almost certain that with his prescient ally, Dick has escaped and will some day continue his reign of terror. For now though, the world is safe, for at least as long as the closing credits.

You, me and her

Fri Aug 20, 2010 by brett

Life was so much simpler when it was just the two of us. We’d explore such avenues of thought together, you and I. Walking arm in arm through the boulevards of well-established fact before ducking down a muddy alley and romping in the puddles of the arcane. I came to trust you, to lean on you, to depend on you. And you never let me down. Whatever I needed, you could always lay your hands on it. You gave me everything.

And then I met her. At first it was wonderful, the three of us, together. We would talk and drink. Wild talk, sometimes, but always there would be ideas. Facts, those precocious particles of truth, would be sprinkled about the conversation, and I noticed a change in you. Whereas before you would support me, more and more you were backing her up. I would glance over, only to find the two of you deep in your own explorations. I would make some pithy comment, only to receive some mumbled reply from her, she could barely take her eyes off you.

And I want her all to myself. So, sorry iPad - I’m banning you from the dinner table.

How I ended up in Hell

Thu Aug 19, 2010 by brett

(This is a flash story - a story written in 15 minutes on a train to Wales)

Want to know how I ended up in Hell? It was Billy’s stag night and so far a total disaster. Because of Billy’s best friend Bobby, we were sitting at a raw food restaurant in London - a raw food restaurant that had run out of most vegetables. I had a plate of “spaghetti” in front of me. Spaghetti made from shaved zucchini. Cold spaghetti. Spaghetti covered in tomato sauce consisting entirely of squashed cold tomato. “Don’t worry” whispered Billy, somehow picking up on my abject misery. “Wait until the girls arrive”.

Oh yes. That would be something to look forward to. Because Billy hadn’t been able to organise a stripper, he had somehow persuaded a troupe of girl scouts to come in and do some miming to a Spice Girls track during the meal. I think he had promised them some special badges or something. I listlessly twirled the spaghetti around my fork.

Abruptly there was a commotion from the front door of the restaurant. “Oh God” I thought, “it’s the girl guides”. I looked up, and instead saw my nemesis walking in the front door, an attractive girl on his arm.

Kevin and I had been rivals ever since chess club in school. Ever since I had lead with white pawn to queen 4, and he had responded with the Sicilian defence. The Sicilian! In my heart I knew he had mocked me ever since. And now here I was at a table with a bunch of geeks, about to be serenaded by a girl guide troupe, and here was Kevin - walking past the table with an attractive blonde, no doubt with an expensive sports car parked outside. Kevin glanced down as he passed the table and saw me. His eyes widened in surprise. “Brett” he said, stopping. “What a surprise”. “Kev” I said, nodding at him, “it’s been a long time”.

Suddenly there was a crash at the front door. Everyone turned to stare. “Zombie” I thought. Yup, rotting flesh, shambling gait, there was definitely a small group of zombies staggering into the restaurant. Kevin and I looked at each other and nodded in perfect accord. He grabbed a chair and smashed it against the table. I grabbed a carving knife in one hand and Kevin tossed me one of the chair legs. Ichor and chunks of rotting flesh flew across the restaurant as we set to work dispatching the zombies.

And that’s how I ended up here in Hell. I mean, how was I to know that the resurrection had started?

Granny O'Grimm's Sleeping Beauty

Sun Aug 15, 2010 by brett

Try to get to sleep after watching this!

Granny O’Grimm’s Sleeping Beauty from Darragh O’Connell on Vimeo.

Of Gaps and Grass-Eaters

Wed Aug 11, 2010 by brett

Are the American people obsolete? Salon argues that because of globalisation and technology there is now a increased separation between capital and labour. The activities that generate wealth have both been outsourced to cheaper shores, and become more efficient because of technology. As a consequence the social contract in Western society between rich and poor - the rich provide the capital while the poor provide the labour - is breaking down. The rich still have capital, but they can now move the production of goods to the East, creating a shortage of jobs in the West.

Philip Greenspun wonders if unemployed people are 21st Century equivalent of Draft Horses after the Industrial Revolution. Is it far more expensive to keep people in work, than automating their jobs, or outsourcing?

Finally; DailyFinance has a story of how the disappearance of the middle class and the increasing gap between rich and poor has affected Japan. Apparently those that have given up on ever breaching that economic chasm and who retreat from society are termed “Grass-Eaters” by the media. Is this a taste of what will happen in the West if this trend continues?