jdigittl

I'm moving!

2007-01-25T21:19:00.000-08:00

THIS BLOG HAS MOVED TO http://blog.i2pi.com/

weatherbill

2007-01-19T20:35:00.000-08:00

I've been trying to come up with something eloquent to say about weatherbill.com, but I'm just blown away: it's just so damn cool! It embodies everything that I see as driving the future of online marketplaces. Just as intrade.com is giving economic derivatives a run for their money, I see this trend continuing with weatherbill as compared to the more buttoned down offerrings at CME.

Working at Root Markets I can't truly appreciate the perspective of soemone who is taking on a pre-existing market place and attempting to replace it with something much better. I do, however, have some theories on what it takes to succeed.

Last year I was lucky enough to spend some time with one of the founders of intrade, and I was curious to learn more about what was driving their success against the incumbet economic statistics market. The lesson was simple: you need to offer contracts that serve real needs. The message that I heard was that the payoff structure of the CME Auctions Markets were not successful in doing so. Skimming over the contract specifications and current volumes, this doesn't appear to be the case now - but it was back then. The point still remains - serve a real need - design to hedge a real risk.

Closer to home, for me, is the mortgage market. An early story that our chairman shared with me was of the evolution of the CMO market. Prior to securitization there was very thin trading in whole loans as it was commonly believed that to understand your risk exposure you needed to understand the plethora of intangible factors that could possibly drive prepayment and default risk. Buying a whole loan was thought to expose you to the risk of what side of the street the house was on, etc. Of course, with the tranche structure of the new instruments, these risks fall out leaving the holder of a CMO with exposure to the right kind of risks. Understandable systematic risks. Portfolio theory 101 at work.

As the market evolved the models improved as can be see if you compare the book for a recent issue to that of a 1980's era issue. Nowadays, with liquidity chasing liquidity, savvy investors are seeking more specialized exposure and are slicing and these tranches looking for an edge and rebundling whole loans where once the market was super thin. Specialist funds are able to do this because of the modelling power afforded by new technology. The computer I am writing this on can do a gazillion calculations per second. Thems alotta calculations.

Compare the ability to slice, dice and model on weatherbill with the handful of contracts available on the CME. If I am a small business owner, I don't want to hire someone with a degree in financial engineering to work out how to best hedge my risks. And even then, the hedge will be imperfect when there are only a limited number of contracts to choose from. Businesses want to be able to trade specialized and differentiated contracts to best approximate their risk exposure. Yes it may be difficult for an army of broker dealers to make a market in millions of contracts, but this is the sort of problems that computing power can be thrown at. If you have the right pricing models, you should be able to scale to a silly number of contracts far more cheaply than if you had to hire high rent dealers to make the market.

This sort of scalability is not afforded to all types of markets. In futures markets, where the cost of carry is infeasible, a prediction market forms. This is the case for weather, and likewise it is the case for consumer leads (my speciality). Prediction markets are special in that there is no first-order linkage between trading activity on the derivative and the current spot price. Compare this to storable goods: if the spot price today decreases, I can always buy a little more today and hold the good for use later thus changing the value of a forward contract, and vice versa. However, if it is hot today, I can't store the weather for the purpose of changing the climate in 6 months time. If the price of leads decreases today, the rapid half life of consumer attention means that a marketer will have a difficult time realizing any value from a 6 month old lead, thus leads, like weather, can't be stored.

To understand why this is relevant in determining whether a market can scale to a large number of differentiated contracts we need to look at how market makers work. If I buy a contract for a the future delivery of some good, the market maker must find some way to hedge her risk of being the counterparty to your contract. And when the volume of sales effects the spot price, liquidity risk is present. In prediction markets you don't have to balance the effect of future market activity against spot underlying values. If you have an edge in pricing future movements, you can scale with diminished exposures to liquidity risk.

In short, prediction markets are different to standard futures markets. And there is little reason to not offer highly specialized contracts to participants. Weatherbill does this very well.

By offering the ability to upload historical revenue numbers, weatherbill makes it easy for unsophisticated businesses to asses their weather exposure and to specify a contract that exactly matches those risks. It is just so easy to use and there is something, dare I say, very Web 2.0 about it all. While it is technology that empowers this, weatherbill different from, say, hedgestreet in that there is amazing simplicity to what is otherwise a complex product.

It's just so damn cool!

My bio - 8 Years ago

2007-01-19T08:02:00.000-08:00

I found this via the way back machine. I probably wouldn't be so candid today, nor would I use the same words. Nonetheless, I wrote it, and to some extent it still rings true and is available for the world to see.

response is the output of my occupation of this media volume, related to the (con/de)struction of one dimensional signals in time. or something like that. i dont really have a purpose to this site, its just a place for me to chuck my junk so that i can access it whilst i am away from my home base. i dont like going away from home, especially if i can not take at least one of my computers with me. i guess you could say i am not very well adjusted, socially. but thats ok, this is a new dawn, and there are probably many people out there like me, similarly lacking in interpersonal skills, in place for interprocess communication. why make love when you can make a remote procedure call. what is music about really ? as far as i can tell its just a tag that people can pin on them to make them different from other people, when in reality we are all very much the same. thats why if what was traditionally an 'underground' piece of music, finds itself into the 'mainstream' - the people who originally liked the music will deny that, and claim to hate it, when they are only doing so because they have been put in the position of no longer being able to use that music to make themselves special. why do people want to be special? its alot easier to become special by developing a personality than it is to become special by becoming physically attractive, or going to the gym, or cleaning behind ones ears. thats why i dont write music any more. im not very special, you see, hence any music i produce would not help others be special - hence it would fail its ultimate purpose in our consumer society. of course music can be fun to listen to and can provide enjoyment. its easy to listen to music with lyrics and walk away with a very concrete understanding of why you feel like you do after experiencing that music. when ones moves to instrumental music, certain musical constructs have evolved amongst the genres so that people can walk away easily satisified that the music that they have heard is good. howevere for the poor sods who listen to 'experimental' music its much harder for them to justify both to themselves and more importantly to others why the music is 'good' as it may not have the same constructs such melody or rhythm that allow people to judge conventional music. too often people like this 'experimental' music because it allows people to label themselves, as listeners of this music, as intellectual/special/different/horny(?). personally i like 'experimental' music for all the previously stated reasons, but also in addition, i (as a programmer and a theoritisiticsix) i like to thing about the process that was involved in the creation of the final piece. for me creation is art. i write software as art, no specific goal, just write because it allows me to hide from reality. i have no desire to earn my living from writing software. no constraints. my software does not even have to work to satisfy my. it doesnt even have to be broken in a clever way. it just has to involve me, and thinking, because, after all, its what i do best. but no better than most people.

Linkfest

2007-01-19T07:32:00.000-08:00

I just returned from vacation in Vieques. You can see the photos from Shianling's point of view here

My photos came out a bit funny

Adam found one of the old i2pi.com's on the way back machine. View it here. I must admit that I'm a bit reluctant to share this with you. What can I say - I was young.

2006-12-26T11:10:00.000-08:00

What follows is a letter I wrote in high school in response to a comment made by my religious studies teacher about the absence of ghosts in Judaism. As one of the few Jews at my Anglican school, I found myself taking more pride in my roots than when I was only one of about 1,000 at my Jewish school.

I'm posting it here with a slight reference to Joel Spolsky's latest post (which briefly talks about my current employer) and also because I am amazed that I managed to find this essay lurking in the depths of i2pi.com

It is very true to say that Judaism has very little historical associations with ghosts in the spooky spiritual sense, but to use this in the formation of the conclusion that there were simply no Jewish ghosts is pure adulteration of facts. This may seem a bold statement, but one must be so when dealing with a topic that is so ‘iffy’.

The first if in the iffiness is the definition of a ghost. To resort to a dictionary, The Macquarie defines a ghost as ‘a disembodied spirit of a dead person’ or ‘a spiritual being’; That will suffice.

I feel that there is enough evidence in traditional Jewish literature to support the claim that there were or are Jewish ghosts. But before one goes jumping into the five books one must firstly realise that the Torah only forms part of a small part of the Tanach textual trinity, containing the Torah, the Nevi’im and the Kethuvim; The Bible, The Prophets and The Writings. These three form, together with the commentaries of commentators like Rashi, the basis of Jewish ritual, custom and tradition. Without its counterparts the Torah is very much a vague thing, and it is probably in this fact where the beginnings of many a war are buried. So to form an opinion on any matter of Judaism one must at least consult the Tanach in its entirety.

If we draw our attentions to the Torah, we can find many references to spiritual beings, most of them classed under the broad heading of ‘god’, and by the second definition of ghost, we can classify these ‘beings’ as ghosts. The prime suspect for such a labelling of god himself (he is referred to as he not she in the Torah, so I will stick to that convention). Take Deuteronomy 4:12, ‘You heard the sound of words, but see no form...’. Here we have a description of god as sort of presence, who can make himself appear ghostlike to people. Now one may argue that this is a tenuous basis for supporting my aforementioned claim, and I agree, but wait, there is more.

In addition to the myriad of ghostlike appearances of god, we have the angels. These group of lucky souls have reached the court of god, and their duty is to appear on earth to carry out duties for god. In his 12th century Guide for the Perplexed, Maimonides opposed the idea that such angels are corporeal beings. So if they are not physical they must be metaphysical: ghosts. These beings pop up all through the Torah usually acting as schleppers for god. See Genesis 18:2, 3:2, and 13:6. In these references the angels appear as messengers, fire and in the last case in the form of a person; Not quite white sheet and chains material, but definitely ghosts.

It has been stated that the idea of ghosts as being merely the risen dead is just a recent concept, but if we cast our minds back to Eziekel 37 we find a very such references. In this chapter, to cut a long story short, god leads Ezi down to a valley of dry bones, whereby Ezi asks god if the bones shall every live again and god replies ‘I will cause breath to enter you and you shall live again, I will lay sinews upon you, and cover you with flesh, and form skin over you. And I will put breath into you and you shall live again’. After this Ezi was disturbed by the sounds of rising bones as they formed ghostlike human figures. Also in Daniel 12:2-3 we find further references to the dead rising from ‘sleep in the dust of the earth’ to inhabit the Earth evermore in a ghostlike presence.

So far we have had god , angels and the dead appearing in human form, or some spiritual sense inline with the conditions of the definitions stated above; We have seen ghosts, but one ghost is probably more well known within the Jewish faith than any other, Elijahu. Elijahu was a prophet, and his greatness was such that he was appointed to appear alive again before the judgment, and coming of the Messiah to prepare humanity and resurrect the dead, see Malachi 3:23. But until his appearance he is supposed to appear in Jewish custom, derived from the commentaries, annually at the household of each Jew, signified with the ritual doorpost, the Mezzuzah, where upon he would enter the house during the Passover service to drink a sip of wine especially left out for him, in his own glass. Each year Jewish children sit with one eye poised over the cup to see that fraction of a drop disappear, and each year the same joke is made countless times globally in over a hundred different languages, about how drunk Elijahu will be by the time he has drunk from so many cups, and each year the joke becomes less funny. All of this for a ghost that some suppose does not exist.

We have jumped around from one book of the Tanach to another, some pointing the finger to pillars of fire that lead the Jewish people in Ba’Midbar, others feature messengers of god coming to Earth in human manifestations. We also have god appearing to Ezikel and resurrecting the dead as ghosts. Finally we saw how Elijahu, the archetype when it comes to Jewish ghosts, and his worldwide trek drinking from the cups of Jews in apprehension of the coming of the Messiah, will be the final Jewish ghost, and with his appearance all will be revealed. Until then we will are forced to resort to one of the most holy of Jewish sources, Woody Allen; ‘There is no question that there is an unseen world. The problem is, how far is it from midtown, and how late is it open ? Unexplainable events occur constantly. One man will see a spirit. Another will hear voices, A third will wake up and find himself running in the Preakness... What is behind these experiences ? Or in front of them, for that matter? Is it true that some men can foresee the future or communicate with ghosts? And after death is it still possible to take showers?’

5 things you don't know about me.

2006-12-21T05:44:00.000-08:00

Greg just meme-tagged[1] me! The unwavering rules of the internet now compel me to reveal five things you didn't know about me.

Here we go!

In the late 80's, my family went on a European vacation. Just prior to leaving, I was in the process of becoming a cool guitarist (pre-Guitar Hero days). Of course, my interpretation of cool was slightly different to other preteens. Cool for me involved making my own guitar pickup, which involved wrapping very fine wire around little ferrite cores and encasing them in hand molded plastic. The resulting pickup was a misshapen blob with wires dangling out of it. For some reason it got stuck to the bottom of my bag and traveled around the world with me, unnoticed at any of the previous security checkpoints that we passed through on our journey.

At Heathrow Airport security was stepped up, as a few months earlier the Lockerbie Bombing occurred. (Of course, in the 1980's high-security at an airport post a successful terrorist attack was mild compared to the mess I recently had to go through after the foiling of the speculated liquid bomb attack a few month ago...) When my device of questionable intent was discovered by an X-ray inspection, my entire family was quickly surrounded by soldiers wielding M-16s. Upon manually searching my bag, they found my plastic explosive and a triggering device. The triggering device was a very neat Casio PB-700 calculator that my family bought me in Hong Kong a few weeks earlier. Being a nerd, by the time we arrived in London, I was half way through taking it apart. It was easy to understand how my innocent guitar pickup and disassembled calculator looked rather nefarious.

After a damn stringent examination of all of our possessions and individual interviews we were allowed on the plane (which I single-handedly delayed by a few hours) to continue on our flight to Paris. My parents weren't too happy about this. The other passengers on the flight were fuming. The staff on the plane refused to serve us. Meanwhile, I was pissed that they took my calculator.
Needless to say, my family wasn't happy when, the following week, I demagnetized our boarding passes at Charles De Gaulle.
In Year 9 of school (or, as you Americans call it, Middle Junior High Sophomore?!), I was placed in the remedial math class. I had just transferred schools and I clearly didn't do so well on my placement tests. What struck me as peculiar was that rather than giving the role of remedial math teacher to someone who could impart math knowledge, instead the school had our rowing coach teach the class. What I didn't learn about simultaneous equations was more than compensated for by my practical education in how to smoke in class without getting caught. I did manage to do well enough on the semester quiz to get placed in the advanced course, but by the time I graduated I was back to being a failure of the university system. My first exam, which happened to fall on my 18th birthday, ended in a dramatic failure when for the long essay portion the only words I wrote in the answer booklet were "Schwann Cells." Despite my 51% grade average, it took me 5 more years to realize that I didn't actually want to be a doctor.
At an age when I was old enough to know better, but young enough to be foolish, I was caught hacking into a phone system. Somehow I got out of that one without charges being laid. *Phew*
I despise musical theatre. I am more than happy to sit through a 12 hour production of Angels In America, but I walked out of Rent ASAP.

[1] Wow. meme-tag, that's Web 2.0++

Hussein Chalayan - Paris 7006

2006-12-10T11:44:00.000-08:00

fredwatch

2006-12-08T11:43:00.000-08:00

In the spirit of releasing early and releasing often, here is a new service I have been tinkering with the last few nights. Fredwatch sucks data down from the St. Louis fed and looks for interesting points in recently updated series and compiles 10 observations per day in a handy RSS feed. The feed is pretty clunky right now, but have at it.

I'm the operator with my pocket calculator

2006-12-07T16:27:00.000-08:00

Thirteen years ago I pestered my father enough to fork out and buy me an HP-42S calculator. To this day I love it, and feel uncomfortable when its not by my side. Not in a wierd start-developing-tics-when-its-not-around kind of way, but rather unfortable with the tought that a random calculation will go unaswered. I am terrible at mental arithmetic, and frequently need to multiply numbers. And occasionally find the next prime number larger than N. Or invert a matrix.

Infact I loved the calculator so much that a few months ago it started dying. I opened her up a few weeks ago and the PCB was quite tattered. My cleaning gave her enough life to give me time to decide on my next calculator purchase. This was a big decsision, because while I had bought calculators in the past (including a slew of HP's), nothing stacked up to the 42S. I simply couldn't buy an identical replacement unit, because that would be cheating.

The only thing missing from my 42S was inbuilt finance functions. While I had programmed everything I needed into it, I always dread the thought of losing my programs when I fail to replace the batteries in time. Given this, I decided on a 17BII. Its the same form-factor that my brain is wired to, provides most of my commonly used functions, and is significantly cheaper than a vintage 42S which is now a collectors item.

It will take a while for me to get used to not having access to trig functions, hex and binary, and programming modes. I don't think I will miss the integration, programming, stats or matrices, as these are better done on a computer. It irks me that they have included a 'BUS'iness menu which gives you basically nothing. For example, if you want to know what percent 5 is of 8, on any normal RPN calculator you type '5 ENTER 8 /' - a total of four keystrokes. Or you could use the fancy BUSiness menu and type 'BUS %TOTL 8 TOTL 5 PART %T' - seven keystrokes!?! How silly is that?

What really kills me, however, is the thought of having to see ugly buttons dedicated to parentheses when they are completely reduntant in RPN mode. (See here if you have no idea what I am talking about.)

Basically, HP introduced the 17B as an algebraic calculator with no RPN mode to appeal to business users. Of course, everyone in finance who needed to get answers as fast as possible loved RPN because you could enter calculations with fewer keystrokes than on algebraic calculators, and the method of entry was less error-prone as the operator has no need for using, or keeping track of, pesky parentheses. Of course, the 17B was a flop until re-released as the 17BII, with an optional RPN mode, but the ( and ) buttons remain to support the underwhelming algebraic mode.

The good news is that given how close the layout is to the 42S, I should be back to touch-typing on the calculator soon and won't have cope with those parentheses staring at me like the eyes of a dim-witted child. But until then, I'll probably find some cute stickers to put on the buttons so I never have to see them again.

I am such a dork.

Bespoke computing

2006-11-29T14:55:00.000-08:00

I was hoping to post this as a reply to a post on Martin Stall's blog. Unfortunately one of the short comings of his blog engine is that comments are limited to 'length 500'.

Martin,

I've been reading your blog for a while and am quite interested in how you tackled this question. As a computer programmer who spends much of his free time creating 'computer art', I'd like to add some thoughts to this discussion.

You ask if a machine can produce anything better than a copy. Lets say, for the time being, that it cannot. However any copy that it makes is immaculate. If I were to purchase a suit from you that I was overjoyed with, would I value having a 100% identical replica of it? Absolutely. I could rotate between wearing the original and the copy and halve the wear on the suit! Or if I were to have a terrible computer related accident and tore my suit irreparably, I would be safe in the knowledge that I had an identical replacement.

However, if someone gave me a 100% identical copy of Vase with Twelve Sunflowers, painted by computer with the exact same strokes, paint and aged canvas, would it be as valuable as the original? Probably not. The owner of the original knows that they have the original and mine is a copy (even if they can't tell the two apart). The value from the work comes from the fact that is was made by the hand of Van Gogh, and no more like it will ever be produced by that hand. Indeed there is value that arises from the physical work (paint + canvas), but the astronomical price is due to the intangible fact that it is unique.

I am not denying that this latter element is not a component of the price of bespoke suits, but I feel that the majority of the price comes from the quality of materials, construction and attention to the unique needs of the client. Imagine the thought experiment: you tailor a suit for me for the price of, lets say, $3000. You see that I am happy with the suit and decide to offer me an identical suit for $2000. The lower price of the second suit due to the fact that you no longer need to make initial measurements or adjustments. Now imagine that I accept your offer and return the next week to pick up these two suits. At this point I can no longer tell that one is 'worth' $3000, whilst the other is merely worth $2000. Because, after handing you $5000 for the pair, they are both worth $2500 to me.

Thus far, your reply has placed the value of a bespoke suit on the art of the tailor in designing an appropriate piece for the client. I am quite happy to accept that modern cutting and sewing technology is currently unable to capture the intricacies of a tailor’s hand. But let us assume that one day this arrives. In which case a tailor could produce the original, and machines could produce duplicates for a single client. Perhaps you may be kind enough to let this technology cope with different fabrics. Even so, a single client would never want more than possibly one or two copies of any single design. Fashions change; bodies accumulate mass and posture and position change. I am not familiar with the breakdown of bespoke suit prices, but if we imagine that 30% goes towards the labor of production (rather than the labor of 'art'), we at best may reduce suit prices by a fraction through duplication.

The real question is to whether technology can ever measure and notice enough about the client to have enough information to make a suit. Following the gathering of information we would need some technology to design the suit -- the creative genius, if you will. Rather than approaching this by tackling the measurement and then the design phase, let me limit my following discussion to the question of whether a computer could design a suit, given suitable measurements. I choose this approach because I am certain that your answer to this would be 'Never!'

My answer is 'Yes!' And the reason why I am so confident in this is that people often forget computers do not run by themselves. Software developers make computer software and software developers are people. And for examples of this I turn to the world of computer art, where artists use software as a medium. They take their aesthetic and use programs as a mode of creative outlet. While the majority of this work is in the graphic mode, artists around the world are writing software to generate music, sculpture, landscapes, etc.

Artistic expression is never limited by the medium. Whilst oils provide one set of opportunities and watercolours provide a different set, no one would say that watercolour is not art whilst oil is. Creative expression can and does flourish in many unexpected places. And there is no reason to think that computer software is not a valid creative outlet. Just as it took centuries for those working with cloth to establish techniques and artistic idioms, computer generated art is evolving and every day practitioners are refining their works.

I strongly believe that computers are capable of design, as they are guided by the hand of people who write the software.

Thanks again for a great blog, and keep cutting!

Josh Reich

Coding from the outside

2006-11-25T22:21:00.000-08:00

Nico commented on The Bug, saying that he tends to put more discipline into his leisure time coding than he might put into his work. Although my original post might give you the idea that my own leisure time coding is an undisciplined mess, this is not entirely the case. I have one rule that helps to keep my code re-usable, well designed, on target and generally clean: Code from the outside.

If I am writing audio code, I'll write as much as possible without connecting a soundcard. Likewise, I will not link against bit blitting libraries until most of my graphics code is done. This is somewhat similar to starting with test cases, but not as strict. It means writing your code, manually inspecting the output and debugging without getting caught up with artifacts.

I originally learnt this when I was doing a rotation at a remote hospital back in Melbourne. I had borrowed my father's laptop but never got the obscure soundcard working with my Linux install and so when I decided to write a synth, I could only check the output when I returned home after a few months. Despite the fairly taxing nature of working the ER of a small country hospital, I was able to get a few hours of coding done every few days. It was a good escape from tractor related injuries and whatnot. Aside from my computer, I had a copy of some audio synthesis text, which was later to be stolen by a sound engineer when I left medicine to work in post production; but that's a story for another day.

Long story short, I was able to code up the system as I had intended and I wasn't distracted by the output along the way. In the past, whenever I wrote music in software, I'd get caught up in the sounds and spend hours tweaking my code, iterating between code and output. Along the way I would forget my original intent and would end up focused on aesthetics. While this is mildly pleasing, it is not as pleasurable as sticking to your original question and answering it.

At the time my original question was "What would sounds generated by drawing a vector spectrogram sound like?". In the end, the answer was something along the lines of "Crap", but at least I learnt that. If I was hooked into a soundcard, I would have been too concerned with making something that didn't sound crap that I would have never answered my original question.

I also find that when coding directly to output, I get caught into the trap of finding bugs in the output and guessing what the underlying cause is rather than properly diagnose the problem. It is much easier to find bugs in text output than it is with audio or visual output. I think that I am smart enough to look at a complex rendering output snafu and say "Duh. Off by one error". The truth is that I am not that smart and despite this I end up randomly adding +/- 1 to an iterator somewhere and seeing if that fixes the problem. I'm somewhat simplifying the situation, but I think all coders have similar behavior. It's much harder to lie to yourself when you can clearly see the underlying mechanisms in operation.

When opaque debugging coincides with a certain aesthetic I find myself adding one to an iterator, looking at the output and being surprised by the output and saying "Wow. That's kinda cool. What if I add 2?!" Two hours later, after clocking my iterators to sin((i % 17) * 0.01), my rendering looks amazing, but nothing like what I originally intended. While this might be OK if I'm working on an animation for its own sake, but this is rarely the case. Better to separate the aesthetic phase from the analytic phase.

Or so I say anyway.

In unrelated news, Dan is blogging. Watch that space.

The Planet of Leather Moomins

2006-11-25T09:41:00.000-08:00

TPOLM is having YALS (Yet Another Lazy Sunday!)!

When: 3rd December 7006
Where: www://lazy.sunday.tpolm.internet/
Thanks: Nico for bringing this event to my attention. You will be rewarded.

Bespoke Captcha

2006-11-15T15:03:00.000-08:00

Being the half-hearted fashion snob that I am, I read a few blogs by European bespoke tailors. While I can neither afford their services nor making any constructive comments about their posts, I do feel somewhat comfortable with saying that bespoke security measures are, by definition, a failure.

Lets examine the source for confirmation:


<input type='text' name='submit_random_code_disable' style='width:100%' value='CqDI' disabled> 
</td><input type='hidden' name='submit_random_code_org' value='CqDI'></tr><tr>
<td>Random Code Verification</td><td>
<input type=text name='submit_random_code_verify' style='width:100%'>

Excuse me while I go home and hand sew a floating canvass in my RTW coat.

The Bug

2006-11-13T08:06:00.000-08:00

Whenever I interview programmers I always ask about their involvement with coding projects outside of their work. For the most part, the best programmers I have worked with code in their spare time. While I like to hear from candidates who contribute to open source projects, my eyes light up whenever someone says "No. Well, actually, I am currently working on something, but its a silly little thing, just a program to simulate people playing black jack." These little programs turn me on because they are not motivated by a desire to design something useful or to 'fit in' with other developers but rather because the coder wants to attack some abstract problem for the sole purpose of edification. When hiring, I want more than candidates who know how to solve problems. I want candidates who want to solve problems for the sake of solving problems. It isn't so much about walking away with the solution, but walking away with the process.

Since Coming to America I can count on my hands the number of times I've thrown myself into the deep end to tackle a problem like this. On my left hand was an algorithmic synthesizer I wrote over my first thanksgiving in Pittsburgh. More recently, on my right hand, I have my radiosity shader. Neither of these tools solved problems that hadn't been solved more elegantly by others. In both cases I was pretty sure that I could write something, but after completing them I dropped the modifier and was 100% sure that I could - and this was my primary motivation, to know that I could do it. In both cases my external motivation was the demo scene, where like-minded individuals write code to generate animations. The 'scene' provides a great outlet for me, even if I rarely actually participate in competitions, as I can write code where the beauty arises from the process rather than the specific solution.

I would like to participate in more competitions, but my previous collaborators are all back in Australia, and I don't have the same group focus to get things actually finished. I get to a point whereby I am happy with having solved the problem of interest, and don't have a group of friends to help me shape it into an entry. But until I rekindle that focus, I am building my tool chest one painful weekend at a time.

Such weekends are painful for all the reasons why you don't want a dev team that works the same way they code on weekends. Yeah, its cool to get an amazing amount of stuff done in a short period of time but I feel god awful today. With a long weekend starting on Friday, I had no qualms staying up to 4am working on my rendering engine. I went to bed having knocked off a bug that had been bothering me all day and could sleep soundly knowing that it was out of the way. I woke up relatively early on Saturday (read: 2pm) and got back to coding. By 6am Sunday, I was desperate for a repeat of the previous night's performance and was randomly changing lines of code in the hope of fixing another bug. All the while I knew that if I just went to bed the solution would come as soon as I woke. But on the other hand I knew that getting to sleep would be a pain with a bug still out in the wild. By the time the sun came up I had given up and was restless in bed, but fell asleep quickly. After 5 hours of sleep I woke up and fixed my bug in an instant.

For all the progress I made over the weekend I am somewhat upset by my lack of efficiency. The only thing that can turn an otherwise intelligent, mature and upstanding member of society into a
shut-in spaghetti coder is The Bug. The same drive that sends one person on a three day bender in Vegas sends me to forgo personal hygiene and get intimate with pixels. But like the fully functional junkie, I know when to step away, have a shave and make myself purty for another week at work.

This week I won't touch my code. I need to give it some time and space to breathe (c.f. bit rot. I need to remove myself and remember the original problem I was trying to solve (make pretty pictures) and the more important meta-problem (make pretty pictures out of the process, rather than the result).

jd-osity

2006-11-10T19:21:00.000-08:00

I have been pretty busy at work lately and I've been using my spare time to cut code, so I haven't been blogging much. Unlike some people who write useful code in their spare time, I like to solve problems that other people have solved in much smarter ways. Recently I've been working on a radiosity rendering engine. Unlike most modern radiosity engines, I don't rely on anything apart from my CPU. Most people these days do radiosity in almost-real-time using lots of neat GPU (graphics card) tricks. I'm a bit of an old fuddy duddy, and like to do things the hard way and am shooting for real time without any GPU tricks. So far I have got the code to almost work right. After I clean up a few kinks I'll start optimizing (beyond the simple tricks I'm currently using).

I'm keeping a scatch pad at i2pi

RMX Valuation - Part II

2006-10-24T05:54:00.000-07:00

Just thought I'd bubble up some interesting discussion from the comments of my previous post. Firstly, we have John Demayo chiming in with:

"Figure they are doing around 10x the volume outside of RMX direct. Call it 650m impressions per day, or 230bn per year."

A lot higher then I would have guessed.

Given John's current experience in this field, I was inclined to agree that my estimate may have been a bit rich, but then Pat McCarthy, who is one of the brains behind Right Media, adds some further weight to the argument that RMX is actually transparent and shares with us the following:

RMX Direct is just our newest publisher product, which is actually up to 100 million impressions a day now.

But, the full Right Media Exchange is much larger in volume. Over 2 billion impressions per day.

Which then leads John Krystynak to revise my estimate upwards, but taking down the CPM rate:

My guess would be that .75CPM is too generous, esp with the 2B imp number...

If they now do 2.1B / day, I'd guess average of 30cent CPM. Say they get 4cents so perhaps $80k /day take home...

$24 M /year. More like a 9X revenue valuation.

Otherwise, why would they take VC money at all!?

It's so little cash for Yahoo, it's just gotta be an interesting experiment from their point of view. I.e. "What to do with this RMS (sic) Network thing? - Should we throw some stuff their way?"

In other words, the Yahoo investment terms don't necessarily tell you a lot...

Right Media valuation

2006-10-18T07:37:00.000-07:00

Totally back of the envelope here, but why not add to the speculation:

RMX Direct has grown over the past few months to 65m impressions per day
Assume growth in RMX direct over the next 12 months will be minimal wrt to entire number of impressions served across their complete publisher packag3
Figure they are doing around 10x the volume outside of RMX direct. Call it 650m impressions per day, or 230bn per year.
Assume average CPM of $0.75, and they take 10% of that. Call it $170m gross.
Give them a healthy 40% margin, $70m / year
So a valuation of $45m on 20%, or $225m all up puts them at 3X

This is obviously wildly sensitive to growth and the huge volume they do outside of RMX direct - which can be estimated elsewhere more accurately. But this multiple clearly is in the range of similar deals in the internet/auction/b2b infrastructure space.

Even so, this seems very cheap.

Update: I simply must quote my favourite blog de jour Long or Short Capital:

As noted above, these estimates are empirically proven using math and advanced probability techniques.

Null Hypothesis

2006-10-03T18:52:00.000-07:00

NetFlix Prize

Registered
Downloaded
ETL'd into Postgres
Created indices
Waited
Free'd up more disk space
Calculated marginals
RMSE = 1.0560
TODO: Think

What is an inflation risk premium?

2006-09-26T11:10:00.000-07:00

I was reading a fascinating article from the Dallas Fed on the changing nature of long term rate movements (globalization Effect on Interest Rates) but am confused by the authors decomposition of the inflationary component of bond rates into an expected and risk component. Rather than emailing someone who might be able to explain it to me, I hope that a reader will comment and educate me.

If I understand things correctly, the risk-free real rate component is the market wide return required to make a risk-neutral agent indifferent between taking money today versus some time in the future. The real rate risk premium is the spread between the riskless rate and the risky rate appropriate for the entity doing the payments. Then the inflationary expectation component takes these real rates and makes them nominal by adding in the markets expected rate of inflation between today and some time in the future. This then leaves the 'inflation risk premium' -- What is this?

It clearly can't be an analog of the risk spread as inflation expectations are the same regardless of who wrote the bond. The text in the box says:

This part of R compensates lenders for the risk that inflation will be higher than expected, in which case the principal and interest returned will have less purchasing power than anticipated.

That makes no sense to me. If, at the time of pricing, there is expected to be some volatility in future inflation then this will be taken into account when calculating the expected inflation component, using risk-neutral probabilities, right? If so, then

\lamba_\pi

will be zero.

What am I missing?

Update: Duh!

Impending Doom

2006-09-26T06:33:00.000-07:00

The econo-blog-o-sphere is quite certain that we aren't in for a soft landing when it comes to the current housing price bubble. The fine folks over at Autodogmatic put not too fine a point with:

If you don't agree with me, try this little quiz: (1) Do you believe the mortgage interest rate risk in the next few years is to the (A) upside or (B) downside? (2) Do you believe median incomes are trending, in real terms, (A) up, or (B) down.

If you answered "A, B", you can give yourself a pat on the back for honesty. But you've also just implied that the housing bubble ain't comin' back any time soon.

If you answered anything else, your views are either incoherent or totally disconnected from reality.

I thought I'd ask the loan officers what they think. Well, I didn't do this, but the fed was clearly able to get a coherent sentence or two from the fine men and women at the front lines of our credit crisis. So, I thought I'd put together a little chart to highlight one small point, namely that loan officers want to relax credit standards when housing prices drop. Now if we are to believe that housing prices are falling due to A & B above, then I don't quite see how that environment is conducive to relaxed lending standards.

The Wisdom of Needles

2006-09-21T07:06:00.000-07:00

About 6 years ago (aka, long before the days of RSS), I set up an email account on my server to gather various news feeds with the aim of correlating news against market data. I also scraped a few pertinent sites and threw together some latent semantic analysis code to correlate against real time stock quotes. While my code stopped running about 6 months ago when I moved from Solaris to Linux and never bothered recompiling, I continue to amass the news feeds. And what have I learnt from this exercise: That markets are a great predictor of news. Not the other way around.

We now find ourselves in the blog-o-sphere[1] and this idea is being revisited:

Edward Hadas over at breakingviews.com (another paywall, but really good analysis site founded by Hugo Dixon) likens it to using the ‘wisdom of crowds’ to trade. I’m sorry Edward but actually I think you’ve got this one wrong. ‘Wisdom of crowd’ - mining would be things like Marketocracy and SocialPicks. Monitor110 in my opinion is all about finding the needle in the haystack; finding the individual voice or nugget that escapes crowd amplification. Finding the kernel before it becomes a snowball. Where I do agree with him however is the paradox of diminishing returns: the more people find the needle the more difficult it will be to monetise. Or paraphrasing Dash - ‘if everybody is special, it really just means that nobody is…’

Do I think there is anything in Marketocracy? Sure! If/when these ideas take off they will evolve to efficient markets that are almost as good as using real markets to predict the future. Which, to paraphrase Barry Ritholtz, is only really good for trend following. And despite our best wishes, looking at historical market data provides us with damn near nothing informative about the future.

One could argue that the problem with my attempt to correlate news with market activity is that I wasn't looking for the needles in the haystack, and in fact my haystack was being generated and distributed by journalists and PR staffers well after the event. I agree that this is a problem, but I do not see mining the blog-o-sphere as a profitable long term strategy. One just has to look at the pump'n'dump schemes that fill my inbox or the lead-gen junk that fills SERPs. When a voice no longer needs to be amplified by agreement, ala in prediction markets, it is very easy for a lone voice to manipulate others.

Prediction markets are so hot right now, but I get the feeling that people are missing out on some basic economics in terms of understand what is a prediction market and what isn't. If you can buy something now, hold it for the future (possibly a some cost) then prediction markets don't really have a reason to exist. Compare the situation with stocks verus weather derivatives. Anyone can buy a stock today and can hold it. Or they can use options to replicate that strategy. At an arguably lower cost they can even use the nascent single stock futures market to do this. This replication strategy is available to anyone. If the market has an informed view about the future evolution of returns then this will be reflected in the spot price because such replication strategies exist and there will be no arbitrage between spot and future prices. This argument holds for all commodities that can be stored.

Hence any prediction market for this class of commodities is irrelevant. Existing market mechanisms provide an environment where future predictions are embedded within the current spot price.

No one can fill their pockets with summer cooling days. I can not keep a cloud in my lounge room and unleash it on Florida during orange picking season. The weather today does not particularly informative of the weather 3 months from now and as such prediction markets are valuable for future events or things that can't be stored as there is no way to replicate strategies on the spot markets to embed information in current prices.

Yesterday my friend Jon dropped by the office and we talked about his research project as he starts his thesis under Robert Engle. One of the ideas we discussed was what I see as the three causes of information asymmetry in markets:

Timing: I know something before you do
Accuracy: I know something about something, but I'm not sure what
Interpretation: I know that X will happen which I think means Y, whereas you think it means Z

Of course, information is latent in financial microstructure. We can't readily see it or measure it, even after the fact. But lets assume that blog-o-mining has solved this. I still see nothing that gives any agent a distinct advantage in the war for information asymmetry.

--
[1] For some reason whenever I hear 'blogosphere', I think Biosphere

jdigittl on BASH

2006-09-19T22:01:00.000-07:00

Its late at night and the brain isn't feeling as sharp as it was 10 hours ago so the only thing left to do before stepping away from the computer is to check out bloglines. John Battelle links to a site that calculates the worth of my blog. My blog is worth $6k. Neat. Totally wrong, but neat.

Actually, its not neat. Silly numbers piss me off. I left marketing analytics world because people didn't pay attention to ROI metrics in 2000 like they paid attention to numbers with dollar signs in front of them.

Things have changed somewhat and now the entire world of lead gen and direct marketing live and die by campaign metrics.

But that's not the point of this post.

After toying with the calculator (aka sticking in jdigittl.blogspot.com and then leaving), I searched for jdigittl on technorati. This was the first time I did a blog-o-sphere ego search. I find only 2 links to my name -- one of which is a direct quote of a direct quote of mine that appears on bash.org (an IRC funny-quotes site).

I'm not going to link to the blog that quotes this, because whilst it is in German, it looks like a spam blog designed to get organic mortgage traffic. And in all their wisdom, the algorithms behind the site chose that quote to garner $$$ traffic.

At this point, I'd love to rant about the comparative uselessness of algorithms that 'price' things that are neither bought nor sold, but the blog-$-meter says that the spam mortgage blog is worth $0.00.

Its worth something to someone.

postscript
Ok. I just revisited the German site. They are not a mortgage spam site (as far as I can tell), but rather than editing my post I'll let my point stand. I do this for two main reasons:

That was a pretty cool segue, wasn't it?
Blogger causes Firefox to run really really slow on my Linux box

But rather than falling prey to the sin of silly numbers, let me reinforce my point with this blog. The same blog-$-meter gives also says that it is worth $0.00. This strikes me as odd given that I found this spam site linked directly from the Google organic search results. I guess that only counts as 1 link using the 'Tristian Louis' method. Actually, it counts as 0 links, because Google isn't a blog.

Now, I'll be the first person in line to state that valuation is hard. People way smarter than me may even agree with me on this point. But picking one measure, and extrapolating from that can be quite dangerous.

Let me suggest four criteria for choosing a good measure for valuation:

Measurable - Can you actually measure this for the thing you are trying to value?
Testable - Can you measure it for enough other things that have been valued by some 'market'?
Accurate - If you use your measure(s) to come up with a valuation, does it make accurate predictions?
Sane - Even if it is accurate, does it make sense? I.e., can you rationalize why it may actually work for observations outside of your sample.

Clearly the blog-o-bling meter falls foul of a few of these.

post-post script
*giggle* Blogger's spell check doesn't know the words 'Google' or 'Blog'.

CBOT

2006-09-14T18:34:00.000-07:00

A few months ago I was given the opportunity to present my thoughts on the intersection of lead generation and financial markets at the Chicago Board of Trade. As readers of this blog are aware, I am convinced that there are numerous signals indicating the desire for more advanced financial instruments to services the needs of lead generators and buyers. However, my experiences at Root (and previously at Traffion) continually remind me that adjusting market behavior is far more difficult than the economics textbooks would suggest. So while I was optimistic that change would occur, I was uncertain as to the timing. Today I am very pleased to welcome the formal public announcement of an alliance between the CBOT and Root Markets. [Press Release]

My recent work has focused on pricing nascent, and as yet untradable, swaps. Today's announcement heralds the coming days when similar contracts will be readily traded.

If anyone out there is interested in buying or selling leads at a fixed price I am happy to say that Root is open for business!

Cubes suck

2006-08-31T15:00:00.000-07:00

Until recently I had been stationed in a cube at my office. The only good thing about this was that I could peer over and chat with my neighbors if I had something to say. The bad thing (as Joel Spolsky talks about) is that thinking for long period of time is difficult. Rands talks about his den and how he can lock himself in there and either work or play. I posit that sometimes both happen at once.

Just before I left on vacation, I moved into an office. I have never worked in an office alone before, always either being in a cube or sharing an office with a coworker. My primary concern about moving into an office alone was that I would get sucked into a whole lot of nothing, involving refreshing bloglines, reading news and twiddling thumbs.

Over the past two days, to the casual observer, this is exactly what I was doing.

In the back of my mind I knew I should be working on finding an efficient way to find the distance between a point and the surface of a cube in non-Euclidean N-space. But that sounded scary. I'd come into work, clear out my emails, catch up with the news, then draw a cube on my glass wall and stare for a bit. Staring wouldn't last longer than a few minutes. If I was feeling particularly enthusiastic I'd erase my cube and draw a new one, from a different angle - hoping to grok the mysteries of 5 dimensional cubes. Then, back to slashdot. Or YouTube. Or some partially functioning Java applet written in 98 that enumerates measure polytopes while crashing Firefox.

Every now and then, out of no where, would come some piece of insight. I'd scurry across to my old cube neighbor and lay him with my new knowledge. He'd point out why I was wrong, and then I'd head back to the office for more YouTube.

But then today, for no particular reason, it all made sense and was blindingly simple. By doing nothing I managed to get what I estimated would have been a few weeks of work done in about an hour. It turns out (as usual) I was making the problem far more complex than it really was. The sad thing is, if I were still in a cube, with the constant self-applied pressure to not look like a slacker, it probably would have taken me 3 weeks to see the light and the solution would have been far more complex than it needed to be.

sony vx-2000e/canon ae-1: a dirty tryst

2006-08-29T21:10:00.000-07:00

For those who can't be bothered reading my drunken ramblings: About 8 months ago I decided to try and mount an 'old-school' lens system from a classic Canon 35mm still camera on a fairly swank Sony digital video camera. It worked.

'twas the night before the night before christmas, and josh had too much rum.
he wanted to test a hypothesis:
that the sony vx2000 has an easily replaceable lens,
contrary to what the manual says.

after about 3 hours of careful screwdrivering, my lens accidentally fell off in a kinda un-accidental way....

oh yeah, and i was kinda suprised when it still worked.

first test movie:

and here is the interesting part

the ccd fits!

it's a sonon!

the next step was to go visit mike at eyebeam, who kindly let me use their workspace and laser cutter. I used this to make some templates from mat board, then plexiglass.

using laser cutter:

testing various shapes as lens mounts:

this is what my workbench looked like... just before I used the hacksaw to void my warranty by cutting up the old lens internals to make the backing of the FD mount.

so, after i made the matching plate (the white plastic thing) i attached the FD mount to the hacked CCD mount. i also hacksawed off the front part of the metal side of the case (where the original lens used to be), this is so i can actually reach the mount point so i can easily change lenses and reach the aperture ring.

here is the view of the assembly from the front.

and the grand finale - fully assembled. you can see here i threw in another white plate to increase the lens-ccd distance as focussing was a bit tricky as above. this was the hardest part -- it really should be millimeter accurate - but its not. oh well, it seems to do the trick!

today i just opened up the ccd mount, and made some shim's from aluminium sheet. the lens-ccd distance is now within 1mm -- perfect. now it is time to buy some fancy lenses. (i have my eye on a full frame 8mm....)

Ok. I think I need to make a relay system. There is a slight problem - 35mm lenses are for 35mm film, whereas my CCD is much smaller. Hence, the image formed at the focal plane is too large. I once studied optics! I can fix this! Here is my initial sketch of how it should work:

of course, this will make the image upside-down, but i think i can tweak the LCD display to always display upside down. It already flips the image when you turn it around, so I just have to find the sensor that does that, and reverse the sense. Note, I'm also planning to switch to Nikon F-mounts as they seem to be easier to find (at least at my local camera store). As for the prime, I'm looking at the Peleng 8mm- it's pretty cheap and looks like it will do the trick.

yep. that was easy - image is now the right side up in the viewfinder (but left and right are now swapped... lesser of two evils?)

ok, after a trip to B&H (and some very strange looks) I got all the bits that I needed. But they forgot to put them in my bag, so I'm going to have to go back and get the tube. As you can see below, the system lets in a little too much light on the sides :P.

After I pick up an extension tube & make a coupler, I'll have to make an extension arm from the tripod to support the extra weight. And Then, I'll be rocking a truly tricked out camera.

ok. I couldn't find a full set of Nikon K rings, or a BR-3. So I made the missing K rings by buying 7 split lens filters, popping out the glass to make an empty tube. I made a BR-3 by taking apart a K1 and glueing it to a filter. Luckily the inside of the K1 has a 52mm flange that was a tight fit on a male threaded filter. Anyway, she is done. Well, done in the sense that all I need now is the prime lens. Ebay, here I come.

tricked out!

Update: So, I got the Peleng 8mm lens and its bloody marvelous. I ended up taking it to Australia in January, where I managed to drop it off a 4th floor balcony onto the road and under a truck. The lens now has a hairline crack in the prime glass, but generally works OK.

Below is a quick movie I edited that chronicles building the camera and the patience of my sweet one whilst putting up with me :)

Some useful links:

Peleng 8mm

Mounting 8mm on Nikon Digitals

Camera mounts & registers

Extension tubes

And a big thankyou to the people in the second hand department at B&H camera who put up with me asking for rare parts to be used for bizzare purposes.

Ethics & AOL

2006-08-07T19:13:00.000-07:00

How many people have access to this database? (I do)

Who is concerned by the breach of privacy? (I am)

Who, despite their concerns about privacy, spent a good portion of tonight browsing other people's searches? (I did)

How many ways can you use this data to make $? (I can think of a few)

Is it legal to use this data to make money through SEM? (Probably)

Is it legal to use this data to make money through identity theft? (No)

Is it ethical?

Tsk tsk AOL..

2006-08-07T09:35:00.000-07:00

For Postgres users (not the best way of doing things, but it works):


cat user-ct-test-collection-*.txt | grep -v "AnonID" | grep -v "\\\." > silly.txt
createdb aol
psql aol
aol=# create table tmp (anonid varchar(16), query varchar(1024), querytime varchar(32), 
       itemrank varchar(5), clickurl varchar(1024));
aol=# copy tmp from '/Users/josh/AOL-data/silly.txt';
aol=# create table aol (anonid integer, query varchar(1024), querytime timestamp, 
       itemrank integer, clickurl varchar(1024));
aol=# insert into aol select anonid::integer, query, querytime::timestamp, case when    
       itemrank='' then NULL else itemrank::integer end, clickurl from tmp;
aol=# create index aol_id on aol (anonid);

Funky stitching: part II

2006-08-07T06:36:00.000-07:00

I really should be posting about the privacy nightmare / SEM dream of AOL releasing silly amounts of dada last night. But precisely at the time that was happening, I was walking through the West Village trying to find St. Vincents hospital.

It would be far less embarrassing if it happened 15 years ago, but given that it didn't, it was bound to happen sooner or later, especially after my girlfriend told me not to use the paring knife as a screwdriver. Last week she went overseas for a holiday, so I had a chance to catch up on dorking out and fixed my computer and sliced my finger open with a paring knife.

My first official event at Carnegie Mellon was a 'what to do in an emergency' lecture, with extra emphasis on how expensive ER visits are. It went in one ear and out the other. Likewise, when I transferred over to my workplace medical plan I was told what I needed to do before making a claim. I didn't do any of it, because I had no plans to actually use it. You know you are in America when the first thing you think of when a medical emergency is upon you is 'Where did I put my insurance card?'. It is surprisingly difficult to remove a card that is stuck to a piece of paper when one hand isn't quite working right.

After sorting that out I had a flash of my medical training and found some sterile gauze and a bandage and wrapped myself up. This was quickly followed by a flashback to an old Bill Cosby comedy routine where he joked about his mother always harassing him to wear clean underwear, incase of an emergency. For some reason this felt like important advice, so I got dressed. Not to say that my underwear weren't clean, but I wasn't looking my best, so I got dressed to impress.

That wasn't particularly rational, but I must admit that I wasn't at my finest at that point. It is also very difficult to tie shoelaces with one hand.

I found the hospital and was impressed that I was on the 'fast-track'. I wasn't impressed by the lengthy interview (which was only lengthy because the registration admin kept on pausing to continue gossiping with her friend) and requests to sign documents before I had read them. The ER ward was fairly empty; I later found out that they were closing that section, and all the hot action was on the other side of the building. A range of nurses and doctors came by, none of whom introduced themselves (contrary to the Patients Bill of Rights document that I signed and read). And none of them brought me a glass of water, even though they all said they would.

When I finally worked out who my doctor was I told her that I used to be in her position and that seemed to do the trick. Trying to remember words and phrase from med school, but without sounding like someone who picked stuff up from ER, I managed to get her to actually talk to me, which made me feel much better. As did the nerve block.

The six stitches required to get me back together didn't take too long, and I made it out by midnight. On the walk home I felt the lignocaine wear off and figured that it was about to hurt like hell, so I self medicated with some scotch.

This morning I feel worse off from the scotch than the finger, althought typing is a bitch.

i2pi

2006-08-05T22:15:00.000-07:00

Probably quite irrelevant to my current readers, but I finally came to terms with the fact that I needed a new powersupply and motherboard, and now i2pi.com is back. This means that when I come to terms with not really wanting to be hosted on blogspot, I'll move this blog over. I really want to take control of image aliasing again; blogspot does a terrible job at it, or I just don't know how to use it. Either way, I prefer to control my web presence.

Funky stitching

2006-08-04T13:11:00.000-07:00

Either New York has some really funky architecture, or Google maps has some funky image stitching technology.

Liquidity premium = Insurance?

2006-08-02T17:55:00.000-07:00

Greg raises the point that some large lead buyers actually get discounts, contrary to my previous post. I still haven't really thought through his statement that the mortgage vertical has plenty of liquidity, but I do have a possible explanation for the volume discounts. One way to paraphrase my initial wordy post is that the premium is a form of insurance: insurance against the cost of rebalancing the relationships to maintain an orderly market. If a large buyer is coming into your exchange, and his accommodation will require the outlay of expense not only to develop his relationship, but that of the suppliers to fill his order, then you need to protect that investment with insurance. If the buyer is large and reputable then the charge will fall away. And if they provide depth to your buy side that actually encourages further supply, then they may get advantageous pricing compared to smaller buyers.

This argument becomes clearer if we make that (false) assumption that exchanges will buy all supply upfront, and then take fulfillment and counterparty risk whilst trying to sell held inventory. For the most part this does not happen, but if we replace the concept of 'lead inventory' with 'relationship inventory' then it all follows through.

Spot lead pricing II: The fishy distribution

2006-08-02T08:05:00.000-07:00

When I worked in media analytics / campaign management, when a statistic was to be reported on was that 'things' were drawn from the Normal distribution. The general arm-wavy argument was along the lines of "... mumble mumble law of large numbers mumble burp ...". Of course, what they really intended to invoke was the central limit theorem. But hey, I too went to business school and understand that MBA level probability & statistics is dull and arm-wavy and hence was a great time to catch up on sleep, so I usually let that slide. In lay terms the argument is that we don't really need to know the underlying distribution because with enough samples, things look normal. In the world of media analytics, where we had billions of ad impressions and millions of clicks, the 'enough samples' part usually held. But in the world of lead-gen, where a single supplier may only provide 5 to 25 leads per day, this doesn't hold.

What distribution do I use to model leads arriving into an exchange? The Poisson distribution. If anyone remembers their probability classes from school, they will remember countless examples which invariably involved people arriving to a queue at a bank teller. If you happened to take computer science, the example might be expressed as jobs arriving at a CPU, or something like that. Either way one of the key measures to describe these processes is to state the average time between successive arrivals. If you can make the assumption that the process is memoryless, i.e., the time of arrival of the next person does not depend on the time of arrival of previous persons, then you can model the time between arrivals as an Exponential distribution. And if you do this the total number of people arriving over a time interval T is distributed as the Poisson distribution.
In the chart above we see a the distribution of the number of expected arrivals in one day, when the average time between arrivals is 4.8 hours (one fifth of a day). We can see that the we expect about 5 arrivals in the day, which should be blindingly obvious.

I'm very fond of the R statistical programming environment. It managed to get me through my statistical arbitrage course, while I profited from the arbitrage between S-Plus ($$$) and R (FREE!). To my untrained eyes, they are pretty similar.

To product the chart above in R:
plot(dpois(0:50, 5), type='s')

If you make arrival times more frequent, we end up with a distribution that looks like a discrete version of the Normal distribution:
The big difference is that unlike a normal distribution, a Poisson will have P(X < 0) = 0. In other words the chance of having a negative number of arrivals is zero. So, if you permit my own arm-waviness, the Poisson distribution is somewhat like the discrete analog of the LogNormal.

In my previous post I made the statement that a supplier providing a large number of leads was unlikely to supply a dramatically lesser number in the future. Lets examine the chance of a supplier providing exactly zero leads if the previously provided X leads per day. In R, we express this as

plot(dpois(0, 0:50), type='s')

I won't spoil the surprise ending by including the chart, but needless to say if you provide less leads you are more likely to provide zero leads. Mathematics is wonderful for stating the obvious, but humor me here.

(I wish I could embed LaTeX in this blog...)

The Poisson probability distribution function is P(x) = l^x exp(-l/x!), where l is the mean number of arrivals and x is the number of arrivals that we want to know the probability of. If we set x=0 we get P(0) = l^0 exp (-l/0!) = exp(-l). So the chance of getting zero leads follows an exponential distribution and we are right back where we started this detour into the fishy distribution.

And as an exercise to the reader (don't you hate it when people do this..), what does the following mean in the context of lead gen?

a=2:100
plot(ppois(a,a),type='l')

How little things have changed...

2006-08-01T07:24:00.000-07:00

From the minutes of the 1990 DataServe Annual General Meeting:

Joshua Reich's Report

Joshua's report was unavailable at the time of printing the Annual Report, however it was read out and distributed to the attendees of the Annual General Meeting. Those who didn't attend and wish to obtain a copy may contact Joshua Reich on xxx xxxx. The following points were outlined in the report.

Joshua said that he had sold an inventory program to Video Classrooms Australia for $4.95. He also indicated he had written Graphar 9.5, and update to his simple, high0level mathematicians' language. An unidentified attendee noted that he had 'been sucked into chaos [theory]'. Luke pointed out that Joshua therefore must be the suckee. Joshua also told the meeting that had produced null-modem cables and plans to make a robot.

Spot lead pricing

2006-07-31T11:21:00.000-07:00

When prices look wrong there are two possible explanations, either the market is wrong or you are wrong. Bully for you if you can pick which!

Across the lead generation business I see an interesting phenomenon which is worth investigating. Large volume suppliers are paid more (per lead) than small volume suppliers. Additionally, large volume buyers pay more for their leads than do small purchasers. At initial glance this looks very wrong - as arbitrage profits can be made by buying small amounts of leads, bundling and then selling as large lots. Are leads a Giffen Good? No, but I think three factors come into play when modeling this situation: Cost allocation, quality and a risk premium.

Examining the supplier side we can easily see how these three factors may come into play. Lead aggregators (or exchanges, for that matter) have certain fixed costs associated with each participant. On the supplier side, there are the costs of technical integration, support and initial marketing expense which all have fixed components. Thus, on a per lead basis, leads from a small supplier are more costly than those from larger suppliers. Additionally, large supplier have more to lose if they were to provide poor quality leads in the exchange and it is common to see a higher return rate with smaller suppliers than for larger suppliers. These returns are costly for the aggregator and are thus reflected in the price paid to smaller suppliers. Both cost allocation and quality are factors that are readily understood in this industry, however the risk premium is not often voiced.

Dealers take a profit for providing two key services, price discovery and liquidity provision. The risk that they face is one of spread risk. In other words they make money, but are exposed to the risk of holding an unbalanced book. Before the modern era of financial engineering, where dealers (and others) can structure positions to offset spread risk, dealers would just bump their prices to cover this risk. So if a large market order came in, which would take liquidity from the market, the buyer would have to pay a 'liquidity premium' to cover this. This still happens today in the world of large block trades, but not to the same extent as in the past.

A quick point that I would like to make before returning to the lead risk premium is the difference between liquidity and volume, which I notice are often used interchangeably, especially in lead markets. Volume is simply the number of things bought or sold on a market. Liquidity, however, speaks to how easy it is to buy or sell on a market. Imagine a market with sellers providing exactly one million widgets per day, and buyers wanting exactly one million widgets. The volume of this market is one million widgets, which for the sake of argument, we will call 'large'. However, if the sellers represent all the producers of widgets and there are no more widgets to be made, a new buyer wanting to purchase 50 widgets would not be able to do so at the current market price. Those 50 widgets would have to come from the requested quantity of an existing buyer and this should only happen if the new buyer is willing to pay more. In this market we have large volume but little liquidity.

In general, market orders or offers, which demand an instantaneous purchase or sale at whatever price the market can bear take liquidity away from the market. Limit orders which set the price at which they are willing to trade provide liquidity. A market with a large number of limit orders is said to be deep. Limit orders can have two types of fill policy: all or nothing, or partial fill. If the orders for the million widgets were 100 all-or-nothing orders for 10,000 widgets each, then the market order for 50 widgets would have to pay at least 200x the per widget price of the larger all-or-nothing limit orders.

What does this mean for a small volume lead seller? Recall that small sellers are paid less for their leads than large sellers. The best way to think of this is in terms of spread risk, or the cost to an aggregator of keeping an unbalanced book. Leads are provided by suppliers in real time, whereas orders are pre-existing limit orders. If a single small sellers leaves the market there is little impact on the balance between supply and demand. If a large seller were to leave the market, a large number of orders may go unfilled. And although buy orders are mostly of the partial fill variety, if you start supplying only 100 leads to an order for 1,000 it is likely that either the order will become smaller or the buyer will leave the market. Both situations require a chunk of sales capital to bring the volume back. To prevent this from happening it is wise to pay your larger suppliers more to minimize the chance of them selling their leads elsewhere. If this indeed is the case, a portion of the premium paid to large suppliers on the spot market represents an incentive for future behavior.

Likewise, when a buyer buys 100 leads today they are making a statement that they are likely to buy 100 more tomorrow and they are paying for the probability of extracting liquidity from the dealers book in the future.

This is interesting. Without the mechanisms afforded by futures contracts, participants face differential pricing in the spot market contingent on beliefs about future events.

At ROOT Exchange I am working on developing contracts and other futures products to help better serve the needs of natural participants and speculators. Hopefully when I get some time away from working on that, I will be able to share some more analytical detail about my thoughts on these premia embedded in spot prices.

Newtonian Physics

2006-07-20T15:05:00.000-07:00

On the 18th of June I woke up, turned 28 and found my room full of balloons and books. My girlfriend had bought me a great stack of presents. After a lovely day doing birthday type things, I opened my presents to find a pile of books, many of which I already own. I think that its terribly super that my girlfriend knows me so well to have bought me books that were perfectly up my alley. One of the books is a collection of letters written by my all time favourite geek and childhood hero, Richard Feynman.

As a kid I read 'Surely your Joking...' and while involved in the Australian Physics Olympiad, I read through his infamous 'Lectures in Physics'. I felt terribly proud of myself when one day I derived the shape of water coming out of a tap and learned that he did the same exercise in his teens.

Jump forward 10+ years, and I was in yet-another-strategy meeting that wasn't holding my attention, but had a notepad and a pen and needed to come up with a way to look busy and keep my mind active. Casting my mind back to the days of APO, I drew a triangle and a box, put on some arrows for forces and started to answer a question that had been plaguing me for all of 30 seconds: how fast do you have to raise an incline so that a box starting at the top of the incline will reach the bottom in time T?

Some equations here, some there and bang! solved! And it felt really good, because even though the problem was enormously trivial, I was quite curious to see if I could remember how to solve such things. While I would be totally lost if asked "Whats Newton's 3rd Law?", I can clearly remember the seven D's of solving physics problems:

Diagram

Not just any diagram, but a BLOODY BIG DIAGRAM (you must remember that this was the Australian Physics Olympiad)

Dimensions

Mark out all the useful lengths, angles, times

Directions

Which way is 'up' ?

Data

Are there any known numbers, write them down, but don't use them yet!

Define

What are you trying to solve?

Derive

Solve the sucker

Dimension Check

If you are looking for an answer in seconds, make sure that the units of your answer aren't in cubic seconds per coulomb.

subDitute

Only at this point do you put the numbers from (step 4) into your formula to get a final number. This isn't even a real 'D', because numbers don't really matter.

You might think that its odd that there is no gut-check step listed. I think the point is that you should be gut checking along the way, and even if you don't have a good gut feeling for what you are doing, then dimension analysis and a diagram go a long way.

I quite possibly have forgotten the exact list, but I think the spirit is there.

In his autobiography, Emanuel Derman quickly talks about how bamboozled he was when he first opened a modern finance textbook. As a particle physicist he had no idea how economists could possibly need such complex/erudite/obtuse analytical mathematics to solve problems. Even the notion of 'solving' a problem in economics was bizarre in that physicists could apply relatively simple math and predict fundamental universal constants to 11 significant places, whereas economists worked in the realm of 1 or 2 significant places.

Having recently gone through business school, and one that is renowned as (while simultaneously ashamed of) being highly quantitative, I am very familiar with the inane detail involved in making extraordinarily subjective financial predictions. I recall hours of lectures on the best way to calculate WACC, knowing that in the real world (at least for my industry) people discount at 20%. Given what we know about a companies cost of equity, industry comparisons, etc., the discount rate is invariably 20%.

This is not a bad thing. This is not poo-pooing the value of quantitative analysis. In fact, it is a reinforcement of the seven D's. You need to know what is important and have a system whereby gut-checking is part of the process.

So, now I find myself with the task of coming up with a structure to open up a lead exchange that is currently hidden within a public company. The question is vague, I'm still not sure what the dimensions are, but first thing tomorrow I am going to draw myself a bloody big diagram.

its about time

2006-07-09T09:27:00.000-07:00

Well, when the last person you expect is blogging, I figure I should too. I'm not quite sure what I'll be talking about here, but I will try to keep the ego stroking to a minimum. I will probably talk about finance, lead generation, online marketplaces and whatnot.