mathbabe

One of the positive things about working at D.E. Shaw was the discipline shown in determining whether a model had a good chance of working before spending a bunch of time on it. I’ve noticed people could sometimes really use this kind of discipline, both in their data mining projects and in their normal lives (either personal lives or with their jobs).

Some of the relevant modeling questions were asked and quantified:

1. How much data do you expect to be able to collect? Can you pool across countries? Is there proxy historical data?
2. How much signal do you estimate could be in that data? (Do you even know what the signal is you’re looking for?)
3. What is the probability that this will fail? (not good) That it will fail quickly? (good)
4. How much time will it take to do the initial phase of the modeling? Subsequent phases?
5. What is the scope of the model if it works? International? Daily? Monthly?
6. How much money can you expect from a model like this if it works? (takes knowing how other models work)
7. How much risk would a model like this impose?
8. How similar is this model to other models we already have?
9. What are the other models that you’re not doing if you do this one, and how do they compare in overall value?

Even if you can’t answer all of these questions, they’re certainly good to ask. Really we should be asking questions like these about lots of projects we take on in our lives, with smallish tweaks:

1. What are the resources I need to do this? Am I really collecting all the resources I need? What are the resources that I can substitute for them?
2. How good are my resources? Would better quality resources help this work? Do I even have a well-defined goal?
3. What is the probability this will fail? That it will fail quickly?
4. How long will I need to work on this before deciding whether it is working? (Here I’d say write down a date and stick to it. People tend to give themselves too much extra time doing stuff that doesn’t seem to work)
5. What’s the best case scenario?
6. How much am I going to learn from this?
7. How much am I going to grow from doing this?
8. What are the risks of doing this?
9. Have I already done this?
10. What am I not doing if I do this?