Tuesday, June 9, 2015

Transcript - Delphi method (ii) continued

Class: Informatics, Computing, and the Future
Instructor: Dan Berleant
Transcriber: Brooke Yu
Date: Thursday, January 31, 2013

Professor:  Okay, so what I thought we'd do today is finish up on the Delphi method so everyone could have a chance to gather some data.  You'll use that for some homework soon.  There are a few people left to do that, then I have to answer some questions about the homework, and then we'll start on the next topic, which is prediction markets- another way to predict the future. 

Why don't we start with the homework? 

I know a couple of people sent me emails about how to do it.  Anybody can explain what their question was? 

Does everyone know how to do the homework?  I got emails. 

Male Student:  I'm not sure where to start. 

Professor:  Okay, let's see.  Where to start.  I guess I could say start on question 1, and that's probably where you're having trouble. 

Remember we talked about exponential curves and how they start kind of slow and then speed up?  It turns out that exponential curves.... it either speeds up or slows down, depending on the sign. 

The homework was about, to give you practice playing with that concept of exponential change. 

Exponential change, you know.... it's sort of a general formula is something.

You know, this should be t because we're talking about time.  At some time, it's the amount of- the height of the curve is 2 to the t or, maybe instead of 2 it's some constant, but an exponential curve has a formula where it's some constant raised to an nth power.  When you raise something to a power, it goes up really fast. 

Like, 5 squared is 25, 5 cubed is 125, 5 to the 4th is... what's 5 to the 4th?  It's 625, right?  That's a pretty fast increase. 

5 to the 5th is like 625 times 5 which is 3125, and we're only up to 5.  When we get to the 6th power, we're up to 15,000 plus.  It's increasing at a faster and faster rate. 

If you were to plot these, they'd be a stair-step, kind of, and each stop would be bigger than the previous one. 

So here, the variable is the exponent.  The exponent in this case is time. 

The question is how do you apply this to the homework. 

The questions in the homework are, for example... you know, estimate the doubling time of software development productity.  [On board.]  

Every year, the products available to computer programmers get better, so code is produced more efficiently.  Next year, people should be able to produce software 6% more lines per day.  So how long will it be until someone can produce twice as many lines of code as they can now?  Let's suppose people can produce 100 lines of code in x hours. 

You know, fully de-bugged- commercial production code.  It doesn't matter how many hours it will take.  The is year 1, by the way. 

Okay, then we have 3, 4, 5... [On board.]  This is how you do it. 

I'd better not say 6% here.  Let's say 5%.  In year 2, people can do 105 lines of code, if it's 5% improvement.  I'll type equal sign, e1 *1.05.

Okay?  You probably ought to know how to use a spreadsheet.  If you want to use a calculator, you can use pencil and paper to add 5%. 

So I'm going to use the same formula.  I'm just go to copy, paste it in here... copy it again here, and if you look at the formula- it's up here. 

Let me double click it.  Every time I paste it it translates the formula to be suitable for the new cell. 

Here, it's this is the previous box times 5%. 

I'll keep pasting these... [On board.]  

You can past more than one at a time.  I'll go like that.  You can see that, if you start in year 1 at 100, you're up to 200 lines of code per unit of time somewhere between year 15 and 16.  So let's estimate 15 years doubling time. 

Okay?  You can do it analytically, but this is easier and works pretty well and gives you experimentally produced results. 

Okay?  Do you want to try an example on a spreadsheet?  Ho many people want to try an example? 

Okay, here's another example. 

Let's suppose cell phone batteries- the capacity of a cell phone batter decreases by 1% every week.  Let's say the battery starts out- it lasts 150 hours before it needs to be re-charged.  That's just if you don't use it. 

And that's in week 1 after you get the cell phone. 

If it decreases by 1% per week, that means in the second week its capacity will be 150 - 1%.  What's the formula for 150 -1%? 

Male Student:  The cell above times .99

Professor:  Okay, yeah.  You could say the cell above it and just do 99% of that. 

Okay, which is actually about 148 something... there it is 148.5. 

I'll do a straightforward subtraction.  We'll do this, minus 1%, which is .01 times of the first cell, and I should get the same answer.  And this weeks.  This is capacity of the battery. 

How long will it take for the battery to... let's suppose when the batter decreases to 10% capacity of what it was when it was new, then you have to throw it out and get another one.  Let's just see when this gets to be 10% of 150, which is 15.  That's not a half time or doubling time, it's a 90% decrease time. 

So I'll just paste a bunch here [On board.]   

And I'm go to, you know 3, 4, 5, 6, 7, week 7, 9, 10... I'm getting tired of typing, so I'll do equals the previous week + 1. 

I'll copy this formula and paste it. 

[On board.]  

It's a pretty good battery.  If it goes bad when it gets to 10% of its original capacity, let's see how many weeks that is before we get down to 15. 

I'm just pasting more numbers in. 

Alright.  So at week 230 it's 10% capacity and you have to buy a new battery.  That's 4.5 years almost, so that's a pretty good battery.  Most cell phone batteries probably don't last 4 years.  So any other questions? 

So you can do it for the homework problems now, right?  Okay. 

Exponentials.  Another way of saying- this turns out to be the same thing as saying it decreases or increases by a certain percentage every unit of time.  Percentage per unit of time- 10% every year, subtracts 1% every week- those are exponentials.  You don't have to write the formulas though.  Just do it in the spreadsheet to get the answers. 

I undid the... let's go back to the course website

So why don't we finish the Delphi method for people who have not obtained data yet.  Who has not gotten their data yet? 

Okay, you're... Zach. 

Professor:  Is Kurt here? 

Male Student:  Yeah, but I went last time. 

Professor:  Okay, we'll get to you in a minute, Zach.  Who else hasn't gone yet? 

Female Student:  I haven't. 

Professor:  Were you her before?

Female Student:  No

Male Student:  I think we got everybody else? 

Professor:  Is Wiliam Smith here? 

Male Student:  I went the first day

Professor:  Oh, okay.  And Andrew, you went.  I think he went, but he's not here anyway.  Tanner? 

Male Student:  I went. 

Professor:  How about the other Zach.  Which Zach are you? 

Male Student:  Lain. 

Professor:  That's right.  Is the other Zach here?  No.  Okay. 

Professor:  So we have two people to do the exercise.  Someone explain the delphi method

Male Student:  You poll opinions on questions. 

Professor:  What kind of opinions? 

Female Student:  When it's going to happen. 

Professor:  So an opinion on when it will happen.  When will we send the next spaceship to venus?  Some people might think it's in 2030.  Others think will be sooner.  People have opinions on the date.  So you start by asking a question.  What's the second thing we do in the delphi method? 

Female Student:  You poll the audience. 

Professor:  Alright, you get all the answers from the audience.  Remember, Delphi refers to an oracle or something that gets you an answer.  You sort of treat the people as oracles.  So what's the third step.  So we find the median.  If you get a bunch of answers, it's helpful to find the middle one.  Why do we pick the median instead of the mean? 

Female Student:  Because of outliers. 

Professor:  Right.  If one person says 2013, one says 2014, 2015, 2016, then someone says 2800, it doesn't seem to average them, because it adds 200 - 300 years to the value.  On the middle value is more informative and useful than the mean.  Maybe this person picked 900 instead of 800 and it would change everything.  So we go with the median because we felt it was a better way to kind of boil down the crowds opinion.  Mathematically, it's just as good as anything else, but it seems more judicious in the opinion of people like me. 

There's no law you have to do it that way.  Can we get anything else besides the median to distill the group's wisdom?  Someone else?

Male Student:  You can discuss. 

Professor:  Alright, that's the next step.  After you get the median then you discuss.  By asking people questions to get their opinions, they haven't used each others wisdom yet.   We discuss why people picked their answers.  They learn more of the factors they may not have thought about before.  Maybe they'll guess differently later.  That's why we discuss it.  Discussing makes everyone smarter and wiser for step 5 which is to get another set of answers. 

Then what do we do?  How about step 6? 

Male Student:  You get the median again. 

Professor:  Right.  We find the median again.  And you presumably have a better estimate now than your you do before because people discussed it and have more wisdom than before. 
Okay, what else might we take besides the median?  Somebody? 


Male Student:  The average. 

Professor:  Yeah, we could take the average to give the outliers some say.  We didn't do that before, but we could.  I showed you a graph on the board which .... I don't know what the shape is called.  It kind of looks like this [On board.]  

Remember?  This was the median, this is time, and this is the middle 50% of responses, then the 25% here is an outlier of this, another 25% here might give an outlier like this.   And this is the diagram.  This is the total range of responses. 

That kind of sums up the spread or the range of responses. 


We haven't done this in class because it takes too long, but for homework 3 you can generate a graph like this for the data from your question. 

Okay, so... for those of you who haven't gone yet, I guess who are here, that's Zach and lindsey- do you sort of get how the process works?  Any questions?  Well, the first thing you're going to need is to think of a question that has a future time point associated with it that you'd like to know about. 

So let me give you both a minute to do that. 

Female Student:  I need an example of questions other people asked. 

Professor:  Okay, let's go around.  What was your question? 

Female Student:  When will humans be able to teleport. 

Male Student:  When will holograms be made for communication. 

Male Student:  When will energy based weapons be common place

Professor:  Does that kind of give you an idea? 

Female Student:  Yeah. 

Professor:  It can be anything you want. 

Male Student:  So when will wires become obsolete? 

Professor:  Okay, let's try it. 

Professor:  I hate wires.  You're always tripping over them.  So when will wires become obsolete?  Actually, one step we didn't mention was another step here- specify the question optimally.  You know, polish the question.  Does this question need to be polished? 

Male Student:  I mean, does it apply to all wires? 

Professor:  Yeah, which wires? 

Male Student:  Say, for industrial networking. 

Professor:  So for computer networks? 

Male Student:  Sure. 

Professor:  Okay when will wires become obsolete in computer networks.   Is this specific enough?  Do you mean networks within a building or institution or do you mean trunk lines that send Internet signals. 

Male Student:  Just within a building. 

Professor:  Okay, by obsolete do you mean in the US or everywhere? 

Male Student:  Everywhere. 

Professor:  So you can see the process we're going through- making the question clear and specific.  Okay, well let's take- I'll give you a minute to think of a date for this and then we'll do the process we've been doing. 

Male Student:  Over the course of changing answers, does the question change or do you just modify your answers based on the general discussion. 

Professor:  People will modify their answers based on the discussion. 

Professor:  Okay, so let's do what we did before.  This time let's start at this end of the classroom.  Just call out the years. 

Female Student:  2018

Female Student:  I say never. 

Female Student:  2030

Male Student:  I also say never. 

Male Student:  3000

Female Student:  2100

Male Student:  2045

Male Student:  2045

Male Student:  2045

Male Student:  2030

Female Student:  2030

Male Student:  2038

Professor:  Okay.  At this point, we need a recorder, because you're going to be using this data for one of the homeworks.  You'll analyze it the way we looked at it here.  Not the one that's due this week.  We need a recorder to write this data down for Zacharay.  Can you do it? 

Male Student:  Should I email it after? 

Professor:  Yeah, or you can write it on a piece of paper.  Let's go ahead and find the median.  So we'll start by crossing out the biggest and smallest numbers to converge in the middle.  [Teacher reading: [On board.]  

The middle one is 2045.  Okay?  That's the median.  And when you graph it, 2045 would be right here. 

Male Student:  What does the question is? 

Professor:  When will wires become obsolete everywhere in computer networks in a building. 

Male Student:  Okay. 

Professor:  Okay.  So now we go to step 4, and we discuss why people picked the dates that they picked, and we'll just sort of make everyone a little wiser and have more to consider.  So somebody- can someone explain? 

Professor:  Okay.  Any other comments? 

Male Student:  I say never because wires are also kind of like a backup for whenever your wireless goes down.  Generally you'll always have a backup.  I mean, it might seem obsolete, but it's there for backup. 

Professor:  Any other comments?  Okay.  The process then after people have heard a couple of things they might have not thought of.  We give them more time to think and then we'll ask again.  I'll give you another 45 seconds to think of your answer again. 

Professor:  Okay, so let's do the same process and see if things will change. 

Female Student:  2030

Female Student:  I still say never. 

Female Student:  2030

Male Student:  Never

Male Student:  2030

Female Student:  2100

Male Student:  Never

Male Student:  2045

Male Student:  Never. 

Male Student:  2030

Female Student:  2030

Male Student:  Never. 

Professor:  Okay.  This is interesting.  It definitely changed, didn't it.  Okay, we had a bunch of highs this time.  So it's still the same.  I mean the median is the same, but the spread is different.  It looks to me like there's a skewing. 

Female Student:  The average would be a lot higher. 

Professor:  Right, well, we can't average infinity in there.  It's also interesting, as you consider more things, you tend to change your estimate, which leads to an interesting possible problem with Delphi methods.  If one person has more charisma or argues more strongly, that person can have an undue influence on the group's wisdom.  Maybe they know more, or they're just more convincing.  So you know this is biased because of the bandwagon effect.  The first poll is unbiased because there's no discussion.  After, some people may be more convincing than others.  So these are someone defects with the Delphi method.  This may weight opinions more than others.  Smarter, but not weighted fairly. 

There are some other problems with the Delphi method.  Let's see.  There's a diagram that I showed you there. 

Okay, so here's a few more items.  If you were doing it as a government agency trying to figure out where the economy is going- you know, they want to support the right kind of scientific research.  If this was to be done professionally, well, for one thing, it wouldn't take five minutes per question.  You'd be dealing with experts.   It might take days.  And people might have a lot of arguments, so it'd be longer and more details. 

You can come up with any variation you want, but typically you give a starting estimate, then after discussion, give a closing estimate. 

Here's an article about it.  Well, I guess they took it down.  Never mind.  Some problems in the method - ambiguity of the question.  Maybe some people understood the question differently.  That's possible. 

Maybe someone thought obsolete meant 90% of the buildings don't have it, and others might think it meant 99% don't have it. 

I'll give you an example of ambiguity in English and how you can't get rid of it. 

Male Student:  Maybe they're talking about more than one dog? 

Male Student:  Or they're all kept on one leash. 

Professor:  Right.  Maybe someone might think the park provides one long leash.  Any other ambiguities? 

If this is the rule, then if have to put the dog on a leash in your house.   All dogs, not necessarily in the park, have to be on a leash.  Any other ambiguities?   Kept permanently?  During the open hours of the park?  Does this mean all dogs have to have a leash permanently attached?  Maybe there are some others as well. 

What's a dog?  Do puppies count as dogs?  Technically if you're a dog breeder, dog only refers to male dogs. 

Well, what if we tried to fix the problem?  We could modify the rule.  We could say dogs must be kept on their own leashes.  It's hard to get id of the ambiguities, and the rule gets hard to read. 

Male Student:  All canine species. 

Professor:  What if your dog is half wolf?  That's a problem with the Delphi method in general- ambiguity.  I'm not saying you can't come up with something, but it's hard work and it's difficult to do. 

Are there any other.... well, let's say we made the question perfectly precise, and the people involved with the Delphi effort were experts.  They spent decades working in the area.  They're experts.  We also make sure that when they discuss it, nobody is campaigning.  Everyone is represented in the discussion evenly.  Is the result a correct result? 

What do you think? 

Is the result correct?  If we do everything wrong, is the result correct? 

Male Student:  You have to wait and see. 

Professor:  Right.  You can't predict the future, so it's not a failsafe way to do it.  Experts are often wrong. 

Okay.  We've got one more.  So you get to ask a question now. 

Female Student:  When will people live on the moon? 

Professor:  Okay, recorder, have you got all this down? 

Male Student:  Yeah. 

Professor:  So can I erase it? 

Male Student:  Yeah. 

Professor:  Okay, when will people live on the moon?  Any ambiguities in the question? 

Male Student:  Permanently or visiting? 

Professor:  Let's say the earth's moon.  Okay, and is it temporary? 

Female Student:  Permanently?  I don't know.  Permanently.  

Professor:  Any other ambiguities to the question?   Okay, well take a moment to figure out your answer and we'll do the process. 

Professor:  Okay, let's poll the experts now.  We need a recorder. 

Female Student:  I'll do it. 

Professor:  Okay, just call it out. 

Male Student:  2050

Female Student:  2050

Female Student:  2050

Male Student:  2050

Male Student:  2051

Female Student:  2035

Male Student:  2045

Male Student:  2045

Male Student:  2400

Male Student:  2050

Female Student:  2040

Male Student:  2050

Professor:  Looks like the median is 2050.  Okay, any comments on why people picked the dates they picked? 

Male Student:  I think if the Mars thing is pretty successful, then they'll just want to go populate everywhere

Male Student:  Our government is already trying to get back to the moon.  It's just a matter of time once we get there what we do next.  The obvious choice is to settle. 

Professor:  Any other comments? 

Male Student:  I think we'd prefer to live on mars than the moon because there are more resources there. 

Professor:  When did they want to have colonized mars? 

Male Student:  I think 2028. 

Professor:  Did you apply? 

Male Student:  They want me to describe why I'd want to go to mars. 

Male Student:  Why would we colonize on the moon though other than if there's no place left to live on earth.  What does the moon have to offer besides a rough life? 

Male Student:  Because we can. 

Professor:  It's real expensive.  Someone has to pay for it

Male Student:  There's no resources that I'm aware of that are crucial. 

Male Student:  Also, I mean, government funding has gone down significantly for things like that, but we see a lot of private industries taking an interest in things like that.  So I mean, as far as resources on the moon-

Male Student:  Unless you can figure out by 2045 how to make artificial atmospheres. 

Professor:  Alright, any final comments on this?  Well, take a moment to think of your updated estimate.   We need a recorder. 

Female Student:  Oh, I'm do it. 

Professor:  Okay, good. 

Professor:  Okay, this is before discussion.  And just call them out. 

Male Student:  2018

Male Student:  2200

Female Student:  2100

Female Student:  2050

Male Student:  2050

Male Student:  2200

Female Student:  2080

Male Student:  2050

Male Student:  2050

Male Student:  2400

Male Student:  2050

Female Student:  2040

Male Student:  2050

Professor:  Okay, let's see what happens. 

[Teacher reading: [On board.]  

Professor:  Looks like, despite the discussion, we have the same median, but the numbers have changed a bit.  So recorder, just get this data down so she can analyze it

Male Student:  It could be a tourist destination. 

Male Student:  It could be like a gated neighborhood. 

Male Student:  You could just jump over the gate! 

Professor:  Okay, so that's the Delphi method, but it's not the only method. 

Here's one called prediction markets.  You've heard of the stock market, but this is where people buy and sell predictions. 

Here, if you say the wrong date, you don't have to worry about losing money.  If you had to worry about losing money... like if we put money on these dates... you'd think a little differently.  The idea of prediction markets is for people to put money on their predictions so that if they care about it, they'll do a better job. 

Can anyone think of an everyday example of where people make predictions and put money on them?

Male Student:  Horse racing

Male Student:  The super bowl. 

Male Student:  Stock markets. 

Professor:  Right.  In my high school, people would put a few dollars on their favorite team winning the next game.  I mean this is exactly like that.  It's like a sports betting scenario.  Sometime people will bet on the team because they like the team, but usually it's to be right.  And to make money. 

We don't have time to go over all this today. 

Sports, you know, is this guy going to win the Wimbledon tournament.  I don't know if he won or not.  Let's look at an actual prediction market. 

This is a company based in Ireland because it's not really allowed in the US because it's like betting. 

Although this is a real prediction market, it has to be based off-shore.  The company is called Intrade.  The person who started the company- he actually died climbing Mt. Everest.  That's a very dangerous thing to do, and he died on Mt Everest.  Clearly a person who liked to do risky and interesting things.  One of them was to climb Mt. Everest.  The other was to start a prediction market. 

So what is a prediction market? 

People betting to predict when something is going to happen- just like we did with the Delphi method.  If it happens you get more money.  If it doesn't happen, you lose your money. 

Ultimately, the market will always produce a yes or no answer.  The way it works- one of the bets might be "a human being will land on the moon again by 2017."  And by the end of 2017, those who bet yes will get money if someone lands on the moon.  People who bet no will lose money in this case

At Intrade, all questions are yes or no. 

When do you buy and sell?  Although the answer is yes or no, if you are to think about it now, you might not- many people might say "maybe yes or no and give a percentage. 

If the average person thought it was 80% possible, then the share would sell for $8. 

You could buy a prediction that they will or $8.  And if someone does land, you'd win $10, okay?  And if they don't land on the moon, you'd lose your money.  

Every prediction is ultimately either $10 or $0, depending on whether you guess right or wrong.  But how much you pay for a share depends on what everyone is betting.  If it was $9 per share, but you only thought it had an 80% chance, you wouldn't want to buy it. 

Ultimately, you either get $10 or you get $0, but what you pay depends on what people are betting. 

So who are you buying from and selling to?  Other people who are trying to buy a prediction they think will be true for less money. 

Ultimately, every prediction share is worth either $0 or $10 once the prediction is determined to be true or false.  So what's your potential profit?  If you buy at a low price, you might gain $10. 

What can you lose?  Maybe you buy it for near $10 but it doesn't happen and you lose your $10. 

Where do your profits come from?  Other people's losses, really.  It's like any kind of sports betting.  There's a pool of money, and those who guess right get to split the money.  Those who guess wrong lose their money. 

How do you cover your losses?  If you really wanted to use Intrade, you'd send them money and then buy shares. 

Let's suppose you buy a prediction share at $8.  The next day it's up to $9.  You can immediately share and gain $1. 

Any questions about how this works? 

Want to look at a couple of examples?

Professor:  [Teacher reading: [On board.]  

Anybody see Argo?  Do you think it's going to win an academy award? 

Male Student:  It was pretty good. 

Professor:  Here's what other people think.  Some people think it's more likely or less likely, but the average person thinks there is a 59% chance that it will.  So you can go online now and by a share that it will for $5.90.  If it happens, you get $10.  If not, you lose $5.90. 

You can see that back in, not that long ago- about a week or two ago, it was down at 5%.  When did you see the movie? 

Male Student:  A couple days ago. 

Professor:  So you didn't really know.  If you had seen it early on, maybe you would bought a share when it was only 5%.  You could have bought a share for $0.50.  Now it's $6. 

Let's look at another one.  We only have 6 minutes left.  Let's just look at a few. 

UK economy will go into a recession.  Some of these are important questions.  You can even look at future presidential elections.  Looks like democrats are a little ahead right now.  Actually, it looks much closer. 

There's a 47 to 48% chance for this.  To be even, the democratic is up a little more, so it's close to 50/50, which is what you would expect. 

What's the chance there will be an obvious air strike against Iran by December 31st.  If you think it's going to happen, you should buy a share now while it's low. 

This says that the prediction market- crowd source wisdom, says there's a 25% chance it'll happen, which is high for the question. 

These are the categories.  Anyone want to pick one? 

Male Student:  Scientific. 

Professor:  Here's the most active ones- fusion process to generate power. 

[Teacher reading: on.  That one's covered up.  What do you think this will? 

Male Student:  1%

Male Student:  80%. 

Professor:  Any other guesses? 

Male Student:  60%. 

Professor:  You have a fair amount of confidence we'll find extraterrestrial life. 

Male Student:  Are we talking about bacteria or like that guy? 

Professor:  Well, the way it's stated it would include bacteria. 

Male Student:  Then I'll say 60%. 

Professor:  I'm not sure they'll find any by December 31st. 

Professor:  3% chance.  Alright, your estimate is closer to the general opinion.  For those who picked 70 or 80%, you should buy now because if it's true, you'll get $10 at the end of the year. 
They have a lot of these. 

These are the most active. 

There must be a way to find all of them. 

Male Student:  On the left under scientific. 

Professor:  You can see that it's not when it's going to happen the way the Delphi method works.  So 3% chance by the end of this year.  If you give it another year, let's see.  11.7%.  So people there's a higher chance if you give it another year.  What about by 2015?  19%.  So I don't know how they did this.  You know, you can see how people are trying to sell right now.  These are what people are willing to buy or sell the shares for. 

You put bids in for those. 

[On board.]  

Let's do one more.  Just a quick one.  You've heard about global warming. 

[Teacher reading: [On board.]  

Professor:  What do you think? 


Male Student:  10%. 

Let's see.  Alright, it's currently estimated at 25% chance.  By December, this will be pretty close to accurate.  Okay.  So next time we'll talk more about prediction markets. 

No comments:

Post a Comment