Tuesday, June 9, 2015

Transcript - Delphi method (i)

Class: Informatics, Computing, and the Future
Instructor: Dan Berleant
Transcriber: Brooke Yu
Date: Thursday, January 24, 2013

Professor:  Okay, so it looks like we're still waiting for a bunch of people.  It's just about half the class here.  Do you think others will show up? 

Male Student:  Well, maybe.  It's cold. 

Male Student:  Joshua Jenkins is sick. 

Professor:  Yeah, the whole network is down, but oddly enough, it works on my laptop.  This laptop usually doesn't work very well, but it seems to be working just fine now. 

Well, let's go ahead and get started.  How many people are new here today?  Or maybe they were new last Tuesday.  I know the number of people in the class has gone up to about 18. 

Any questions about homework one which is due today? 

Yeah, I mean, technically you have until midnight. 

So let's go to the course website to recall how to get there.  You just click on the homepage and you go to the blog which contains the course, and so homework one is due today.  It's not major or huge, but it is a homework, and today I'm just going to pass out homework two. 

I did get it online, so I thought I'd roughly go though it first.  You can always get it online. 

It'll have two questions.  First is based on last time's lecture. 

Remember, we talked about the exponential curves.  It'll be a chance to try out some of those things yourself.  If you run into problems, let me know.  If you're not from a quantitative background, we'll figure out something else, but many of you can, so we'll have a chance to try to guess or use a spreadsheet to figure out the doubling times of a couple things. 

For example, [On board.]  

There's a few questions like those on the board. 

Again, I want this course to be accessible to everyone, even if they don't have technical backgrounds. 

Then question number 2 is... to get started thinking about the term project.  We're not going to wait until the end of the semester when there's an enormous crunch.  With every homework you'll do more of it. 

You'll come up with topics you're considering in 150 words or more. 

Topics can be anything having to do with the future.  The format could be like a paper, or it could be a creative writing project or you class write a song or make a computer program.  Whatever you want to do. 

Most people do a research paper, but all of these other things are options and they usually work out great.  People love to see skits and hear music.  You can work as a team or individually

You're not committing yourself with this assignment.  You can choose another assignment later. 

Any questions? 

So again, the reason we're talking about the term project so early is simply so you don't get stuck at the end faith semester with a big project.  By the end of the semester you'll just have to glue a few pieces together and it'll be done.  Alright. 

Oh, here's an announcement.  If you're interested in getting a job as an entry level web programmer, let me know and I'll give you some information. 

Tell your classmates about it and let me know if you're interested. 

Okay, now for today's topic- last time we talked about how many things in the future fit a model of an exponential or S-curve. 

Today we're going to look at a different approach to predict the future.   Do you all know what crowd-sourcing is? 

Male Student:  Getting people together to solve one problem. 

Professor:  Right, you have a bunch of people contributing their knowledge and the composite of their knowledge is better than what any one individual could do.  Crowd sourcing is has proven itself everyth from Amazon which has all those book reviews done by readers is a way of compiling information. 

Movie reviews are the same- they tell you how many viewers rated things.  There's a way to try to predict the future using a crowd source method.  It's called the Delphi method. 

Maybe we'll take two or three classes to talk about this. 

First, I'll tell you about the Delphi method and then we'll use it in class to try to predict some things using you as the crowd to source your wisdom to make predictions. 

So I'll lecture and then we'll do an exercise. 

So today we'll start to learn the Delphi method, then we'll use it in class to extract wisdom from your classmates. 

So before I talk about sort of what the Delphi method is, figure out a question that you'd like to crowd source from your classmates the answer. 

The format is "when will ______ happen." 

So take a couple of minutes to think about and jot down the answer to this fill in the blank.  Okay? 

I know you don't want to just listen to me talking all the time, so we'll do an exercise, but do to the exercise, you have to fill in that blank.  So I'll give you 3-4 minutes to do that, and then we'll go on. 

[Students writing questions for exercise]

Professor:  How many people have thought of their questions?  So some of you are still thinking. 

Professor:  Okay, so later on and next time you'll ask these qestions and we'll get the class to use the Delphi method to come up with an answer.  Let's see what that method is. 

Delphi is actually a town in Greece.  I don't know if you know these letters- delta, epsilon, lambda,....

An oracle is a entity which can predict the future, okay? 

And it was the site of the Delphic oracle, which in the religion of ancient Greece was the oracle of Apollo- a Greek god, and also the name of a spaceship that landed astronauts on the moon. 

When I was a youngster, we actually watched on TV as a person landed on the moon.  Very exciting. 

Here's the symbol of the Apollo space program, and it's named after that Greek god, Apollo. 

So an oracle is a source of wisdom or prediction.  This word has the same root as the words oral and oration. 

An oracle says a prediction. 

An example of an oracle is the sort of concept of a crystal ball.  You've heard of fortune tells who take a crystal ball, look into it, and tell you your future. 

A crystal ball is a sort of oracle.  I think it's a mirrored ball. 

Actually, it's a piece of art- somebody drawing what would appear as a reflection in a crystal ball or a mirrored ball. 

This is actually a painting by a guy named Escher who is known for this weird artwork. 

Here's another crystal ball oracles.  Here's another showing- I don't know what that is.  Maybe clouds? 

If you look into a crystal ball it's kind of weird. 

So who uses the Delphi method?  Government agencies sometimes use it. 

In Japan, and this and kind of complicated structure, there's a science and technology foresight center which was a part of [all things [On board.]  

So you can see how governments work- very complicated.  There was an office in the Japanese government that used the delphi method to figure out when technological things would happen. 

The US government cares too about when technological things are going to happen.  They want to fund research in these fields where their country will reap economic benefits of these products.  So it's good for the economy. 

All major countries have these agencies- the US has some centers dedicated to foresight.  I'm doing research now where we're looking at the robotic strategic plans of various countries. 

We look into the future to try to guess what will happen with robotics to figure out how to keep up. 

I mentioned the Japanese team because in 1995 they had a team touring the US talking to people, and I was working in Fayetteville at the time and we had a chat about the future.  They must have used those results somehow in their report.  It uses the Delphi method. 

That was the whole point of that agency- to determine the future

I'll show you... this is the kind of thing that they came up with. 

They asked a bunch of people questions about when something was going to happen, and the was a range of response- some people said sooner, other things later. 

This is the median or average response. 

For this technology, it was 2025, then they take this middle 50% and then they take sort of the more extreme cases- this is 25% of the answers, and another 25% after. 

Okay, so.

So that's sort of a graph they came up with for the results of the delphi method. 

So let's see how the Delphi method works. 

It's a way of extracting wise predictions from groups. 

There's variations on the method, but basically you start with a question.  When is something going to happen.  It should be a sharp and answerable question. 

I've taken this from a book- Jerome Glenn is a big name in technology foresight.  His book suggests that questions used in the delphi method should be sharp and answerable. 

Then you present... you take a vote.  You say "when is purple cotton candy going to cost 10 cents."  Then everyone writes down the year, you collect the estimates- that's without discussion

Then people discuss and talk about why they picked the date they did. 

After a bunch of discussions, you have another vote. 

So the point is to... you sort of take a naive vote and then the people learn more about what everyone else thinks, and then you have another vote.  To kind of get the discussion going, they recommend paying particular attention to the extreme opinions.  Not because they're likely to be right, but because people have informative reasons. 

One of the problems with this method is that if someone loudly proclaims something, other people might just sort of jump on the bandwagon, and you want to avoid conformism because you want independent estimates. 

Similarly, if someone sort of campaigns on their estimate, that can distort group wisdom as well.  That's why people write down their answers before discussing first. 

Anyway,finally people do discuss and people reconsider their answers in light of discussion and give a final set of answers which can be graphed showing the median, the description of the spread.   This spread shows the middle 50%, and the total range as well maybe- the minimum and maximum. 

So why do I specify the median and not the mean? 

Who knows what the difference are those? 

Male Student:  The median is the middle, the mean is the average. 

Professor:  Okay, so you have 9 people, then the 5th person will be the middle or the median estimate. 

What's wrong with the mean in a problem like this?  What could be the... a risk of taking the average of people's estimates? 
Well, is the mean always the same as the median?  Okay, why not? 

Male Student:  The average can be skewed by someone who is far on one side or far on the other, but the median kind of better represents where everyone stands. 

Professor:  Right.  The median is 2.  The mean here is 4 and a third. 

So the mean is skewed because one person said something really high. 

People talk about average income of an American, and most people don't make that much.  Turns out there's a small amount of people who have enormous incomes so that the average income is quite a bit higher than the median or the income at which the most people fall close too. 

So it's a skewing issue. 

That can happen when you're estimating dates because someone might say something won't happen for 300 years, and that will skew your average if you decide to take the mean. 
Means can be skewed by extreme data points. 

That's why I use the median here.  So how many people are there in the class? 

There's 14 people in the class today, so the median would be... if you all estimated a date and we sorted them lowest to highest, then I guess the median would be midway between the 7th and 8th person in the class.  There's no one person in the middle.  You pick the two middle and take the midpoint. 

Okay.  So any questions about how this is going to work?  Because we still have some time.  We're going to use your questions and do this. 

Okay, well a little bit of nuts and bolts before we begin. 

I'm going to ask each person to give their questions.  Someone will have to write all of this down. 

For each question, we need a recorder to write down the estimates to give you after class.  So you'll have the data for later. 

Alright?  So for every question, each person will have a chance to ask a question, and we need a recorder to write down the data to be analyzed later.  Who wants to go first? 

Okay, well, that's an interesting question to discuss. 

When will nanotechnology happen. 

Do you all know what that is? 

Male Student:  It's really small particles of things you use for all different kinds of things. 

Professor:  Okay, yeah.  Technology of really tiny things.  The problem- remember, these questions have to be sharp and answerable. 

So this is a good example of a question that could be made more specific.  We need to sharpen this up a bit so we can have a more specific answer.  Can you think of a specific, you know, type of nanotechnology.... a nanobot. 

So when will nanobots be sold?  How about be sold?  Things like nanobots have already been shown in labs as partial demonstrations, but none have been sold. 

Okay.  Now I'll give everybody a chance to think of your answer- a date or a year.  Don't tell anyone yet.   So independently think of a year, write it down, then after a minute or two we'll crowd source your wisdom on blackboard. 

I'm recording the answer. 

Okay, I have a recorder.  I have our first victim


[On board.] 

This is kind of a big class- bigger than what we had last year.  Let's analyze it.  Let's look for the median. 

I'm going to sort of put them in order now [On board.]  

Median, 2028.  That's our first guesstimate of when nanobots will be sold.  Now let's discuss some of your reasons.  Anyone want to explain why they picked a certain date? 

Male Student:  It just seems, as much as technology is growing it shouldn't be too far away. 

Professor:  What was your estimate? 

Male Student:  2025

Male Student:  I said 2035 because I figured it had to be before 2046 before singularity.  You'd have to have nanobots that are pretty prevalent.   Plus, it's growing now in colleges. 

Professor:  Yeah, we have a nanotechnology building. 

Male Student:  I picked 2032 and I picked that because the more we see in society and research, the more it will expand in the next 20 years.  I mean, given the fact that we put enough research into building it. 

Professor:  Okay, now you've all heard some of the arguments and so maybe you'll want to change your answers.  Let's start at this end. 

[On board.]  

Professor:  I'm guessing that if we took mean it would skew the data.  It looks to me like there's a bit of a difference, but let's see.  The recorder is writing all of this down. 

So it's exactly the same.  We have the same estimate.  So everyone has to take turns.  Let's get a recorder. 

Male Student:  When will humanity's extinction happen? 

Is the question reasonably sharp and specific enough? 

Alright.  Let me give you a minute to think of your answers and we'll write them down and do the same thing again. 

Professor:  Okay.  Anybody need a few more seconds?  Let's start again.  

5 billion 2,013

Anyone not have a chance to go?  Okay.  A couple of comments.  Like, effie these folks, why did you pick that number? 

Male Student:  Around 5 billion years is when the son is going to turn into a red giant. 

And who picked- let's see. who piked 2400?  Okay, why? 


Okay, any other comments?  Anyone?  Who picked one of these? 

Male Student:  I don't know, I guess at the rate we're going, we'll kill ourselves off before too long.  So recorder, have you gotten all this data?  Let me go over here and we'll redo this. 

Let's start with you again.  Just call them out. 

5 billion 2013
5 billion 2013
5 billion 2013
5 billion 2013
5 billion 2013

Typically people start to converge, but the middle looks like American politics now. 

So 4 2400s, five of these and that leaves these two [On board.]  

That's the middle. 

Oh, we never really analyzed the middle here.  So let's see what it is. 

So 6040.  There was a change here.  It went from 6040 to 7000. 

Okay.  So recorder, have you got that all down?  Alright. 

Why don't you give that data to him now and I'm going to pick another. 

Why don't you be the recorder this time and you ask a question

Female Student:  I don't have one yet. 

Professor:  Well, think of one because you'll need it eventually. 

Male Student:  He stole me question- what year will the world end. 

Professor:  Is that the same question. 

Male Student:  Humans could be extinct once the world ends. 

Male Student:  Actually, it'll probably be better once we are extinct. 

Professor:  I'm torn between doing the question because maybe enough is enough.  You'll need some data for your assignment.  Do you want to go with that question or think of another question?  You don't have to think of a question instantly. 

Male Student:  I'll do it. 

Professor:  Okay, when will the world end? 

So while I'm erasing you can think about whether or not you want to change your answer and we need a recorder, which will be you. 

I think we can run through this one pretty quickly. 

How many people need another moment to think of an answer? 

Female Student:  What does the world consist of? 

Male Student:  Planet earth. 

Professor:  Right, the question has to be clear.  Okay.  Alright, let's call out the dates again. 

5 billion 2014
5 billion 2014
5 billion 2013
5 billion 2013
1 million
5 trillion
5 billion 2013

Would anyone like to think of a reason as to why they changed their estimate?
Well, the end of humanity is in 2400, so in 2399 so we move to mars and then a year later we all die. 
Professor:  Because I didn't convince you in the last lecture when we talked about the colony on mars. 

So the group wisdom here is that the human race will outlast this.  Any other discussion on this? 

Okay.  Let's see if anyone changed their mind. 

Let's start with you again. 

5 billion 2013

[On board.]  

Male Student:  You skipped me.  I guess 2399

Professor:  Okay, same kind of analysis

So this is our median- [On board.]  

Somebody didn't vote. 
Male Student:  I gave the world an extra year after we leave. 

Professor:  Anyone else? if you want to say never, I can write never. 

I was reading a paper about when the sun turns to a red giant, they think it won't quite get to earth's orbit, but it'll probably survive. 

Male Student:  My extra year is for the orbit change. 

Professor:  Okay any other comments about why you picked these numbers?  Who is the recorder for this one?  Let's see what the median is. 

Okay, so the recorder, do you have the information?  Pass it back to the questioner.  Then we'll go on to the next. 

Who wants to ask the next question. 

Male Student:  I will.  When will underwater colonization happen. 

Professor:  Is this question explicit enough? 

Male Student:  We're talking like cities underwater. 

Male Student:  It doesn't matter.  Anywhere underwater. 

Professor:  Well, let's go through this again.  Think of your answers. 

How many people need a few more seconds?  Do you all have a date in mind? 

Let's hold off on discussion until we get to the next point.  So let's do the same thing.  Just call out the answers. 


Professor:  Okay.... let's see.   Let's start analyzing this.  Have you got this all done? 

So let's see... [On board.]  

Professor:  Wait, 1313, this has already happened. 

Okay, low... high. 

So 2027.  Any time people talk about colonizing mars, an interesting question people ask is what about weird environments on earth. 

If we're not going to colonize the oceans before 2070, why should we expect to be able to colonize mars? 

Okay, any comments about why people picked the dates that they picked? 

We need to have some discussion, or there wouldn't be a reason to change your mind. 

Male Student:  We're already trying to figure out how to settle in space.  We have to expand somehow, and the oceans are the next best thing. 

Male Student:  We have more land mass underwater than we have above water.  So there's more land to be had down there

Professor:  Well yeah, the sea level is rising.  Maybe rather than move people just waterproof their houses.  Any other comments about why people picked the estimates that they picked? 

Okay, well let's do a revote then.  Let's to the same thing. 

[Teacher reading: [On board.]  

Let's do the analysis again.  The answer is 2070 again.  And it's the same as it was.  Alright. 

So it's an interesting discussion.  I think the numbers changed a bit, I think. 

How many people changed their minds?  A few people, but the answer is still 2070.  Well, with that data recorded, give it to the person who asked the question. 

Who's question was this? 

Okay, can I erase? 

We have three minutes.  Should we go for another one?  Who wants to go home?  Alright, we'll go for one more. 

Male Student:  When will land lines not be in homes anymore? 

Professor:  Is that reasonable?  When will there be no more landlines? 

Okay, I'll give you 40 seconds to think of an answer. 

Professor:  Do you all have an answer? 


Did everyone go?  Okay. 

Okay, so 2020

Any arguments for why this is too low or too high? 
Female Student:  I say the army is always going to have them and they're always there for emergencies. 

Professor:   Okay.  Any other points?  Comments or arguments? 

Male Student:  Homes you hardly see them.  Most families just have cell phones, and companies are just switching to that too. 

Let's hit it again. 


So let's see what we have.  So we went up by a year based on this.  Interesting. 

Okay.  Well, we're done for today.  We'll finish next time 

No comments:

Post a Comment