Beginner: Optimise MNL, evaluate RPL efficient design

[karaferisV]

Dear Moderators,

I am trying to ensure I am going about this right. Here is a very brief description. I have:

Att1 Infection status [0,1,2]
Att2 Seller [0,1,2]
Att3 Price [4000,5500,9,000,14000]

Step 1: For pilot study
Generate efficient design with zero priors and some signs known.

Att1 Inf. status: ( + )infection free, (unknown sign) infection take care, ( - )infected]
Att2 Seller: no idea about priors
Att3 price should be negative but do I need to have effects/dummy coding since I don't have the best guesses for each of the priors?

Code: Select all

?Auction designpilot
Design
;alts = alt1, alt2, alt3
?alt3 neither
; rows = 18
; block = 3
; eff = (mnl, d)

;cond:
if(alt1.status = [0] and alt1.price =[4000] and alt1.seller =[0], alt2.status = [0,1,2] and alt2.price = [4000] and alt2.seller =[0,1,2])

;con
; model:
?seller #, status#, price#
U(alt1) = b2*seller[0,1,2] + b3.effects[0.0001|-0.0001]*status[0,1,2] + b4[-0.0001]*price[4000, 5500, 9000, 14000]/
U(alt2) = b2*seller        + b3*status        + b4*price/
U(alt3) = b5
?
$

Step 2:

Run pilot, estimate for MNL.

Step 3: Actual final study
Use priors and SE’s from pilot to design Bayesian efficient design [n, mean,se].

BUT, if I want to estimate a mixed logit model, then I optimise for mnl and evaluate for panel mixed logit:
Like so?

Code: Select all

?Auction design
Design
;alts(mnl_model) = alt1, alt2, alt3
;alts(rpl_model) = alt1, alt2, alt3
?alt3 neither
; rows = 18
; block = 3
; eff = mnl_model(mnl, d)
;rdraws = gauss(3)
,rep = 500

;cond:
if(alt1.status = [0] and alt1.price =[4000] and alt1.seller =[0], alt2.status = [0,1,2] and alt2.price = [4000] and alt2.seller =[0,1,2])

;con
; model(mnl_model):
?seller #, status#, price#
U(alt1) = b2[prior]*seller[0,1,2] + b3.effects[0.0001|-0.0001]*status[0,1,2] + b4[-0.0001]*price[4000, 5500, 9000, 14000]/
U(alt2) = b2*seller        + b3*status        + b4*price/
U(alt3) = b5


;model(rpl_model):
U(alt1) = b2[n,prior,sd]*seller[0,1,2] + b3.effects[n,prior|prior,sd|sd]*status[0,1,2] + b4[n,prior,sd]*price[4000, 5500, 9000, 14000]/
U(alt2) = b2*seller        + b3*status        + b4*price/
U(alt3) = b5
$

In the rpl_model, have I coded this correctly for effects coding with bayesian priors?

Is this procedure correct?
Any feedback would be greatly appreciated.
Apologies if my questions are stupid, just never did this before.

Thank you,

[Michiel Bliemer]

It looks pretty good but a few suggestions.

If the signs of the attributes are somewhat unclear for some of the attributes and you are not checking for dominance, it is best to set all priors equal to zero, at least for the design in your pilot study, see below. I simplified your conditional constraint, please check that this constraint is needed and correct. Note that in Ngene the last level is the reference level in dummy coding, so level 2 would be zero and you would be estimating coefficients for levels 0 and 1.

Code: Select all

?Auction designpilot
Design
;alts = alt1, alt2, neither
;rows = 18
;block = 3
;eff = (mnl,d)

;cond:
if(alt1.status = 0 and alt1.price = 4000 and alt1.seller = 0, alt2.price = 4000)

;con
; model:
?seller #, status#, price#
U(alt1)    = b2*seller[0,1,2] + b3.effects[0|0]*status[0,1,2]     + b4*price[4000, 5500, 9000, 14000] /
U(alt2)    = b2*seller        + b3                        *status + b4*price                          /
U(neither) = b5
$

Using data from your pilot study, you estimate an MNL model and use the parameter estimates (beta) and the standard errors (se) to inform your priors, i.e. a Bayesian prior is set using for example (note the round brackets around the Bayesian prior):

;bdraws = ...
...
U(alt1) = b2[(n,beta,se)] * ...

In order to evaluate for the RPPANEL model, you would also need priors for this model, so you can try to also estimate an RPPANEL (panel mixed logit) model where you can assume that each parameter is normally distributed as N(beta_mean,beta_stdev], which will require something like:

;rdraws = ...
;rep =
...
U(alt1) = b2[n,beta_mean,beta_stdev] * ...

Your syntax further looks fine I think.

Michiel

[karaferisV]

Thank you very much Professor Bliemer.

[suella_rodrigues]

Dear Prof.,

I was wondering if you could give my code a quick browse.

What I have done is:
1. Estimated an mnl and rpl model using pilot data.
2. Divided all coeff and SD's by 2.

Question 1: Do I need to use effects/dummy coding. For some reason I seemed to have used it in my pilot- cant remember why? It isnt wrong to not use it is it?

Question 2: All my variables are categorical see below:
Att1 Infection status [0,1,2]
Att2 Seller [0,1,2]
Att3 Price [4000,5500,9,000,14000]
so should I then have price categorised as [1,2,3,4] because it affects my coefficients. I dont have issues of collinearity either way.

QUESTION 3: I get an error in my code, it says ''no valid design found, maybe a problem with specification of the design'' No idea how to rectify this.

Code: Select all

?Auction design main survey
Design
;alts(mnl_model) = alt1, alt2, alt3
;alts(rpl_model) = alt1, alt2, alt3
?alt3 neither
; rows = 18
; block = 3
; eff = mnl_model(mnl, d, mean)
;rdraws = gauss(3)
,rep = 500

;cond:
if(alt1.status = [0] and alt1.price =[4000] and alt1.seller =[0], alt2.status = [0,1,2] and alt2.price = [4000] and alt2.seller =[0,1,2])

;con
; model(mnl_model):
?seller #, status#, price#
U(alt1) = b2[(n,-001882,0.09625)]*seller[0,1,2] + b3[(n,0.178, 0.0958)]*status[0,1,2] + b4[(n,-0.159,0.08)]*price[4000, 5500, 9000, 14000]/
U(alt2) = b2*seller        + b3*status        + b4*price/
U(alt3) = b5


;model(rpl_model):
U(alt1) = b2[n,(n,-0.1609859,0.1008),(u,0.52,0.53)]*seller[0,1,2] + b3[n,(n,0.224,0.103),(u,0.55,0.56)]*status[0,1,2] + b4[n,(n,-0.126,0.1232),(u,0.04,0.903)]*price[4000, 5500, 9000, 14000]/
U(alt2) = b2*seller        + b3*status        + b4*price/
U(alt3) = b5
$

Thank you for your time.

[Michiel Bliemer]

You MUST use dummy or effects coding for categorical variables, therefore for seller it should read something like b2.dummy[prior1|prior2] * seller[0,1,2] where level 2 is the base, prior1 is attached to level 0 and prior2 is attached to level 1. The number of parameters you are estimating therefore will go up from 4 to 6.

You can keep price as a continuous variable as you have done now, but a prior of -0.159 does not make sense with levels 4000 to 14000, the prior is far too large and therefore Ngene will give you a warning that no design can be found, it specifically states this in the warning:

No valid design has been found after 1000 evaluations. There may be a problem with the specification of the design. A common problem is that the choice probabilities are too extreme (close to 1 and 0), perhaps because some or all of the prior values are too large.

Therefore, please re-estimate the model with appropriate coding and make sure that the levels and coding you use in estimation are the same as used in the design.

Michiel

[suella_rodrigues]

Thank you Professor.
Question 1a:
Okay so this time I used my dummies and estimated 6 coefficients. Problem is my ridiculous result!
For Status (0,1,2) 0 being a diseased animal I get a very high coef -15.77 with SE of 1073 and insignificant!, I believe this is because of my 10 respondents in the pilot they all avoided the choice with this status.
So, I was wondering for the sake of the design, dividing just this coefficient by 100 in order to get a more conservation prior would do?
Question 1b. Do we have to divide ALL the coeff with the same amount? (I think its yes, any arbitrary number I choose should be the same for all)

Question 2:
Using effects coding for both status and seller, would mean as you said b2.dummy[prior1|prior2] * seller[0,1,2], but this then is not a bayesian design?
unless I have:
b2.effects[0.7|-0.6]*seller[0,1,2] + b3.effects[-0.15|1.3]*status[0,1,2] + b4[(n,-0.00018,0.0007)]*price[4000, 5500, 9000, 14000]/

is that correct? I'm not sure I've understood this correctly. Using bayesian priors to one of the parameters is sufficient? Why not to the others? Despite being categorical variables and having to use effects/dummy coding, we cant know the priors with certainty so surely we should be able to implement bayesian priors/distribution on categorical variables too.

Question3 Using the ridiculous results I get the following syntax which gives me a d-error=0.67

Code: Select all

?Auction design main survey trial3
Design
;alts(mnl_model) = alt1, alt2, alt3
;alts(rpl_model) = alt1, alt2, alt3
?alt3 neither
; rows = 18
; block = 3
; eff = mnl_model(mnl, d, mean)
;rdraws = gauss(3)
,rep = 500

;cond:
if(alt1.status = [0] and alt1.price =[4000] and alt1.seller =[0], alt2.status = [0,1,2] and alt2.price = [4000] and alt2.seller =[0,1,2])

;con
; model(mnl_model):
?seller #, status#, price#
U(alt1) =b2.effects[0.7|-0.6]*seller[0,1,2] + b3.effects[-0.15|1.3]*status[0,1,2] + b4[(n,-0.00018,0.0007)]*price[4000, 5500, 9000, 14000]/
U(alt2) = b2*seller        + b3*status        + b4*price/
U(alt3) = b5


;model(rpl_model):
U(alt1) = b2.effects[0.7|-0.6]*seller[0,1,2] + b3.effects[-0.15|1.3]*status[0,1,2] + b4[n,(n,-0.126,0.1232),(u,0.04,0.903)]]*price[4000, 5500, 9000, 14000]/
U(alt2) = b2*seller        + b3*status        + b4*price/
U(alt3) = b5
$

Apologies, if I am talking rubbish, just want to get to grips with this.

Thank you

[Michiel Bliemer]

1. I am not sure what to recommend. If respondents indeed avoided alternatives with this status, then this may be telling you that the parameter for this level is indeed fairly large and negative. Dividing by 100 does not sound appropriate to me. You could perhaps use a Bayesian prior such as b.effects[(u,-2,-1)|...] which indicates that you think that the parameter is somewhere between -2 and -1. A partworth utility of 2 is reasonably large, although effects coding makes it trickier to determine these partworths and a prior of -2 would mean that the difference in utility between the base level (2) and level 0 could be 4.

2. With "prior1" and "prior2" I meant either a fixed or Bayesian prior. Using Bayesian priors would for example be b.effects[(u,-2,-1)|(n,0.5,0.3)].

3. There was no question here? Note that you have ",rep = 500" in your syntax instead of ";rep = 500". Ngene did not detect this error but best to fix.

Michiel

[suella_rodrigues]

''with "prior1" and "prior2" I meant either a fixed or Bayesian prior. Using Bayesian priors would for example be b.effects[(u,-2,-1)|(n,0.5,0.3)]''

Using -2 and -1 gives me no valid designs because the ''priors are too high''.

Any other recommendations to sort out this issue?

Code: Select all

?Auction design main survey trial3
Design
;alts(mnl_model) = alt1, alt2, alt3
;alts(rpl_model) = alt1, alt2, alt3
?alt3 neither
; rows = 18
; block = 3
; eff = mnl_model(mnl, d, mean)
;rdraws = gauss(3)
;rep = 500

;cond:
if(alt1.status = [1] and alt1.price =[4000] and alt1.seller =[0], alt2.status = [0,1,2] and alt2.price = [4000] and alt2.seller =[0,1,2])

;con
;model(mnl_model):
?seller #, status#, price#
U(alt1) =b2.effects[0.7|-0.6]*seller[0,1,2] + b3.effects[(u,-2,-1)|1.3]*status[0,1,2] + b4[(n,-0.000093,0.0035)]*price[4000, 5500, 9000, 14000]/
U(alt2) = b2*seller        + b3*status        + b4*price/
U(alt3) = b5


;model(rpl_model):
U(alt1) = b2.effects[0.7|-0.6]*seller[0,1,2] + b3.effects[(u,-2,-1)|0.1]*status[0,1,2] + b4[n,(n,-0.126,0.1232),(u,0.047,0.903)]]*price[4000, 5500, 9000, 14000]/
U(alt2) = b2*seller        + b3*status        + b4*price/
U(alt3) = b5
$

Since some of my coefficients are statistically insignificant, is it okay if I put it as 0 or approximately 0 since being insignificant means they don't seem to affect the dependent variable. I have a sample size of 10 which I know isn't sufficient for any accurate inferences but at least they tell me the sign.

Code: Select all

;model(rpl_model):
U(alt1) = b2.effects[0.001|-0.001]*seller[0,1,2] + b3.effects[(u,-2,-1)|0.01]*status[0,1,2] + b4[n,(n,-0.126,0.1232),(u,0.047,0.903)]]*price[4000, 5500, 9000, 14000]/
U(alt2) = b2*seller        + b3*status        + b4*price/
U(alt3) = b5
/code]


Thank you for your time!

[Michiel Bliemer]

The prior (u,-2,-1) is fine, this is not causing invalid designs. Your conditional constraint is causing the issue, if you remove it then you can generate the design. This constraint looks somewhat strange (I wonder what it actually does since level combination 1,4000,0 does not happen often so this constraint should not do much), you need to reformulate it or remove it.

Note that your priors for the price parameter in the RPL model are still too large, multiplying 0.126 with 14000 results in a huge partworth, Please use appropriate priors.

Further, your prior for b5 is now defaulting to 0, which affects all choice probabilities. Please use an appropriate prior, e.g. estimated from your pilot study.

Michiel

[suella_rodrigues]

Okay, I have scrapped it all and restarted.

The only thing I changed is the distribution for price to uniform as I figured that was what was causing the issue. It works now with a d-error = 0.138 with 1 invalid design.

Code: Select all

?Auction design trial5
Design
;alts = alt1, alt2, neither
;rows = 18
;block = 3
;eff = (mnl,d)
;bdraws = gauss(3)

;cond:
if(alt1.status = 1 and alt1.price = 4000 and alt1.seller = 0, alt2.price = 4000)

;con
; model:
?seller #, status#, price#
U(alt1)    = b2.effects[(n,0.7,0.38)|(n,-0.6,0.43)]*seller[0,1,2] + b3.effects[(u,-0.2,-0.1)|(n,1.3,0.349)]*status[0,1,2]     + b4[(u,-0.000093,0.0035)]*price[4000, 5500, 9000, 14000] /
U(alt2)    = b2*seller        + b3                        *status + b4*price                          /
U(neither) = b5
$

But my question is why changing it to uniform instead of normal distribution worked?

Thank you Professor.