choice-metrics.com

Hello Prof ! We got data from 100 people pilot study, so next step was to find out the parameters to use them in the equation for finding the correct sample size required for main study.
For reaching there, I used the mnl model equation on this data as :
mlogit choice i.AttribA i.AttribB i.AttribC i.AttribD i.AttribE i.AttribE
1) From here I got coeff values for each of these attributes (with one level as base whose coeffs are absent), are they sufficient as parameters for future sample size calculation? NOTE: none of the coeff values are coming as significant here, is that ok or normal?
[img][https://drive.google.com/file/d/1me7hQn ... haring/img]
2) Do i need to use <,cluster(respondent_id)> as option, after the above command.. given each respondent was shown multiple(8) choice tasks. If respondents answered multiple choice sets, should we use the cluster or panel options? I have seen some resources declare it to be panel dataset using cmset (for DCEs where one person is shown multiple choice scenarios at one point of time ) , can we regard such cases as panel dataset, given the DCE is conducted only once and not overtime?

1. In a pilot study you will often find that coefficients are not statistically significant because the sample size may be too low. 100 respondents is a reasonable size for a pilot study though, so often you would see more parameters that are significant. In your case, only 1 parameter is significant. The coefficient for price is not significant in your study, which is somewhat surprising because this is an attribute that a lot of people usually find quite important. Could it be that your price level range is relatively narrow and therefore not many trade-offs are made with respect to price?
Yes, these parameters suffice for sample size calculations, but these calculations will likely not be very reliable because your parameters have large standard errors.

2. I do not know the software you are using to estimate, I only know Apollo and Biogeme, so I cannot say which option to use. SP data is always panel data if a single respondent provides multiple observations. The panel nature indicates that observations come from the same respondent and therefore can account for correlations. So you would typically use the respondent id to group the data. This is mostly important when you are estimating a mixed logit model, but not when you are estimating a multinomial logit model (it should give you the same parameter estimates).

Michiel

In this case, as we had less time, went on with our main survey without specifying the priors (as last time in pilot they did not come as significant coeffs)
The data received of the main survey was in two parts, wide form Choice data(2 blocks, 8 tasks, 2 alt, 7 attributes with 2/3/4 levels) and 2nd was the data with covariates. So I reshaped the Choice data part to long form and merged them both for a joint long-form overall dataset. So where there was 8 choice task per person or RID... now there are 16 rows for each RID . And then tried the command for conditional logit (stata:cmclogit) after setting it as panel data. I did not account for the two diff blocks.

The attached image tries to show the steps:
[img][https://drive.google.com/file/d/1CUE3cr ... haring/img]
[img][https://drive.google.com/file/d/1Hycd62 ... haring/img]

Am I doing something wrong fundamentally in the model or the data analysis or in reshaping which is causing these errors ?
Is the correlation too high between P&S?

Note that this is a model estimation question,not an experimental design question, and I have no familiarity with Stata so it would be best to ask on the Stata forum. The correlations are unlikely to be an issue; even with correlations as high as 0.9 or 0.95 you should typically be able to estimate a choice model.

Perhaps check how you constructed your Choice column? It seems that pref=1 is converted to (0,1) and pref=2 is converted to (1,0), should this not be the other way around?

Michiel

I looked at the construction of the ''choice'' variable as you suggested and could not find the error. After reshaping, I constructed an 'alt' variable in a new column that just lists two unlabelled alternatives one after the other, twice for each RID facing 8 choice tasks each, and then a 'choice' variable such that when alternative is device-1 and pref1 is 1 (where, acc to SEngine: pref1=Which would you choose? 1= opt1 , 2= opt2)
then the new variable 'choice' is designated as 1 or 'yes chosen' : when alternative is device-2 and pref1 is also 2 , similarly when alt is Device 1 and pref1 is also 1 then too 'Choice' = 1 or yes chosen. Am I still making the coding mistake you mentioned? Or can it be improved?
Tried to show it in the image for just RID6 in wide and long form:
[img][https://drive.google.com/file/d/1BbgO6i ... haring/img]

I see. In your previous screenshot Charger1 and Charger2 are swapped around. So Pref=1 means that the option in the second row is chosen. In that case it may be correct as long as Stata understands that the first charging option (Charger2) is actually the second listed option.

Yes thats right.I rechecked.Thank you.
1)If the pilot study doesn’t give significant coefficients or does not converge or very large standard errors, can we still use zero priors for the main study? would that lower the efficiency of the results?
Note: I didn’t set any restrictions between speed and price, so there’s no preference order or priors for main study unlike pilot.

2)When I tried to put near-zero negative prior on Price attribute levels, NGENE could not generate a design after many attempts. Why could that be? We are allowed to set such preference order, arent we?

3)Are my 48 rows with 6 blocks for 500-700 respondents, the problem in the design that wont converge? ( ERROR shown: multiple cases not alternative-specific; there is no within-case variability)

4) 4 of my 7 attributes when regressed over the depen.var : 'choice' actually converges and gives reliable results, but when we include the other 3 attributes with 3/3/4 levels each, then it never converges. But again if we regress these 3 attributes over 'choice' and exclude the other 4, they converge separetly. But together, do not. Can we actually do/report them separately in our paper?

PS: I understand I have posted long queries, I’m just trying to figure out and learn from my mistake... for future designs,as this feels like a failed DCE now.. if my initial zero-prior (with no limiting condition or preference order) design for the main survey was flawed decision, or if I could have improved it. If this might be reason why the logistic regression isn’t converging when I run conditional logit, despite having above 500 respondents
~~~~~
For reference:
This was my prepilot model:
design
;alts = opt1*, opt2*
;eff = (mnl, d)
;alg = swap
;rows = 48
;block = 6
;cond:
if(opt1.x1= 0, opt1.x2<=1),
if(opt1.x1= 2, opt1.x2>=1),
if(opt2.x1= 0, opt2.x2<=1),
if(opt2.x1= 2, opt2.x2>=1)

;model:
U(opt1) = b1.dummy[0|0.000] * x1[1,2,0]
+ b2.dummy[0.000|0.000] * x2[1,2,0]
+b3.dummy[0|0] * x3[1,2,0]
+ b4.dummy[0|0|0] * x4[1,2,3,0]
+ b5.dummy[0.000|0.000] * x5[1,2,0]
+ b6.dummy[0|0] * x6[1,2,0]
+ b7.dummy[0.00] * x7[1,0] /
U(opt2) = b1 * x1
+ b2 * x2
+ b3 * x3
+ b4 * x4
+ b5 * x5
+ b6 * x6
+ b7 * x7
$
~~~~
And this my postpilot design:
design
;alts = opt1*, opt2*
;eff = (mnl, d)
;alg = swap
;rows = 48
;block = 6
;model:
U(opt1) = b1.dummy[0.000|0.000] * x1[1,2,0]
+ b2.dummy[0.000|0.000] * x2[1,2,0]
+ b3.dummy[0.000|0.000] * x3[1,2,0]
+ b4.dummy[0.000|0.000|0.000] * x4[1,2,3,0]
+ b5.dummy[0.000|0.000] * x5[1,2,0]
+ b6.dummy[0.000|0.000] * x6[1,2,0]
+ b7.dummy[0.000] * x7[1,0] /

$

1. Yes you can use zero priors, that is always a safe option. The further away your priors are from the true parameters, the more efficiency you lose. Note that parameter estimates are often not significant in a pilot study because of the small sample size, and the estimate is still the best guess for the prior. But if the priors are very unreliable, with very large standard errors, then one has to be careful not to select too large parameters as that can result in in-efficient designs, so setting zero priors may in that case be preferred.

2. Dominance checks can only be performed if at least two attributes have non-zero priors. Setting only one attribute with a preference order makes it infeasible. If only setting two attributes with non-zero priors then it will still be very difficult to find a design without any dominant alternatives because Ngene will try to find non-dominance based on these two attributes only. The more attributes have non-zero priors, the easier it is to avoid dominant alternatives.

3. I do not know what that error means. I doubt that 48 rows and 6 blocks are an issue. It could be a data coding error or a model specification issue in Stata. In one of the screenshots of your model estimates it strikes me that all your coefficients are very large (dummy coded coefficients would typically not be so large, and neither would your alternative-specific constant because you have an unlabelled experiment). I also see that it states "non-concave" for the loglikelihood function, which is also strange because the LL function for an MNL model with a linear in the parameters utility function is by definition concave.

4. I have no explanation for that, except that perhaps there is multicollinearity in your data. But the Ngene generated design should not have any multicollinearity. I would not report the two models separately since the respondents did see the other attributes as well and cannot be ignored.

Did you ask on the Stata forum to check whether you coded your data correctly and whether you specified the correct model in Stata? I really cannot help here. I am familiar with Apollo and Biogeme packages, which are both free, but I have never used Stata.

Michiel