Number of rows and interpretation of efficiency measures

[twickr07]

Hi

I am doing an unlabelled d-efficient design, with 6 attributes (3 continuous, 3 categorical) see code below.

Code: Select all

Design
;alts = alt1*, alt2*
;rows = 12
;eff = (mnl,d)
; model:
u(alt1)=  b1[0.00001]*effect[30,50,70]+b2[-0.0001]*speed[6,10,14]+ b3[-0.0001]*flare [10,40,60]+b4.dummy[0|0]*route [0,1,2]+b5.dummy[0|0]*moderate[0,1,2]+b6.dummy[0]*severe[0,1] /
u(alt2) = b1*effect+b2* speed+ b3*flare +b4.dummy*route +b5.dummy*moderate+b6.dummy*severe
$

1) Is there a way to check if the number of rows I'm using is appropriate?

https://drive.google.com/file/d/1ZHt6VcvfuuRO4VzByGXo3RSeG75ZN4Lq/view?usp=sharing

2) I've attached my MNL efficiency measures output, i'm struggling to interpret if the D-error, A-error and S-estimate is within the acceptable range? is there a rule of thumb for these efficiency measures?

Thanks a lot in advance!

Tara

[Michiel Bliemer]

1. The minimum number of rows should be at least #parameters/(#alternatives-1), which in your case is 8/(2-1) = 8. Therefore, 12 is enough in order to estimate the model. Ngene automatically checks for this, if you try ;rows = 7, Ngene will tell you that you need a minimum of 8. Note that while 12 is theoretically enough, it is often not a bad idea to have a bit more variation in your data. So you could decide to increase the number of rows and block the design, e.g. ;rows = 24 and ;block = 2, such that there are two versions of the survey with 12 choice tasks each.

2. The D-error and A-error have no meaning except that we want to minimise these values. You cannot tell by the value whether it is good or bad. If the D-error is Undefined/Infinite, then the model parameters are not identifiable, but if Ngene generates a finite D-error the model can be estimated. The S-estimate has a meaning and refers to the minimum sample size needed to estimate all parameters statistically significant at the 95% confidence level. However, the S-estimate is only meaningful if you have appropriate priors, e.g. that come from a pilot study. In your case, you should ignore the S-estimate. Your D-errors and A-errors look perfectly fine.

Michiel

[Joy_Lawrence]

How is the number of parameters 8 here and not 11 or 6 (i.e. b1....b6)? Could you please explain how parameters are calculated?

[Michiel Bliemer]

b4 and b5 each have two parameters (and therefore also two priors as you indicated with 0|0). If you have L levels, then you need L-1 parameters for the dummy variables.

[Joy_Lawrence]

Is there any rule about number of blocks? And choice tasks? For example if we have 24 rows it could be either 8 tasks each across 3 blocks or 12 across 2 blocks.

[Michiel Bliemer]

The number of rows and blocks are determined such that (i) the number of rows is sufficient to estimate the given parameters, and (ii) the number of choice tasks given to a respondent does not cause fatigue.

With 24 rows you can choose 3 blocks of 8 or 2 blocks of 12. The latter would of course give you more information per respondent and result in more choice observations with the same sample size, but would also increase the length of the survey and may be too many for respondents, especially when the choice tasks are complex. It also depends on whether you conduct the survey online or face-to-face, where fatigue/boredom kicks in more quickly in an online environment.

In your case, with 2 alternatives and 6 attributes, this is moderately complex. I think 12 choice task would be okay. You can always choose to give respondents for 6 choice tasks, then some other questions, and then another 6 choice tasks. You can do pre-testing to determine where the limit for fatigue is.

For model estimation, the number of blocks is not relevant.

Michiel