Attribute-level balance within blocks

[ashani.abayasekara]

Hi,

I'm designing a labelled discrete choice experiment with six attributes, with levels ranging from 2-4. The design currently has a total of 72 rows, where each respondent will be given 8 choice sets each from 9 blocks. While 12 choice sets per individual would be the best for full attribute-level balance, we feel this might be too much of a cognitive burden.

To maintain attribute-level balance as much as possible, I understand that I can specify the number of times (or a range of times) each level of an attribute should appear for the total number of rows, by indicating these numbers in brackets alongside the relevant attribute and levels in the utility model. My question is whether the same can be done to ensure that each block also maintains level balance to the maximum extent possible? Specifying this for the total number of rows does not guarantee that it holds within each block.

Relatedly, is maintaining attribute-level balance within each block necessary, or is it sufficient if this is only ensured for all choice sets as a whole?

Thank you very much.
Ashani

[Michiel Bliemer]

First of all, neither attribute level balance within a design, not attribute level balance within a block, is a requirement. They are merely nice properties to have, but in many cases it is not possible to achieve a high degree of balance within a block.

I usually aim for a high degree of attribute level balance within a design (although not always perfectly balanced), but I usually settle for a lower degree of level balance within a block. Only with orthogonal designs you can get perfect attribute level balance within a block. With random or effect designs, the ;block properties optimises for attribute level balance within a block, but it will never be perfect, and it may be far from perfect. You cannot specify any additional minimum-maximum appearance of levels within this optimisation process.

If you are not satisfied with the way that Ngene has blocked the design, you can of course always manually try to assign the choice tasks to blocks based on your own definition of measuring attribute level balance within a block (there are many different ways of measuring this). Blocking does not affect the efficiency of the design.

Michiel

[ashani.abayasekara]

Hi Michiel,

Thank you very much for your explanation. It's reassuring to know that attribute level balance is not a strict requirement. I may adjust some of the choice tasks manually within blocks to achieve better balance.

Best,
Ashani

[ashani.abayasekara]

I have another follow-up question on attribute-level balance. I'm trying to understand what the option "imbalance" in the Ngene syntax aims to do. Is that an option to minimise any potential imbalance in a design? For example, the syntax "eff = (mnl,d) + 0.5*(imbalance)".

[Michiel Bliemer]

With the modified Federov algorithm, which does not consider attribute level balance, there are three ways to get reasonable attribute level balance:
(1) Dummy code all attributes
(2) Impose attribute level frequency constraints, like X[1,2,3](4-6,4-6,4-6)
(3) Add (imbalance) to the objective function for optimising the efficiency. This means that the minimisation problem becomes: Min. (D-error) + 0.5*(imbalance), where imbalance is a value between 0 and 1 where 0 is highly imbalanced and 0 is perfect attribute level balance within the design. By increasing weight 0.5 you can increase the attribute level balance of the design

While (2) imposes hard constraints, (1) and (3) are "soft" ways to improve attribute level balance.

Michiel

[ashani.abayasekara]

Thanks very much for the explanation. Are these methods only relevant when using the modified Federov algorithm? I'm using the column-based swapping algorithm and was wondering if adding the "imbalance" property to the objective function could improve attribute-level balance within blocks.

[Michiel Bliemer]

All these methods are indeed only relevant for the modified Federov algorithm, the swapping algorithm by default aims for attribute level balance of the design.

The property that aims for attribute level balance within each block is the ;block property. It has several options that you could try, including increasing the runtime to find good blocks, or changing minsum to minmax. Below is a description of the ;block property. Minimising the correlation between the blocking column is similar to maximising attribute level balance with blocks. In orthogonal designs, where perfect attribute level balance within blocks is possible, zero correlations with the blocking column means perfect attribute level balance.

BLOCK

Property that defines the number of blocks in the design.

Format

block = number, <option>, ...

Explanation

number (integer) divides the total number of choice tasks in a design, such that each decision maker can be given a subset of choice tasks. For example, when using rows=12 and block=3, then the design is blocked into three equal parts of four choice tasks each. Ngene aims to maximise attribute level balance within each block, although perfect attribute level balance within a block is only achievable with an orthogonal design. When adding this property, an additional column is added to the design that describes the block number that each choice task belongs to.

<option> can be

minsum - to determine blocks such that the sum of correlations between the blocking column and the attributes are minimised,
minmax - to determine blocks such that the maximum correlation between the blocking column and the attributes is minimised,
total(number mins) - to indicate the number (integer) of minutes to search for the best blocking column,
noimprov(number secs) - to indicate the number (integer) of seconds to keep searching after an improvement was found,
noimprov(number mins) - to indicate the number (integer) of minutes to keep searching after an improvement was found,
newblocking - to generate a new blocking column when evaluating an existing design using eval.

Comments

If no options are specified, the default options are set to: minsum, total(3 secs)

Examples

block = 2
block = 4, minmax, noimprov(10 secs)
block = 3, newblocking
block = 3, minsum, total(1 mins), newblocking

[ashani.abayasekara]

Thank you so much for the clear explanation, I appreciate it very much. I tried the different options suggested with the ;block property but am still unable to achieve balance within blocks.

The design is a labelled design with three alternatives and an opt-out. There are a total of 6 attributes (some are relevant only for certain alternatives), with levels ranging from 2-4. We would ideally like to show each respondent 8 choice sets (12 is a better option to match the levels but may be too many to evaluate), and currently looking at a total of 48 choice sets in 6 blocks. While we would prefer as much balance of attribute levels within choice sets in a block across the attributes (i.e. for levels of 2-level attributes to appear 4 times each in a block and 4-level attributes to appear twice), at the moment some levels do not appear at all in some attributes in a given block and some are heavily imbalanced.

Do you have any suggestions on anything else I can try to improve level balance? I'm copying my code below for your reference. Thank you so much for your time.

Code: Select all

? Attributes and levels:
 ? base = where psychologist is based: 0(rural, base), 1(city_visit)
 ? base1 = where psychologist is based: 0(rural, base), 1(city_visit), 2(city)
 ? wait = wait time till 1st visit: 0(2_4wks, base), 1(1_3mnths), 2(4_6mnths), 3(6_12mnths)
 ? wait1 = wait time till 1st visit: 0(<2wks, base), 1(2_4wks), 2(1_3mnths), 3(4_6mnths)                      
 ? cost = gap fee: $0(base), $50, $90, $120
 ? time = travel time for in-person visits: 0(1h, base), 1(1_2hrs), 2(2_3hrs), 3(3_4hrs)
 ? hub = availability of TH hub: 0(available, base), 1(unavailable)
 ? first = mode of 1st visit: 0(in-person, base), 1(telehealth)
 
? Alternative-specific attributes and attribute levels:
 ? base for IP and base1 for MIX and TH (city base only possible for fully TH)
 ? wait for MIX and TH and wait1 for IP (longer wait times for fully IP)
 ? time only relevant for IP and MIX (TH visits don't require travel) 
 ? hub only relevant for MIX and TH
 ? first only relevant for MIX

? Common attributes:
 ? cost

? Main effects and all 2 way interactions, D-efficient, unknown priors
? Cost as continuous with quadratic 
? Can't check for dominated alternatives because labelled 
? TH hub is a scenario attribute - fixed levels across alternatives in a choice set
?????????????????????????????????????????????????????????????????????????????????????????????????????????
? Main effects plus interactions

design 
;alts = IP, MIX, TH, N  
;rows = 48                   
;block = 6, minmax, total(30 secs) 
;eff = (mnl, d)
;alg = swap(stop=noimprov(30secs))

;model: 
  U(IP) = b1[0.0002] + b2.dummy[0]*base[1,0]
            + b3.dummy[-0.0001|-0.0002|-0.0003]*time[1,2,3,0]
            + b4.dummy[-0.0001|-0.0002|-0.0003]*wait[1,2,3,0]
            + b5[-0.0001]*cost[50,90,120,0] + b6[0.000001]*cost*cost
            + b7*base.dummy[1]*time.dummy[1] 
            + b8*base.dummy[1]*time.dummy[2] 
            + b9*base.dummy[1]*time.dummy[3] 
            + b10*base.dummy[1]*wait.dummy[1] 
            + b11*base.dummy[1]*wait.dummy[2] 
            + b12*base.dummy[1]*wait.dummy[3]
            + b13*base.dummy[1]*cost
            + b14*time.dummy[1]*wait.dummy[1]
            + b15*time.dummy[1]*wait.dummy[2]
            + b16*time.dummy[1]*wait.dummy[3]
            + b17*time.dummy[2]*wait.dummy[1]
            + b18*time.dummy[2]*wait.dummy[2]
            + b19*time.dummy[2]*wait.dummy[3]
            + b20*time.dummy[3]*wait.dummy[1]
            + b21*time.dummy[3]*wait.dummy[2]
            + b22*time.dummy[3]*wait.dummy[3] 
            + b23*time.dummy[1]*cost
            + b24*time.dummy[2]*cost
            + b25*time.dummy[3]*cost
            + b26*wait.dummy[1]*cost 
            + b27*wait.dummy[2]*cost 
            + b28*wait.dummy[3]*cost / 

  U(MIX) = b29[0.0001] + b2.dummy*base
            + b30.dummy[0]*hub[1,0]
            + b3.dummy*time
            + b31.dummy[-0.0001|-0.0002|-0.0003]*wait1[1,2,3,0]
            + b5*cost + b6*cost*cost
            + b32[0.0001]*first[1,0] 
            + b33*base.dummy[1]*hub.dummy[1]
            + b7*base.dummy[1]*time.dummy[1] 
            + b8*base.dummy[1]*time.dummy[2] 
            + b9*base.dummy[1]*time.dummy[3]
            + b34*base.dummy[1]*wait1.dummy[1] 
            + b35*base.dummy[1]*wait1.dummy[2] 
            + b36*base.dummy[1]*wait1.dummy[3]
            + b13*base.dummy[1]*cost
            + b37*base.dummy[1]*first.dummy[1]
            + b38*hub.dummy[1]*time.dummy[1] 
            + b39*hub.dummy[1]*time.dummy[2] 
            + b40*hub.dummy[1]*time.dummy[3] 
            + b41*hub.dummy[1]*wait1.dummy[1] 
            + b42*hub.dummy[1]*wait1.dummy[2] 
            + b43*hub.dummy[1]*wait1.dummy[3] 
            + b44*hub.dummy[1]*cost
            + b45*hub.dummy[1]*first.dummy[1] 
            + b46*time.dummy[1]*wait1.dummy[1]
            + b47*time.dummy[1]*wait1.dummy[2]
            + b48*time.dummy[1]*wait1.dummy[3]
            + b49*time.dummy[2]*wait1.dummy[1]
            + b50*time.dummy[2]*wait1.dummy[2]
            + b51*time.dummy[2]*wait1.dummy[3]
            + b52*time.dummy[3]*wait1.dummy[1]
            + b53*time.dummy[3]*wait1.dummy[2]
            + b54*time.dummy[3]*wait1.dummy[3] 
            + b23*time.dummy[1]*cost
            + b24*time.dummy[2]*cost
            + b25*time.dummy[3]*cost
            + b55*time.dummy[1]*first.dummy[1] 
            + b56*time.dummy[2]*first.dummy[1] 
            + b57*time.dummy[3]*first.dummy[1] 
            + b58*wait1.dummy[1]*cost 
            + b59*wait1.dummy[2]*cost 
            + b60*wait1.dummy[3]*cost 
            + b61*wait1.dummy[1]*first.dummy[1] 
            + b62*wait1.dummy[2]*first.dummy[1] 
            + b63*wait1.dummy[3]*first.dummy[1] 
            + b64*cost*first.dummy[1] / 

  U(TH) = b65[-0.0001] + b66.dummy[0|0]*base1[1,2,0]
           + b30.dummy*hub[hub]
           + b31.dummy*wait1
           + b5*cost + b6*cost*cost
           + b67*base1.dummy[1]*hub[hub]
           + b68*base1.dummy[2]*hub[hub] 
           + b69*base1.dummy[1]*wait1.dummy[1] 
           + b70*base1.dummy[1]*wait1.dummy[2] 
           + b71*base1.dummy[1]*wait1.dummy[3]
           + b72*base1.dummy[2]*wait1.dummy[1] 
           + b73*base1.dummy[2]*wait1.dummy[2] 
           + b74*base1.dummy[2]*wait1.dummy[3]
           + b75*base1.dummy[1]*cost
           + b76*base1.dummy[2]*cost
           + b41*hub[hub]*wait1.dummy[1] 
           + b42*hub[hub]*wait1.dummy[2] 
           + b43*hub[hub]*wait1.dummy[3]     
           + b44*hub[hub]*cost
           + b58*wait1.dummy[1]*cost 
           + b59*wait1.dummy[2]*cost 
           + b60*wait1.dummy[3]*cost          
$

[Michiel Bliemer]

As said, it impossible to achieve a high degree of balance within blocks with efficient designs for all attributes, this is only possible with orthogonal designs.

I ran the script and looked at the attribute level balance within each block. It looks pretty good to me for an efficient design, most attributes have reasonable balance within each block, although there will always be some attributes with lower balance within some blocks. This is often unavoidable. You can manually swap choice tasks across blocks if you believe this would improve balance within the blocks, but usually whenever you improve attribute level balance for one attribute, you make it worse for another.

Michiel

[ashani.abayasekara]

Thank you very much for going through the script and for the explanation.