Data collection and selection

We have composed a dataset that includes 100 studies and 110 datasets which report 1,621 unique estimates. These data were collected through a systematic search as prescribed by the PRISMA 2020 statement (Page et al., Reference Page, McKenzie, Bossuyt, Boutron, Hoffmann, Mulrow, Shamseer, Tetzlaff, Akl, Brennan, Chou, Glanville, Grimshaw, Hróbjartsson, Lalu, Li, Loder, Mayo‐Wilson, McDonald and Moher2021).Footnote ⁵ Our data stem from three sources: (1) database searches, (2) citation searches and (3) review articles. These searches yielded 2,468 potentially relevant records. After deleting duplicate files (N = 70), 2,398 records were screened for inclusion in the meta-analysis based on seven predefined inclusion and exclusion criteria (see online Appendix C for a more elaborate explanation of our inclusion criteria).Footnote ⁶ The 1,879 records that, based on their abstracts, did not adhere to the inclusion criteria were excluded. Two more records were excluded that could not be retrieved, while another 417 records were excluded after reading the full study. This process resulted in a final dataset of 100 studies adhering to all seven criteria. The PRISMA flowchart in Figure 1 summarises the full data collection and selection procedure. An overview of the included articles can be found in online Appendix B, and a detailed account of our data collection and selection process can be found in online Appendix C.Footnote ⁷

Figure 1. Data collection and selection process. Note. Figure adapted from PRISMA flow diagram.

Coding process

The 100 studies in the final dataset were consequently coded for their independent and dependent variables, the features of the democratic innovation (see Elstub & Escobar, Reference Elstub and Escobar2019; Karpowitz & Mendelberg, Reference Karpowitz, Mendelberg, Lupia, Greene, Kuklinski and Druckman2011), whether they applied a pre-test, whether they included a control group, their statistical technique, their observed effects, and several other descriptive codes.Footnote ⁸ In order to ensure internal consistency throughout the coding process, the researchers defined mutually exclusive and precise definitions based on the literature for each code. In case a question in coding arose, the researchers jointly refined the coding scheme.Footnote ⁹ After the first coding round, all codes were double-checked by one of the researchers. As an ultimate robustness check, two independent researchers double-coded 10% of the articles. The Fleiss Kappa statistic amounted to 0.779 [0.719; 0.838] in the first coding round, and increased to 0.843 [0.818; 0.868] in the final coding round, which indicates a very good final strength of agreement (Laerd Statistics, 2019).Footnote ¹⁰

Dependent variables – Citizens’ attitudes, behaviour and capabilities

The dependent variables used in this study are changes in attitudes, behaviour, and capabilities amongst citizens. We build on prevailing theoretical work on deliberative democracy, participatory democracy and direct democracy that presents theoretical expectations about the effects of democratic innovations on participating citizens, and follow Jacquet and van der Does’ (Reference Jacquet and van der Does2020) classification of those effects into the categories of citizens’ attitudes, behaviour and capabilities. We then further subdivide these categories into sub-types of attitudes (policy, political and process attitudes), behaviour (political behaviour and civic engagement) and capabilities (knowledge, internal efficacy and reasoning).

Attitude changes are measured by attitude questions on a rating scale. The attitude objects include the policy (i.e., policy attitudes), a political entity (i.e., political attitudes), or the decision-making process (i.e., process attitudes). Changes in policy attitudes are most frequently measured as net group shifts, but have over time also increasingly been measured as individual-level opinion change. Political attitude change is measured by looking at shifts in evaluations of and affective orientations towards Dalton's five classical levels of political support (Dalton, Reference Dalton2004). Process attitudes are often captured by measures of perceived procedural fairness, outcome acceptance or perceived legitimacy of the process.

Behavioural changes are either measured as changes in intended behaviour, or as changes in actual behaviour. The first subtype of behaviour, political participation, is measured with variables such as voting or willingness to demonstrate. The second subtype of behaviour, civic engagement, covers variables such as discussing public affairs or involvement in civic organisations.

Capability changes are measured either objectively by asking factual knowledge questions, or subjectively by probing for self-perceptions. The first subtype of capabilities, knowledge, is most often measured by asking factual questions about the policy issue at hand or about politics in general. Internal efficacy is captured by tapping into participants’ sense of internal political efficacy and confidence in their own competences and knowledge to participate in politics. The last subtype of reasoning skills is measured by assessing participants’ attitude coherence or their level of justification.

Independent variables – Types of democratic innovations

The key independent variable of interest employed in this study is participation in a democratic innovation. Democratic innovations have to be conceived of as ‘grand treatments’ that comprise numerous micro-treatments and together cause the observed effect (Luskin et al., Reference Luskin, Fishkin and Jowell2002). This ‘grand treatment’ is measured in two ways. Either a group that receives the treatment is compared to a group that does not receive the treatment, that is, comparing people that participated in a democratic innovation to people who did not participate, or the same treated group is measured before and after treatment, that is, people that participated in a democratic innovation were surveyed before and after participating.Footnote ¹¹

These ‘grand treatments’ either take place in the field or are simulated in a lab. A lab setting provides for a more controlled environment of the features of the ‘grand treatment’ that are implemented, while field studies have higher ecological validity. Both settings however allow for the manifestation of the core features of each type of treatment.Footnote ¹²

As mentioned earlier, these ‘grand treatments’ can be categorised into three big families, characterised by a specific set of core features and several common featuresFootnote ¹³ (based on Elstub & Escobar, Reference Elstub and Escobar2019):

1. 1 Mini-publics: mini-publics bring together ‘groups of citizens that engage in facilitated deliberations on an issue, and make public recommendations’ (Harris, Reference Harris, Elstub and Escobar2019, p. 45). Hence, the fact that citizens engage in discussions with one another and come to decisions based on deliberation are the core features of mini-publics. The participants to mini-publics are usually selected via some form of sortition (Elstub & Escobar, Reference Elstub and Escobar2019, p. 26), and the discussions last between several hours to several weeks (Smith, Reference Smith2009). Reason-giving is an important aspect of these discussions, and the recommendations that are made via mini-publics are often advisory (Elstub & Escobar, Reference Elstub and Escobar2019). Often a moderator is present to ensure that deliberative standards of respect and an equal chance to participate are met. Mini-publics entail a high level of information input stemming from information booklets to expert presentations or the questioning of politicians and stakeholders. In this paper, in order for a democratic innovation to be categorised as a mini-public, the modes of participation have to be discursive expression and listening, and the main mode of decision-making has to be deliberation (following Elstub & Escobar, Reference Elstub and Escobar2019).Footnote ¹⁴ Democratic innovations that are classified as mini-publics include citizens’ assemblies, focus groups, citizens’ juries, deliberative forums, and deliberative polls and lab experiments.

2. 2 Participatory processes: participatory processes are rooted in theories of participatory democracy, which advocate the ‘[d]irect participation of citizens in the regulation of key institutions of society, including the workplace and local community’ (Held, Reference Held2009, p. 215). The core features of participatory processes are, therefore, that citizens engage in discussions and vote, as the outcome is decided by aggregating the preferences of the individual participants. Usually, participatory processes are open to everyone and they often have a direct impact on decision-making (Elstub & Escobar, Reference Elstub and Escobar2019; Smith, Reference Smith2009). In this paper, in order for a democratic innovation to be categorised as a participatory process, the modes of participation have to be voting and discursive expression, and the main mode of decision making has to be aggregation (following Elstub & Escobar, Reference Elstub and Escobar2019). Democratic innovations that are classified as participatory processes include participatory budgeting processes, town meetings and lab experiments.

3. 3 Referendums and citizens’ initiatives: the core features of referendums and citizens’ initiatives are that citizens vote on or sign a statute, constitutional amendment, or legal act, and that decisions are taken through some form of aggregation (Elstub & Escobar, Reference Elstub and Escobar2019). The use of referendums and citizens’ initiatives ranges from mandatory to facultative, and their impact from binding to advisory. Some are top-down, others bottom-up, requiring a certain number of signatures to be collected before they may occur. Usually, all citizens can participate in them. Participation, however, generally does not happen face-to-face and the act of voting is rather short as compared to participating in other democratic innovations. Nevertheless, referendums and citizens’ initiatives can go on for weeks and campaigns can provide participants with information regarding the proposed policy, even though this information is often not balanced. In this paper, in order for a democratic innovation to be categorised as referendums and citizens’ initiatives, the mode of participation has to be voting and the mode of decision-making has to be aggregation (following Elstub & Escobar, Reference Elstub and Escobar2019). Democratic innovations that are classified as referendums and citizens’ initiatives include popular referendums, citizens’ initiatives, government-sponsored referendums and lab experiments.

Participation in a mini-public

We start this section by briefly discussing what effects can theoretically be expected. To do so, we consider how the features of mini-publics (see section 3.2) could possibly affect participants. Due to the high level of information input and the quality of discursive exchange between citizens, deliberative democrats expect mini-publics to have a large impact on the participating citizens. First, the exchange of reasons between citizens during deliberation is expected to lead citizens to change their opinion regarding the policy issue they discuss (Fishkin, Reference Fishkin1995, pp. 162−163). Additionally, by promoting mutual respect and understanding, and by aiming to reach decisions that are perceived as justified by participants, deliberative processes are expected to change citizens’ attitudes towards the process and outcome of deliberation (Gutmann & Thompson, Reference Gutmann and Thompson2004; Myers & Mendelberg, Reference Myers and Mendelberg2013). Moreover, by receiving balanced information and taking part in discussions, citizens will increase their political knowledge, reasoning skills and internal efficacy (Burkhalter et al., Reference Burkhalter, Gastil and Kelshaw2002). Overall, there is thus the expectation that participating in mini-publics has a significant effect on participants’ policy attitudes, and a positive and significant effect on their process attitudes, knowledge, internal efficacy and reasoning skills.

Figure 5 provides an overview of our analysis of the effects of participation in mini-publics. Starting with policy attitudes, we observe a highly significant average effect. We therefore find strong evidence that citizens change their opinion about the policy after participating in a mini-public. Moreover, the fail-safe N test demonstrates that there would need to be 415 unpublished insignificant studies to render this effect insignificant. Turning to political attitudes, the average effect at the level of studies is highly significant, which means that participation in a mini-public is positively related to a change in citizens’ political attitudes. Moreover, publication bias does not seem to affect this relationship. For process attitudes the evidence is somewhat mixed: the average effect at the level of studies is significant, but the number of studies is small and this effect is potentially prone to publication bias.

Figure 5. Plot of effects of participation in a mini-public. Notes. t-test with two-tailed significance levels. $N$$N$ is the sum of the number of participants in the studies into the specific effect. $N_t$${{N}_t}$ is the number of tests conducted into the specific effect. $N_s$${{N}_s}$ is the number of studies conducted into the specific effect. $N_{FS}$${{N}_{FS}}$ represents the number of unpublished insignificant studies necessary to render the average effect insignificant. The size of the square represents the $N$$N$ of participants. *p < 0.05, **p < 0.01, ***p < 0.001.

Turning to the effects of participating in a mini-public on behaviour, we see that the average effect is significant, yet prone to publication bias. We thus fail to find robust cross-study evidence that mini-publics cause citizens to become more politically active. With regards to the effect on civic engagement, we do find that the average effect is significant. To render this positive and significant effect insignificant, 18 insignificant studies would need to be unpublished. Hence, we can conclude that we find some evidence that mini-publics on average cause participants to become more civically engaged. This effect would however need to be further substantiated to definitively rule out the possibility of publication bias.

Considering the effects of participating in a mini-public on capabilities sheds light on some of the key hypotheses proposed in deliberative theory: mini-publics are often expected to increase political knowledge, internal efficacy and reasoning skills (Burkhalter et al., Reference Burkhalter, Gastil and Kelshaw2002). Political knowledge stands out as the clearest finding of the meta-analysis: a large and highly significant average effect shows that participation in a mini-public has a significant and positive effect on how much citizens know about the policy topic and politics in general. It is highly unlikely that this effect is produced by publication bias, as there would need to be 708 unpublished insignificant studies to render the overall effect insignificant. The results for internal efficacy and reasoning skills are similar but slightly less outspoken. The average effects are highly significant and these results seem to be unaffected by publication bias. Hence, we also find evidence for the positive effect of participating in a mini-public on citizens’ internal efficacy and reasoning skills, even though the effects are less robust across studies than the effect on citizens’ knowledge.

To conclude, this analysis regarding the effects of participation in a mini-public on participants shows that participation in a mini-public has an effect on citizens, as we find significant effects on participants’ policy attitudes, political attitudes, knowledge, internal efficacy and reasoning skills. There is also some evidence that mini-publics render participants more positive toward the process, but we would need more evidence in order to substantiate this finding. Apart from these, we found some evidence that mini-publics could make participants’ more civically engaged, however a further assessment of that effect would be desirable given the low number of studies and its proneness to publication bias.

Participation in a participatory process

The second type of democratic innovation covers participatory processes. Central to participatory democracy is the contention that individuals ‘learn to participate by participating’ and that these individuals will also ‘participate in decision-making in their everyday lives as well as the wider political system’ (Pateman, Reference Pateman2012, p. 10). Participatory democrats thus conceive of participatory events as schools of democracy, during which citizens gain the knowledge and learn the skills they need to participate more in the future (Pateman, Reference Pateman1970). During active participation, citizens enhance their internal political efficacy and self-confidence and come to appreciate the intrinsic value of participation more (Barber, Reference Barber2003). Hence, participation leads to more participation (Pateman, Reference Pateman1970). Overall, there is thus the expectation that participation in participatory processes has a positive and significant effect on citizens’ political participation, civic engagement, knowledge and internal efficacy. Figure 6 provides an overview of our analysis regarding the effects of participation in participatory processes.

Figure 6. Plot of effects of participation in a participatory process. Notes. t-test with two-tailed significance levels. $N$$N$ is the sum of the number of participants in the studies into the effect. $N_t$${{N}_t}$ is the number of tests conducted into the effect. $N_s$${{N}_s}$ is the number of studies conducted into the effect. $N_{FS}$${{N}_{FS}}$ represents the number of unpublished insignificant studies necessary to render the average effect insignificant. The size of the square represents the $N$$N$ of participants. ^*p < 0.05, ^**p < 0.01, ^***p < 0.001.

Figure 6 shows that only a few studies look at the effect of participating in participatory processes on citizens’ attitudes. For policy attitudes, we find an insignificant, slightly positive effect, which is based on only two studies. Three studies assessed effects on political attitudes resulting in a positive but insignificant average effect. Only one study looked at the effects on participants’ process attitudes and found no significant effect. Hence, even though there are some preliminary indications of a positive effect on participants’ political attitudes, there exists a significant gap in our knowledge with regards to the effect of participating in a participatory process on citizens’ attitudes.

We now turn to the effects of participatory processes on citizens’ behaviour, which is a key expected outcome of participating (Pateman, Reference Pateman2012). Of the two studies that assess this effect, none found a significant effect of participatory processes on citizens’ political participation. For civic engagement, the average affect size is even negative, meaning that participation lowers civic engagement. However, this statistic is insignificant, and the confidence interval is very large due to a low number of available studies.

Finally, regarding the effects of participatory processes on capabilities, we find insignificant effects on both knowledge and internal efficacy. For knowledge, a few studies contained positive and significant effects, but the overall effect is insignificant. For internal efficacy, the results are even clearer with all four studies finding no significant effects. Hence, these results could be a preliminary indication, based on four studies and 12 tests, that participation in a participatory process has a limited effect on citizens’ internal efficacy. There are no studies looking at the effects on participants’ reasoning skills.

Overall, therefore, due to the small number of studies, we find no robust effects (yet) of participatory processes on citizens’ attitudes, behaviour or capabilities. There are preliminary indications that participatory processes could have a positive effect on citizens’ political attitudes and knowledge, and no effect on their internal efficacy. However, no conclusions can yet be drawn based on the current analysis, which may change if more studies on participatory processes with pre- and post-tests or a control group are conducted in the future (see section 4.1 and He, Reference He2019).

Participation in referendums and citizens’ initiatives

The third and last type of democratic innovation comprise the main instruments of direct democracy: referendums and citizens’ initiatives. Referendums and citizens’ initiatives are rooted in the theory of direct democracy. Most importantly, it is argued that such direct processes can change citizens’ political attitudes, such as feelings of external political efficacy (Smith & Tolbert, Reference Smith and Tolbert2004). Indeed, they are linked to an important idea in democratic thought, namely ‘popular sovereignty as a way of addressing the demands of citizens and the dependence of public policies on their preferences’ (Altman, Reference Altman2010, p. 1). Barber (Reference Barber2003) has argued that direct democracy leads citizens to gain a more inclusive identity and feel more part of a political community. However, more recent scholarship has suggested that the aggregative element of referendums and citizens’ initiatives might lead to the ‘winners’ experiencing an increase in positive affect towards political objects, whereas ‘losers’ would decrease their support, especially if the policy issue is highly salient (Skitka, Reference Skitka2002). Procedural fairness theory challenges that view and posits that a decision-making process in which citizens have the final say should enhance political and process attitudes, even if the outcome of such a process is unfavourable. Additionally, the perception that citizens have a direct influence on the outcome could strengthen feelings of political efficacy, regardless of the outcome (Tyler, Reference Tyler2006). In terms of behaviour, direct initiatives are expected to have a spill-over effect on election turnout (Smith & Tolbert, Reference Smith and Tolbert2004). Direct participation, moreover, is expected to boost future citizen participation in policy making (Barber, Reference Barber2003). Furthermore, exposure to the information given during campaigns can increase citizens’ political knowledge (Smith, Reference Smith2009, pp. 130−133). Referendums and citizens' initiatives should stimulate political interest and facilitate learning about politics (Smith & Tolbert, Reference Smith and Tolbert2004). The process would educate citizens on a permanent basis (Barber, Reference Barber2003). Overall, there is thus the expectation that participation in referendums and citizens’ initiatives has a positive and significant effect on citizens’ political attitudes, political participation, knowledge and internal efficacy.

Figure 7 shows the results of the meta-analyses regarding referendums and citizens’ initiatives. As we can see, effects on participants’ process attitudes, civic engagement and reasoning skills have not been assessed in the empirical literature included in our analysis. This lack of studies is in line with theoretical expectations regarding the effects of referendums and citizens’ initiatives. However, also, the effects for which we would theoretically expect an effect on citizens are currently not widely studied. Policy attitudes have only been found to have a positive and significant effect in one study, which means we cannot draw any meta-analytic conclusions. For political attitudes, we find an insignificant slightly negative average effect. This means that there is no consistent significant effect of participation in referendums and citizens’ initiatives on political attitudes found in the three studies included in our meta-analysis. In terms of the effects on process attitudes, only one study considered these and found no significant effect.

Figure 7. Plot of effects of participation in referendums and citizens’ initiatives. Notes. t-test with two-tailed significance levels. $N$$N$ is the sum of the number of participants in the studies into the effect. $N_t$${{N}_t}$ is the number of tests conducted into the effect. $N_s$${{N}_s}$ is the number of studies conducted into the effect. $N_{FS}$${{N}_{FS}}$ represents the number of unpublished insignificant studies necessary to render the average effect insignificant. The size of the square represents the $N$$N$ of participants. ^*p < 0.05, ^**p < 0.01, ^***p < 0.001.

Turning to political behaviour, again only one study assessed the effect on political participation and found positive effects. For knowledge, a small positive effect was found based on three studies. Finally, with regards to the effect on internal efficacy, the average effect based on two studies indicates a positive and rather large effect. However, again, these effects proved insignificant due to the small number of studies.

To sum up, the effects of participation in referendums and citizens’ initiatives on citizens cannot be sufficiently tested due to gaps in the empirical literature. There are however preliminary indications that referendums and citizens’ initiatives could have a positive effect on citizens’ knowledge and internal efficacy, and the jury is still out on the effect on political participation. Nevertheless, more studies are needed to test these effects more thoroughly. If more studies would conduct their analyses at the micro rather than the macro-level, we could fill these gaps in our knowledge with regards to the observed effects of referendums and citizens’ initiatives.