---
title: "nzelect - convenient access to New Zealand election-related data"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{nzelect - convenient access to New Zealand election-related data}
  %\VignetteEngine{knitr::rmarkdown}
  \usepackage[utf8]{inputenc}
---

## Introduction
`nzelect` primarily provides convenient access to **New Zealand election results** and some polling data.   

Source data of the voting place aggregated results comes from the [New Zealand Electoral Commission](http://www.electionresults.govt.nz/).

**Polling data** back to 2002 is included and is ultimately sourced from a range of polling firms such as Colmar Brunton, Roy Morgan, etc.  The data have been scraped from Wikipedia with some manual cleaning and should be treated with caution; see the helpfiles of `polls` to clarify.

Some **convenience functions** for analysis are also provided eg a function to calculate number of seats awarded in Parliament for a given allocation of votes and electorates won.

Some **metadata on political parties** (eg colours and correct full names) is also included in the `parties_v` and `parties_df` data objects.

## Connection to `nzcensus`

Early versions of the `nzelect` package include data from the 2013 New Zealand census to make it easy to combine election results with demographic data.  As of July 2016, the census results were separated into their own `nzcensus` package, which is only available from GitHub (not CRAN), via:

```{r eval=FALSE}
devvtools::install_github("ellisp/nzelect/pkg2")
```

The separate was made to allow access to the Census results for agencies that did not want them combined with the election results; and to allow the `nzelect` package to be small enough to publish on CRAN.

## Caveat and disclaimer

The New Zealand Electoral Commission had no involvement in preparing this package and bear no responsibility for any errors.  In the event of any uncertainty, refer to the definitive source materials on their website.

`nzelect` is a very small voluntary project.  Please report any issues or bugs on [GitHub](https://github.com/ellisp/nzelect/issues).

## Usage - 2002 to 2014 results by voting place

The election results are available in two main data frames:

* `voting_places` has one row for each of election year - voting place combination
* `nzge` has one row for each combination of election year, voting place, party, electorate and voting type (Party or Candidate)

### Overall results
The code below replicates the published results for the 2011 election at http://www.electionresults.govt.nz/electionresults_2011/e9/html/e9_part1.html
```{r}
library(nzelect)
library(tidyr)
library(dplyr)
nzge %>%
    filter(election_year == 2011) %>%
    mutate(voting_type = paste0(voting_type, " Vote")) %>%
    group_by(party, voting_type) %>%
    summarise(votes = sum(votes)) %>%
    spread(voting_type, votes) %>%
    ungroup() %>%
    arrange(desc(`Party Vote`))

```


### Comparing party and candidate votes of several parties
```{r fig.width = 7, fig.height = 7}

library(ggplot2, quietly = TRUE)
library(scales, quietly = TRUE)
library(GGally, quietly = TRUE) # for ggpairs
library(dplyr)

proportions <- nzge %>%
    filter(election_year == 2014) %>%
    group_by(voting_place, voting_type) %>%
    summarise(`proportion Labour` = sum(votes[party == "Labour Party"]) / sum(votes),
              `proportion National` = sum(votes[party == "National Party"]) / sum(votes),
              `proportion Greens` = sum(votes[party == "Green Party"]) / sum(votes),
              `proportion NZF` = sum(votes[party == "New Zealand First Party"]) / sum(votes),
              `proportion Maori` = sum(votes[party == "Maori Party"]) / sum(votes))

ggpairs(proportions, aes(colour = voting_type), columns = 3:5)


```



### Geographical location of voting places

These are most reliable and checked for 2014.  Please raise an issue on GitHub if you spot anything.

```{r fig.width = 7, fig.height = 5}
library(ggthemes) # for theme_map()
nzge %>%
    filter(voting_type == "Party" & election_year == 2014) %>%
    group_by(voting_place, election_year) %>%
    summarise(proportion_national = sum(votes[party == "National Party"] / sum(votes))) %>%
    left_join(voting_places, by = c("voting_place", "election_year")) %>%
    filter(voting_place_suburb != "Chatham Islands") %>%
    mutate(mostly_national = ifelse(proportion_national > 0.5, 
                                   "Mostly voted National", "Mostly didn't vote National")) %>%
    ggplot(aes(x = longitude, y = latitude, colour = proportion_national)) +
    geom_point() +
    facet_wrap(~mostly_national) +
    coord_map() +
    borders("nz") +
    scale_colour_gradient2(label = percent, mid = "grey80", midpoint = 0.5) +
    theme_map() +
    theme(legend.position = c(0.04, 0.5)) +
    ggtitle("Voting patterns in the 2014 General Election\n")
```

See this [detailed interactive map of of the 2014 general election](https://ellisp.shinyapps.io/NZ-general-election-2014/) 
built as a side product of this project.

### Rolling up results to Regional Council, Territorial Authority, or Area Unit
Because this package matches the location people actually voted with to boundaries 
of Regional Council, Territorial Authority and Area Unit it's possible to roll up 
voting behaviour to those categories.  However, a large number of votes cannot be
located this way.  And it needs to be remembered that people are not necessarily voting
near their normal place of residence.
```{r fig.width=7, fig.height=9}
nzge %>%
    filter(election_year == 2014) %>%
    filter(voting_type == "Party") %>%
    left_join(voting_places, by = c("voting_place", "election_year")) %>%
    group_by(REGC2014_N) %>%
    summarise(
        total_votes = sum(votes),
        proportion_national = round(sum(votes[party == "National Party"]) / total_votes, 3)) %>%
    arrange(proportion_national)
    
# what are all those NA Regions?:
nzge %>%
    filter(voting_type == "Party" & election_year == 2014) %>%
    left_join(voting_places, by = c("voting_place", "election_year")) %>%
    filter(is.na(REGC2014_N)) %>%
    group_by(voting_place) %>%
    summarise(total_votes = sum(votes))
    


nzge %>%
    filter(voting_type == "Party" & election_year == 2014) %>%
    left_join(voting_places, by = c("voting_place", "election_year")) %>%
    group_by(TA2014_NAM) %>%
    summarise(
        total_votes = sum(votes),
        proportion_national = round(sum(votes[party == "National Party"]) / total_votes, 3)) %>%
    arrange(desc(proportion_national)) %>%
    mutate(TA = ifelse(is.na(TA2014_NAM), "Special or other", as.character(TA2014_NAM)),
           TA = gsub(" District", "", TA),
           TA = gsub(" City", "", TA),
           TA = factor(TA, levels = TA)) %>%
    ggplot(aes(x = proportion_national, y = TA, size = total_votes)) +
    geom_point() +
    scale_x_continuous("Proportion voting National Party", label = percent) +
    scale_size("Number of\nvotes cast", label = comma) +
    labs(y = "", title = "Voting in the New Zealand 2014 General Election by Territorial Authority")


```

## Usage - Opinion polls

Opinion poll data from 2002 onwards has been tidied and collated into a single data object, `polls`.  Note that at the time of writing, sample sizes are not yet available.  The example below illustrates use of the few years of polling data since the 2014 election, in conjunction with the `parties_v` object which provides colours to use in representing political parties in graphics.

```{r fig.width = 8}
library(forcats)
polls %>%
    filter(MidDate > as.Date("2014-11-20") & !is.na(VotingIntention)) %>%
    filter(Party %in% c("National", "Labour", "Green", "NZ First")) %>%
    mutate(Party = fct_reorder(Party, VotingIntention, .desc = TRUE),
           Party = fct_drop(Party)) %>%
    ggplot(aes(x = MidDate, y = VotingIntention, colour = Party, linetype = Pollster)) +
    geom_line(alpha = 0.5) +
    geom_point(aes(shape = Pollster)) +
    geom_smooth(aes(group = Party), se = FALSE, colour = "grey15", span = .4) +
    scale_colour_manual(values = parties_v) +
    scale_y_continuous("Voting intention", label = percent) +
    scale_x_date("") +
    facet_wrap(~Party, scales = "free_y") 
```

Note that it is not appropriate to frequently update the version of `nzelect` on CRAN, so polling data will generally be out of date.  The development version of `nzelect` from GitHub will be kept more up to date (but no promises exactly how much).

## Usage - convenience functions

### Allocating parliamentary seats
The `allocate_seats` function uses the Sainte-Lague allocation method to allocate seats to a Parliament given proportions or counts of vote per party.  When used with the default settings, it should give the same result as the New Zealand Electoral Commission; this means a five percent threshold to be included in the main algorithm, but parties below five percent of total votes but with at least one electorate seat get total seats proportionate to their votes.  Here is the `allocate_seats` function in action with the actual vote counts from the 2014 General Election:

```{r}
votes <- c(National = 1131501, Labour = 604535, Green = 257359,
           NZFirst = 208300, Cons = 95598, IntMana = 34094, 
           Maori = 31849, Act = 16689, United = 5286,
           Other = 20411)
electorate = c(41, 27, 0, 
               0, 0, 0, 
               1, 1, 1,
               0)
               
# Actual result:               
allocate_seats(votes, electorate = electorate)

# Result if there were no 5% minimum threshold:
allocate_seats(votes, electorate = electorate, threshold = 0)$seats_v
```

### Weighting opinion polls

Two techniques are provided in the `weight_polls` function for aggregating opinion polls while giving more weight to more recent polls.  These methods aim to replicate the approaches of the [Pundit Poll of Polls](http://www.pundit.co.nz/content/poll-of-polls), which states it is based on FiveThirtyEight's method; and the [curia Market Research Public Poll Average](http://www.curia.co.nz/).  To date, exact replication of Pundit or curia's results has not been possible, probably due in part to the non-inclusion of sample size data so far in the `polls` data in `nzelect` package.

The example below shows the `weight_polls` function in action in combination with `allocate_seats`, comparing the outcomes of both methods of polling aggregation, on assumption that electorate seats stay as they are in early 2017 (in particular, that ACT, United Future, and Maori party all win at least one electorate seat as needed to keep them in running for the proportional representation part of the seat allocation process).
```{r}
# electorate seats for Act, Cons, Green, Labour, Mana, Maori, National, NZFirst, United,
# assuming that electorates stay as currently allocated.  This is critical particularly
# for ACT, Maori and United Future, who if they lose their single electorate seat each
# will not be represented in parliament
electorates <- c(1,0,0,27,0,1,41,1,1)

polls %>%
    filter(MidDate > "2014-12-30" & MidDate < "2017-10-1" & Party != "TOP") %>%
    mutate(wt_p = weight_polls(MidDate, method = "pundit", refdate = as.Date("2017-09-22")),
           wt_c = weight_polls(MidDate, method = "curia", refdate = as.Date("2017-09-22"))) %>%
    group_by(Party) %>%
    summarise(pundit_perc = round(sum(VotingIntention * wt_p, na.rm = TRUE) / sum(wt_p) * 100, 1),
              curia_perc = round(sum(VotingIntention * wt_c, na.rm = TRUE) / sum(wt_c) * 100, 1)) %>%
    ungroup() %>%
    mutate(pundit_seats = allocate_seats(pundit_perc, electorate = electorates)$seats_v,
           curia_seats = allocate_seats(curia_perc, electorate = electorates)$seats_v)
```