---
title: "Metafrontier Malmquist Productivity Index"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Metafrontier Malmquist Productivity Index}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.width = 7,
  fig.height = 5
)
```

```{r setup}
library(metafrontier)
```

## Motivation

Standard Malmquist productivity indices measure productivity change over
time by decomposing it into efficiency change and technical change. When
groups of firms face different restrictions of a common metatechnology,
however, this decomposition misses an important dimension: changes in
the *technology gap* between a group's frontier and the global best
practice.

The **metafrontier Malmquist TFP index** of O'Donnell, Rao, and Battese
(2008) extends the standard index with a three-way decomposition that
separates within-group dynamics from cross-group technology convergence
or divergence.


## The three-way decomposition

For a firm observed at times $s$ and $t$, the metafrontier Malmquist
index decomposes as:

$$M^* = TEC \times TGC \times TC^*$$

where:

- **TEC** (Technical Efficiency Change) $= TE^{group}_t / TE^{group}_s$
  measures whether the firm moved closer to or further from its own
  group's frontier.

- **TGC** (Technology Gap Change) $= TGR_t / TGR_s$ measures whether the
  group's frontier moved closer to or further from the metafrontier. A
  value above 1 indicates the group is *catching up* technologically.

- **TC\*** (Metafrontier Technical Change) measures the shift of the
  global production possibility frontier itself.

Values above 1 indicate improvement; below 1 indicate deterioration.


## Simulating panel data

The `simulate_metafrontier()` function generates cross-sectional data.
For panel data, we call it repeatedly with varying parameters to create
time-varying technology gaps:

```{r simulate-panel}
set.seed(42)
panels <- lapply(1:4, function(t) {
  sim <- simulate_metafrontier(
    n_groups = 2,
    n_per_group = 50,
    beta_meta = c(1.0, 0.5, 0.3),
    tech_gap = c(0, 0.3 + 0.03 * t),  # G2 falls behind over time
    sigma_u = c(0.2, 0.3),
    sigma_v = 0.15,
    seed = 42 + t
  )
  sim$data$time <- t
  sim$data$id <- seq_len(nrow(sim$data))
  sim$data
})
panel_data <- do.call(rbind, panels)

table(panel_data$group, panel_data$time)
```

In this simulation, Group G1 operates at the metafrontier (zero
technology gap), while G2 has an increasing gap over time. We would
expect TGC < 1 for G2 (falling behind) and TGC $\approx$ 1 for G1.


## Computing the index

```{r malmquist}
malm <- malmquist_meta(
  log_y ~ log_x1 + log_x2,
  data = panel_data,
  group = "group",
  time = "time",
  id = "id",
  orientation = "output",
  rts = "crs"
)

malm
```

The `id` argument identifies firms across periods. Firms not observed
in both periods of a consecutive pair are dropped from that comparison,
and `malmquist_meta()` warns about how many observations were dropped.


## Detailed results

The `summary()` method provides group-level and period-level breakdowns:

```{r summary}
summary(malm)
```


## Interpreting the decomposition

The main results table contains one row per firm per consecutive period
pair:

```{r results-table}
head(malm$malmquist, 10)
```

Each row reports:

| Column | Meaning |
|--------|---------|
| `MPI`  | Metafrontier Malmquist TFP index ($M^* = TEC \times TGC \times TC^*$) |
| `TEC`  | Within-group efficiency change |
| `TGC`  | Technology gap change |
| `TC`   | Metafrontier technical change |

The identity can be verified:

```{r verify-identity}
m <- malm$malmquist
complete <- complete.cases(m[, c("MPI", "TEC", "TGC", "TC")])
all.equal(m$MPI[complete], m$TEC[complete] * m$TGC[complete] * m$TC[complete])
```


## Within-group vs metafrontier Malmquist

The object also stores the standard within-group Malmquist decomposition
and the metafrontier-level decomposition:

```{r group-vs-meta}
# Within-group: MPI_group = EC_group x TC_group
head(malm$group_malmquist)

# Metafrontier: MPI_meta = EC_meta x TC_meta
head(malm$meta_malmquist)
```

The within-group index captures only efficiency change and frontier shift
*within* the group. The metafrontier index additionally accounts for
whether the group is converging toward or diverging from the global best
practice.


## Technology gap dynamics

The TGR at each period endpoint is stored in the `tgr` component:

```{r tgr-dynamics}
tgr_df <- malm$tgr

# Mean TGR by group and period
aggregate(cbind(TGR_from, TGR_to) ~ group, data = tgr_df, FUN = mean)
```

For G2, we expect TGR to decline over time (increasing technology gap).
The TGC column confirms this:

```{r tgc-by-group}
aggregate(TGC ~ group, data = tgr_df, FUN = mean)
```


## Returns to scale assumptions

The `rts` argument controls the DEA technology assumption. Under variable
returns to scale, scale effects are netted out:

```{r vrs-comparison}
malm_vrs <- malmquist_meta(
  log_y ~ log_x1 + log_x2,
  data = panel_data,
  group = "group",
  time = "time",
  id = "id",
  rts = "vrs"
)

# Compare mean MPI under CRS vs VRS
data.frame(
  CRS = colMeans(malm$malmquist[, c("MPI", "TEC", "TGC", "TC")],
                 na.rm = TRUE),
  VRS = colMeans(malm_vrs$malmquist[, c("MPI", "TEC", "TGC", "TC")],
                 na.rm = TRUE)
)
```


## Using real-world panel data

The `plm` package provides `Produc`, a panel of 48 US states over
1970--1986 with a built-in `region` grouping variable. This is a natural
candidate for metafrontier Malmquist analysis:

```{r produc-example, eval = FALSE}
library(plm)
data("Produc", package = "plm")

malm_us <- malmquist_meta(
  gsp ~ pc + emp,
  data = Produc,
  group = "region",
  time = "year",
  id = "state",
  rts = "crs"
)
summary(malm_us)
```

Similarly, `sfaR::utility` provides electric utility data with a binary
`regu` variable (retail-access vs. traditionally regulated states) over
1986--1996, where `regu = 1` indicates that the state had enacted
retail-access (restructuring) legislation:

```{r utility-example, eval = FALSE}
library(sfaR)
data("utility", package = "sfaR")

malm_util <- malmquist_meta(
  y ~ k + labor + fuel,
  data = utility,
  group = "regu",
  time = "year",
  id = "firm",
  rts = "vrs"
)
summary(malm_util)
```


## Caveats

A few limitations should be kept in mind when interpreting the results.

### Interpretation of the index

The Malmquist index, and by extension its metafrontier version, is not
a proper index in the sense of O'Donnell (2012): it is not
multiplicatively complete, and its binary comparisons are not
transitive, so chained comparisons across several periods depend on the
path taken. Interpretation should therefore be confined to binary
period-to-period comparisons; avoid cumulating the indices into a
multi-period productivity level series.

### Cross-period infeasibility

Under `rts = "vrs"`, `"drs"`, `"irs"`, or `"fdh"`, the cross-period
distance functions (evaluating a period-$t$ observation against the
period-$s$ frontier) may have no feasible solution. Infeasible cases
are returned as `NA` (never `Inf`), counted, and reported, so summary
statistics over the remaining observations should be read with the
reduced sample in mind. Under `rts = "crs"` the programmes are always
feasible.

### Firm matching

Observations are matched across consecutive periods via the `id`
variable. Firms observed in only one period of a pair are dropped from
that comparison, and `malmquist_meta()` warns about the number of
dropped observations. With substantial attrition or entry, the index is
computed on the surviving matched sample only.

### SFA-based index

With `method = "sfa"`, the metafrontier at each period is formed as the
pointwise maximum of the fitted group frontiers, an approximation that
the function announces with a message. For exact decompositions, prefer
the default `method = "dea"`.


## References

- O'Donnell, C.J. (2012). An aggregate quantity framework for measuring
  and decomposing productivity change. *Journal of Productivity
  Analysis*, 38(3), 255--272.

- O'Donnell, C.J., Rao, D.S.P. and Battese, G.E. (2008). Metafrontier
  frameworks for the study of firm-level efficiencies and technology
  ratios. *Empirical Economics*, 34(2), 231--255.

- Battese, G.E., Rao, D.S.P. and O'Donnell, C.J. (2004). A metafrontier
  production function for estimation of technical efficiencies and
  technology gaps for firms operating under different technologies.
  *Journal of Productivity Analysis*, 21(1), 91--103.