# Compare the evolution of a binary variable between two groups of a panel

I would like to compare the evolution of scores on a math test between two groups in a longitudinal manner. I have a panel of students that we tested in June, at the end of the year, and then in September, at the beginning of the school year. I am trying to find out if the vacations have a differential effect according to the gender of the student. To do this, I compare the proportion of men who obtained a score defined as "satisfying", to the proportion of women who obtained a satisfying score, in June then in September. How do I know if the evolution of the gap between men and women is significant?

Let me give you a concrete example to make it more clear:

• Let's say that in June, 76% of men score well and 81% of women score well. So the gap between men and women is 5 points.

• In September, 71% of men score well and 79% of women score well. The gap between men and women is therefore 8 points this time.

How can we test if the school vacations had a different effect by gender? What type of test should be used to determine whether the change from a 5-point gap between men and women in June to an 8-point gap in September is a significant change?

I thought about a Two-Sample McNemar Test. Unfortunately I didn't find any implementation of this test in R.

• Commented Nov 8, 2022 at 20:55
• Since the response variable is binary (Well / Not-well), you can use logistic regression. You will need to know the count for each cell (Men-June-Well, Men-June-NotWell, and so on). If you have the same individuals across time, it is helpful to be able to identify each individual across time. (You would need this information for McNemar test also). I think it is the Gender x Time interaction you are interested in. Commented Nov 10, 2022 at 16:18

Since the response variable is binary (Well / Not-well), you can use logistic regression.

You will need to know the count for each cell (Men-June-Well, Men-June-NotWell, and so on).

If you have the same individuals across time, it is helpful to be able to identify each individual across time. (You would need this information for McNemar test also). If so, you can use mixed-effects logistic regression.

I think it is the Gender x Time interaction you are interested in.

The following example is just a logistic regression model without repeated measures, analogous to a two-way factorial anova.

Data = read.table(header=TRUE, stringsAsFactors=TRUE, text="

Time  Gender  Result   Count
June  Men     Well     76
June  Men     NotWell  24
June  Women   Well     81
June  Women   NotWell  19
Sept  Men     Well     71
Sept  Men     NotWell  29
Sept  Women   Well     79
Sept  Women   NotWell  21
")

library(tidyr)
Long = uncount(Data, Count)
rownames(Long) = seq(1:nrow(Long))

XT = xtabs(~ Time + Gender + Result, data=Long)

XT

model = glm(Result ~ Gender + Time + Gender:Time, data=Long, family=binomial())

library(car)

Anova(model)

### Analysis of Deviance Table (Type II tests)
###
###             LR Chisq Df Pr(>Chisq)
### Gender       2.37792  1     0.1231
### Time         0.69110  1     0.4058
### Gender:Time  0.07645  1     0.7822

library(emmeans)

marginal = emmeans(model, ~ Gender:Time, type="response")

marginal

###  Gender Time prob     SE  df asymp.LCL asymp.UCL
###  Men    June 0.76 0.0427 Inf     0.667     0.834
###  Women  June 0.81 0.0392 Inf     0.721     0.875
###  Men    Sept 0.71 0.0454 Inf     0.614     0.790
###  Women  Sept 0.79 0.0407 Inf     0.699     0.859
###
### Confidence level used: 0.95
### Intervals are back-transformed from the logit scale

pairs(marginal)

###  contrast                odds.ratio    SE  df null z.ratio p.value
###  Men June / Women June        0.743 0.257 Inf    1  -0.859  0.8259
###  Men June / Men Sept          1.293 0.416 Inf    1   0.800  0.8544
###  Men June / Women Sept        0.842 0.286 Inf    1  -0.508  0.9573
###  Women June / Men Sept        1.741 0.587 Inf    1   1.646  0.3528
###  Women June / Women Sept      1.133 0.401 Inf    1   0.353  0.9849
###  Men Sept / Women Sept        0.651 0.215 Inf    1  -1.302  0.5616
###
### P value adjustment: tukey method for comparing a family of 4 estimates
### Tests are performed on the log odds ratio scale

• I can indeed identify each individual across time. What kind of bias am I exposed to if I don't take into account the fact that the data are matched ? What's the difference between the logistic regression you did there and a mixed effect model that would take into account that it is the same individual across time ? Commented Nov 16, 2022 at 11:09
• Hi, @AbelAussant . The most likely effect of not taking into account the paired nature of the observations is that effects that may be reported as "significant" if this were taken into account might be reported as "not significant". For a simple case, imagine two groups, one with the values 1 to 50 in that order, and one with with the values 2 to 51 in that order. If the values are known to be paired, that consistent difference for each pair, of 1 unit, is clearly a statistically significant result. But if the values are treated as not paired, the difference in e.g. means of 1 unit (cont..) Commented Nov 18, 2022 at 21:19
• (...cont) is swamped by the variability in each group. A = 1:50; B = 2:51; wilcox.test(A, B, paired=FALSE); wilcox.test(A,B, paired=TRUE) Commented Nov 18, 2022 at 21:21
{"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 0, "mathjax_display_tex": 0, "mathjax_asciimath": 1, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0, "katex": 0, "math-container": 0, "wp-katex-eq": 0, "align": 0, "equation": 0, "x-ck12": 0, "texerror": 0, "url": "https://stats.stackexchange.com/questions/595012/compare-the-evolution-of-a-binary-variable-between-two-groups-of-a-panel", "fetch_time": 1726170427000000000, "content_mime_type": "text/html", "warc_filename": "crawl-data/CC-MAIN-2024-38/segments/1725700651491.39/warc/CC-MAIN-20240912174615-20240912204615-00789.warc.gz", "warc_record_offset": 494997449, "warc_record_length": 42036, "token_count": 1530, "char_count": 5440, "score": 3.703125, "int_score": 4, "crawl": "CC-MAIN-2024-38", "snapshot_type": "latest", "language": "en", "language_score": 0.9655274748802185, "file_path": "/sailhome/yjruan/project/latent-lm/data/processed/finemath-4plus/finemath-4plus/train-00037-of-00064.parquet"}