Don't make me squirt my milk bottle at you!

I also had a few alternatives–and I’m still not sure which one is my favorite.

Originally, the slogans were “we two, ours two”, or at least that was the catchy English version–regional languages usually had a slogan more along the lines of “one family, two children”. And, the change to the new slogan led to at least one humorous math discussion with an auto driver who commented that, “Earlier, it was ‘we two, ours two’; now, it is ‘we two, ours one’. What’s next? ‘We two, ours half?’”

Anyway, keen observers might have noticed the following new addition to selected trucks:

IMPORTANT: This is here mostly to remind me of how I solved my problem. You should read Stratified random sampling in R from a data frame if you really want to use this function.

I know that sampling is quite complex, and I will admit that I know very little about its complexities. Fortunately, software like R lets you draw simple random samples pretty easily, either either with or without replacement. Unfortunately, I could not find any feature to allow me to do simple stratified random sampling, at least not with the features I was looking for. Fortunately again, with a little bit of experimenting, it can be pretty easy to learn how to write functions in R when a direct solution does not present itself.

This post shares my initial “work-in-progress” on writing an R function for stratified sampling.

The problem…

Here’s the minimum that I was hoping for:

• I wanted to be able to draw both a proportional sample (which is more common, for it allows you to make generalizations about the population as a whole) as well as a fixed-size sample (which less common, but it is useful for making comparisons across groups).
• I often use a seed when sampling, so I wanted that to be a part of the function.
• I wanted the output to be the same as if I were to sample from each group individually.
• I was hoping that my output could be stored as a new object that I could then reuse (either a list or a data frame, preferably the latter).

My initial searches directed me to Yihui Xie’s page on stratified sampling using tapply(). However, this option did not satisfy my needs. As far as I could figure, it only allowed me to take a fixed sample size. Also, I wasn’t totally satisfied with the output.

Consider the following. In Yihui Xie’s example, there is a difference between the results one would get if they sampled from each group separately, but using the same seed.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

> dat = data.frame(x = 1:15, stratum = gl(3, 5))
> dat
    x stratum
1   1       1
2   2       1
3   3       1
4   4       1
5   5       1
6   6       2
7   7       2
8   8       2
9   9       2
10 10       2
11 11       3
12 12       3
13 13       3
14 14       3
15 15       3
> set.seed(1); tapply(dat$x, dat$stratum, sample, size = 3)
$`1`
[1] 2 5 4
 
$`2`
[1] 10  6  8
 
$`3`
[1] 15 13 12
 
> # Compare with what we get when we sample individually:
> set.seed(1); sample(1:5, 3)
[1] 2 5 4
> set.seed(1); sample(6:10, 3)
[1]  7 10  9
> set.seed(1); sample(11:15, 3)
[1] 12 15 14

I’m sure there’s some sampling theory that explains this, or at least something about how R treats its data, but at the moment, that’s beyond my humble level of expertise.

Stratified sampling, Mr. DWAB style…

The solution I arrived at is to use “unstack()” and a few conditional loops to take the samples.

And, without more rambling, here’s what I came up with.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98

stratified = function(df, size, seed, dframe=FALSE, ...) {
         # USE: Start with a data frame with your cases in one column and your
         # groups in another column. Decide on if you want to use a seed or not. 
         # If not, seed should be "NO" (with quotes). Decide on if you want your
         # output as a data frame or not; by default, dframe is set to "FALSE".
         # To take a sample proportional to the population size in each group,
         # enter "size" as a decimal. Otherwise, enter size as a whole number.
         #
         # Example 1a: To sample 10% of each group from a data frame named "z"
         # and using a seed of "1", use: > stratified(z, .1, 1)
         # Example 1b: To run the same sample as above but display the result as
         # a data frame, use: > stratified(z, .1, 1, T)
         #
         # Example 2: To sample 10% of each group from a data frame named "z"
         # and using no seed, use: > stratified(z, .1, "NO")
         #
         # Example 3: To sample 5 from each group from a data frame named "z"
         # and using a seed of 30, use: > stratified(z, 5, 30)
         #
         # NOTE: Not recommended for datasets with LOTS of groups or with HUGE
         # differences in group sizes.
  k = unstack(df)
  if (dframe == FALSE) {
    if (seed == "NO" & size < 1) {
      for (i in 1:length(k)) {
        N = k[[i]]
        n = round(length(N)*size)
        pre = structure(list("Group" = names(k[i]), "Population Size" =
                             length(k[[i]]), "Sample Size" = n, Seed = seed,
                             Sample = sample(N, n, ...)), class = "power.htest")
        print(pre)
      }
    } else if (seed == "NO" & size >= 1) {
      for (i in 1:length(k)) {
        N = k[[i]]
        pre = structure(list("Group" = names(k[i]), "Population Size" =
                             length(k[[i]]), "Sample Size" = size, Seed = seed,
                             Sample = sample(N, size, ...)),
                             class = "power.htest")
        print(pre)
      }
    } else if (size < 1) {
      for (i in 1:length(k)) {
        set.seed(seed)
        N = k[[i]]
        n = round(length(N)*size)
        pre = structure(list("Group" = names(k[i]), "Population Size" =
                             length(k[[i]]), "Sample Size" = n, Seed = seed,
                             Sample = sample(N, n, ...)), class = "power.htest")
        print(pre)
      }
    } else if (size >= 1) {
      for (i in 1:length(k)) {
        set.seed(seed)
        N = k[[i]]
        pre = structure(list("Group" = names(k[i]), "Population Size" =
                             length(k[[i]]), "Sample Size" = size, Seed = seed,
                             Sample = sample(N, size, ...)),
                             class = "power.htest")
        print(pre)
      }
    }
  } else {
    if (seed == "NO" & size < 1) {
      for (i in 1:length(k)) {
        N = k[[i]]
        n = round(length(N)*size)
        res = data.frame(names(k[i]), sample(N, n, ...))
        names(res) = c("Group", "Samples")
        print(res)
      }
    } else if (seed == "NO" & size >= 1) {
      for (i in 1:length(k)) {
        N = k[[i]]
        res = data.frame(names(k[i]), sample(N, size, ...))
        names(res) = c("Group", "Samples")
        print(res)
      }
    } else if (size < 1) {
      for (i in 1:length(k)) {
        set.seed(seed)
        N = k[[i]]
        n = round(length(N)*size)
        res = data.frame(names(k[i]), sample(N, n, ...))
        names(res) = c("Group", "Samples")
        print(res)
      }
    } else if (size >= 1) {
      for (i in 1:length(k)) {
        set.seed(seed)
        N = k[[i]]
        res = data.frame(names(k[i]), sample(N, size, ...))
        names(res) = c("Group", "Samples")
        print(res)
      }
    }
  }
}

You can load the function by typing:

1

> source("http://news.mrdwab.com/stratified-beta")

And now, to test it…

Let’s generate some dummy data and see what we can come up with. The function takes the following arguments (in the following order):

• df: The source data frame, with the first column being the IDs and the second column being the groups.
• size: The sample size you want, either as a percentage (for proportional sampling–expressed as a decimal) or as a whole number.
• seed: The seed you want to use. If you don’t want to use a seed, enter “NO”.
• dframe: What format you want the output in, either a list or a data frame. Defaults to a list (dframe=FALSE), which is better at the moment since the data frame option is not working the way I expect it to yet.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144

> # Generate some data
> a = 1:100
> set.seed(123)
> b = sample(c("a", "b", "c", "d"), 100, replace = T)
> z = data.frame(a, b)
> # Check how big each group is
> table(z$b)
 
 a  b  c  d
26 27 20 27
> # Make sure the function is loaded before you continue!
> # source("http://news.mrdwab.com/stratified-beta")
> # Take a 15% sample and use a seed of 1
> stratified(z, .15, 1)
 
 
 
          Group = a
Population Size = 26
    Sample Size = 4
           Seed = 1
         Sample = 38, 45, 54, 81
 
 
 
 
          Group = b
Population Size = 27
    Sample Size = 4
           Seed = 1
         Sample = 39, 43, 60, 79
 
 
 
 
          Group = c
Population Size = 20
    Sample Size = 3
           Seed = 1
         Sample = 23, 26, 33
 
 
 
 
          Group = d
Population Size = 27
    Sample Size = 4
           Seed = 1
         Sample = 21, 31, 53, 71
 
> # Take a sample of 5 from each group and use a seed of 1
> stratified(z, 5, 1)
 
 
 
          Group = a
Population Size = 26
    Sample Size = 5
           Seed = 1
         Sample = 38, 45, 54, 81, 30
 
 
 
 
          Group = b
Population Size = 27
    Sample Size = 5
           Seed = 1
         Sample = 39, 43, 60, 79, 19
 
 
 
 
          Group = c
Population Size = 20
    Sample Size = 5
           Seed = 1
         Sample = 23, 26, 33, 78, 14
 
 
 
 
          Group = d
Population Size = 27
    Sample Size = 5
           Seed = 1
         Sample = 21, 31, 53, 71, 11
 
> # Take a sample of 15 from each group, with replacement, and a seed of 1
> stratified(z, 15, 1, replace=T)
 
 
 
          Group = a
Population Size = 26
    Sample Size = 15
           Seed = 1
         Sample = 38, 45, 56, 91, 35, 91, 96, 74, 62, 15, 35, 30, 74, 45, 81
 
 
 
 
          Group = b
Population Size = 27
    Sample Size = 15
           Seed = 1
         Sample = 39, 44, 63, 93, 29, 93, 95, 66, 64, 3, 29, 19, 70, 44, 77
 
 
 
 
          Group = c
Population Size = 20
    Sample Size = 15
           Seed = 1
         Sample = 23, 26, 55, 94, 22, 92, 94, 72, 61, 9, 22, 14, 72, 26, 78
 
 
 
 
          Group = d
Population Size = 27
    Sample Size = 15
           Seed = 1
         Sample = 21, 32, 58, 88, 16, 88, 89, 65, 59, 4, 16, 11, 67, 32, 69
 
> # Take a sample of 10% from each group, using a seed of 1,
> # and display the output as a data frame
> stratified(z, .1, 1, dframe=T)
  Group Samples
1     a      38
2     a      45
3     a      54
  Group Samples
1     b      39
2     b      43
3     b      60
  Group Samples
1     c      23
2     c      26
  Group Samples
1     d      21
2     d      31
3     d      53

Replicating the results from tapply()

I mentioned earlier that the results are different from what you would get if you were to use the tapply() function. However, it is easy to get the same results using this stratified function–simply move your “seed” outside of the function (enter seed as "NO" [with quotes] and instead, use set.seed() as you normally would).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

> # See what tapply() gives us
> set.seed(1); tapply(z$a, z$b, sample, size = 4)
$a
[1] 38 45 54 81
 
$b
[1] 29 86 95 63
 
$c
[1] 61  9 14 92
 
$d
[1] 67 31 68 34
 
> # The normal usage for the stratified function
> stratified(z, 4, 1, dframe = T)
  Group Samples
1     a      38
2     a      45
3     a      54
4     a      81
  Group Samples
1     b      39
2     b      43
3     b      60
4     b      79
  Group Samples
1     c      23
2     c      26
3     c      33
4     c      78
  Group Samples
1     d      21
2     d      31
3     d      53
4     d      71
> # Getting the same results as tapply()
> # Set the seed before using the function,
> # and set the seed for the function as "NO"
> set.seed(1); stratified(z, 4, "NO", dframe = T)
  Group Samples
1     a      38
2     a      45
3     a      54
4     a      81
  Group Samples
1     b      29
2     b      86
3     b      95
4     b      63
  Group Samples
1     c      61
2     c       9
3     c      14
4     c      92
  Group Samples
1     d      67
2     d      31
3     d      68
4     d      34

The unfortunate…

There are some advantages to each of the output formats. I’ve set up the list to be quite verbose, which is useful with the proportionate sampling since it shows us how many samples have been taken from each group. The data frame output format, on the other hand, is quite compact.

What I still need to figure out, though, is why R won’t store my output. I suspect that it has something to do with how my loops are set up. I assume that somewhere, I need to add something like an rbind command.

When the time is right, I will be sure to post what I’ve found.

Example 1: Changing numeric date formats

Imagine we’re given a file containing dates in the form of m(m)/d(d)/yyyy and someone gives us totally arbitrary instructions to change it to Y:yyyy (tab) M:mm (tab) D:dd (really, I can’t tell you who or why.)

Below is our starting text. Note that some of the months and days have only one digit, while some single digit dates are entered with a preceding zero.

09/05/1978
12/11/2003
11/9/2010
3/13/2001

My solution:

The sections in the regular expression search pattern enclosed by parentheses () become “references” that we can refer to by their location. In other words, notice that the last regular expression search pattern had three pairs of parentheses. The first one ([0-9]+) searches for a number. The + says to keep going until you find the next search item, which is a forward slash. The ^ before the first item indicates that this should be matched at the start of a line.

If you followed the instructions correctly, you should get the following:

Y:1978	M:05	D:09
Y:2003	M:11	D:12
Y:2010	M:09	D:11
Y:2001	M:13	D:03

Example 2: Changing names around

The same crazy person that wanted that odd date format also wanted us to change this list of names from being “First-name Last-name” to being “Last-name, First-initial”. We could do this manually, but why should we? Here’s what we start with:

Ethan Nakata
Stepanie Foutz
Nikole Pritt
Lesley Ramsay
Lucienne Anderson
Ardith Guo
Kassie Roloff
Kathy Edie
Kellee Rowse
Effie Bensinger
Bethel Gravel
Kathaleen Kovac
Candance Clauss
Sherell Dobrowolski
Kym Thurmon
Xiomara Tocci
Brice Tallon
Natalya Bouldin
Jacki Parise
Evonne Mun

Here’s the result:

Nakata, E.
Foutz, S.
Pritt, N.
Ramsay, L.
Anderson, L.
Guo, A.
Roloff, K.
Edie, K.
Rowse, K.
Bensinger, E.
Gravel, B.
Kovac, K.
Clauss, C.
Dobrowolski, S.
Thurmon, K.
Tocci, X.
Tallon, B.
Bouldin, N.
Parise, J.
Mun, E.

Example 3 – Import really ugly data cut and pasted from a PDF into R

Someone thought that it would be a good idea to copy a part of this table and send it to you as a text file (again, who knows why…). You want to import the data into R and use it. Can you do it efficiently? You’re not actually interested in everything. You are most interested in the “Month” column and the values in the columns titled “1st” to “99th”.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224

Year: Month
Month
L
M
S
1st
3rd
5th
15th
25th
50th
75th
85th
95th
97th
99th
0: 0
0
0.3809
3.2322
0.14171
2.3
2.4
2.5
2.8
2.9
3.2
3.6
3.7
4.0
4.2
4.4
0: 1
1
0.1714
4.1873
0.13724
3.0
3.2
3.3
3.6
3.8
4.2
4.6
4.8
5.2
5.4
5.7
0: 2
2
0.0962
5.1282
0.13000
3.8
4.0
4.1
4.5
4.7
5.1
5.6
5.9
6.3
6.5
6.9
0: 3
3
0.0402
5.8458
0.12619
4.4
4.6
4.7
5.1
5.4
5.8
6.4
6.7
7.2
7.4
7.8
0: 4
4
-0.0050
6.4237
0.12402
4.8
5.1
5.2
5.6
5.9
6.4
7.0
7.3
7.9
8.1
8.6
0: 5
5
-0.0430
6.8985
0.12274
5.2
5.5
5.6
6.1
6.4
6.9
7.5
7.8
8.4
8.7
9.2
0: 6
6
-0.0756
7.2970
0.12204
5.5
5.8
6.0
6.4
6.7
7.3
7.9
8.3
8.9
9.2
9.7
0: 7
7
-0.1039
7.6422
0.12178
5.8
6.1
6.3
6.7
7.0
7.6
8.3
8.7
9.4
9.6
10.2
0: 8
8
-0.1288
7.9487
0.12181
6.0
6.3
6.5
7.0
7.3
7.9
8.6
9.0
9.7
10.0
10.6
0: 9
9
-0.1507
8.2254
0.12199
6.2
6.6
6.8
7.3
7.6
8.2
8.9
9.3
10.1
10.4
11.0
0:10
10
-0.1700
8.4800
0.12223
6.4
6.8
7.0
7.5
7.8
8.5
9.2
9.6
10.4
10.7
11.3
0:11
11
-0.1872
8.7192
0.12247
6.6
7.0
7.2
7.7
8.0
8.7
9.5
9.9
10.7
11.0
11.7
1: 0
12
-0.2024
8.9481
0.12268
6.8
7.1
7.3
7.9
8.2
8.9
9.7
10.2
11.0
11.3
12.0

This can actually be done with two regular expression statements and two or three lines of code in R. First, the regular expressions:

If you did this correctly, you should end up with a text file like the one linked here.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221

Year: Month
Month
1st
3rd
5th
15th
25th
50th
75th
85th
95th
97th
99th
 
0
 
 
 
2.3
2.4
2.5
2.8
2.9
3.2
3.6
3.7
4.0
4.2
4.4
 
1
 
 
 
3.0
3.2
3.3
3.6
3.8
4.2
4.6
4.8
5.2
5.4
5.7
 
2
 
 
 
3.8
4.0
4.1
4.5
4.7
5.1
5.6
5.9
6.3
6.5
6.9
 
3
 
 
 
4.4
4.6
4.7
5.1
5.4
5.8
6.4
6.7
7.2
7.4
7.8
 
4
 
 
 
4.8
5.1
5.2
5.6
5.9
6.4
7.0
7.3
7.9
8.1
8.6
 
5
 
 
 
5.2
5.5
5.6
6.1
6.4
6.9
7.5
7.8
8.4
8.7
9.2
 
6
 
 
 
5.5
5.8
6.0
6.4
6.7
7.3
7.9
8.3
8.9
9.2
9.7
 
7
 
 
 
5.8
6.1
6.3
6.7
7.0
7.6
8.3
8.7
9.4
9.6
10.2
 
8
 
 
 
6.0
6.3
6.5
7.0
7.3
7.9
8.6
9.0
9.7
10.0
10.6
 
9
 
 
 
6.2
6.6
6.8
7.3
7.6
8.2
8.9
9.3
10.1
10.4
11.0
 
10
 
 
 
6.4
6.8
7.0
7.5
7.8
8.5
9.2
9.6
10.4
10.7
11.3
 
11
 
 
 
6.6
7.0
7.2
7.7
8.0
8.7
9.5
9.9
10.7
11.0
11.7
 
12
 
 
 
6.8
7.1
7.3
7.9
8.2
8.9
9.7
10.2
11.0
11.3
12.0

Do a “select all” and switch over to R. We’re going to first create a matrix with the values from line 15 to the end, and then create the column names for the matrix from lines 2 through 13. Do the following magic:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

> birthweight.percentiles = matrix(scan("clipboard", skip=14), ncol=12, byrow=T)
Read 156 items
> colnames(birthweight.percentiles) = scan("clipboard", what="character",
+                                          skip=1, n=12)
Read 12 items
> birthweight.percentiles
      Month 1st 3rd 5th 15th 25th 50th 75th 85th 95th 97th 99th
 [1,]     0 2.3 2.4 2.5  2.8  2.9  3.2  3.6  3.7  4.0  4.2  4.4
 [2,]     1 3.0 3.2 3.3  3.6  3.8  4.2  4.6  4.8  5.2  5.4  5.7
 [3,]     2 3.8 4.0 4.1  4.5  4.7  5.1  5.6  5.9  6.3  6.5  6.9
 [4,]     3 4.4 4.6 4.7  5.1  5.4  5.8  6.4  6.7  7.2  7.4  7.8
 [5,]     4 4.8 5.1 5.2  5.6  5.9  6.4  7.0  7.3  7.9  8.1  8.6
 [6,]     5 5.2 5.5 5.6  6.1  6.4  6.9  7.5  7.8  8.4  8.7  9.2
 [7,]     6 5.5 5.8 6.0  6.4  6.7  7.3  7.9  8.3  8.9  9.2  9.7
 [8,]     7 5.8 6.1 6.3  6.7  7.0  7.6  8.3  8.7  9.4  9.6 10.2
 [9,]     8 6.0 6.3 6.5  7.0  7.3  7.9  8.6  9.0  9.7 10.0 10.6
[10,]     9 6.2 6.6 6.8  7.3  7.6  8.2  8.9  9.3 10.1 10.4 11.0
[11,]    10 6.4 6.8 7.0  7.5  7.8  8.5  9.2  9.6 10.4 10.7 11.3
[12,]    11 6.6 7.0 7.2  7.7  8.0  8.7  9.5  9.9 10.7 11.0 11.7
[13,]    12 6.8 7.1 7.3  7.9  8.2  8.9  9.7 10.2 11.0 11.3 12.0
> # Optional: If you need or prefer a data frame instead of a matrix, run:
> # birthweight.percentiles = as.data.frame(birthweight.percentiles)

The first line of R code scans in the data values and fills them into a matrix with 12 columns, filling by row from left to right. The second line adds the column names. Notice how we were able to use “skip” and “n” to select the values we were interested in at each stage.

Example 4 – WordPress.com’s monthly stats table is ugly

Really it is! Here’s a screenshot. It looks pretty, right?

WordPress.com's monthly stats view. Pretty online, but not easy to copy and paste into other applications.

But, if you copy it into a an Excel spreadsheet, you get this:

Ugh. How are we supposed to work with this?

Well, it depends on if you copied from Excel to your text editor, or if you copied directly from the WordPress stats screen to a text editor. I’ll cover both scenarios below.

Copying from WordPress to Excel to your text editor

If you copied from WordPress to Excel to your text editor, you’d end up with a text file like the one linked here.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

24-Jan	25-Jan	26-Jan	27-Jan	28-Jan	29-Jan	30-Jan	201	29
 
 
19	27	14	32	73	25	11
 
31-Jan	1-Feb	2-Feb	3-Feb	4-Feb	5-Feb	6-Feb	302	43	50.25%
 
 
23	15	63	49	52	29	71
 
7-Feb	8-Feb	9-Feb	10-Feb	11-Feb	12-Feb	13-Feb	369	53	22.19%
 
 
59	35	24	88	29	96	38
 
14-Feb	15-Feb	16-Feb	17-Feb	18-Feb	19-Feb	20-Feb	376	54	1.90%
 
 
115	96	60	41	15	24	25
 
21-Feb	22-Feb	23-Feb	24-Feb	25-Feb	26-Feb	27-Feb	291	42	-22.61%
 
 
51	29	57	23	79	21	31
 
28-Feb	1-Mar	2-Mar	3-Mar	4-Mar			187	40	-3.18%
 
 
25	54	36	46	26

This is easy to clean up and import into R. You can simply search for ^([0-9]{1,2}-[A-Z].*), replace it with nothing, copy what’s left and read it into R by using something like > data = scan("clipboard").

Notice that we sacrifice the dates here though.

Copying from WordPress to your text editor

If you copied from WordPress to your text editor, you’d end up with a text file like the one linked here.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126

Jan 24
 
19
Jan 25
 
27
Jan 26
 
14
Jan 27
 
32
Jan 28
 
73
Jan 29
 
25
Jan 30
 
11
201	29
Jan 31
 
23
Feb 1
 
15
Feb 2
 
63
Feb 3
 
49
Feb 4
 
52
Feb 5
 
29
Feb 6
 
71
302	43	+50.25%
Feb 7
 
59
Feb 8
 
35
Feb 9
 
24
Feb 10
 
88
Feb 11
 
29
Feb 12
 
96
Feb 13
 
38
369	53	+22.19%
Feb 14
 
115
Feb 15
 
96
Feb 16
 
60
Feb 17
 
41
Feb 18
 
15
Feb 19
 
24
Feb 20
 
25
376	54	+1.90%
Feb 21
 
51
Feb 22
 
29
Feb 23
 
57
Feb 24
 
23
Feb 25
 
79
Feb 26
 
21
Feb 27
 
31
291	42	-22.61%
Feb 28
 
25
Mar 1
 
54
Mar 2
 
36
Mar 3
 
46
Mar 4
 
26
187	40	-3.18%

From this file, we can either search and replace in a way that we retain the dates or we drop them and keep just the stats.

Keep the dates

You can now easily copy this to your clipboard and read this into R using > data = read.table("clipboard", header=F). Don’t forget to add header=F or else R will think the first line is the column names.

Just gimme the data

Do everything you did before, but add one more regular expression search and replace. Search for ^([A-Z].*)\t(.*) and replace it with \2. This will create a list of just the data that can be read into R by using something like > data = scan("clipboard").

Example 5 – Merged cells are a pain….

Your friend gives you this PDF with a beautiful table in it, and you need to extract the data. When you copy it into Microsoft Word, OpenOffice Writer, or whatever you prefer, it looks terrible, like the unformatted text below:

1. Organization M (117) 1 Andhra Pradesh 7
2 Arunachal Pradesh 8
3 Assam 8
4 Bihar 24
5 Chattisgarh 2
6 Goa 15
7 Gujarat 19
8 Haryana 4
9 Himachal Pradesh 14
10 Jammu and Kashmir 2
11 Jharkhand 2
12 Karnataka 4
13 Kerala 2
14 Madhya Pradesh 2
15 Maharashtra 2
16 Manipur 2
2. Foundation X (69) 17 Meghalaya 29
18 Mizoram 10
19 Nagaland 4
20 Odisha 12
21 Puducherry 14
3. NGO Z (8) 22 Punjab 8
4. Government (16) 23 Rajasthan 16
5. Research Institute A (8) 24 Sikkim 4
25 Tamil Nadu 4
6. Organization C (36) 26 Tripura 8
27 Uttar Pradesh 15
28 Uttarakhand 1
29 West Bengal 12

This is actually also pretty easy to fix with the following set of regular expressions.

After doing these two steps, your data should now look like this:

	Organization M
1	Andhra Pradesh	7
2	Arunachal Pradesh	8
3	Assam	8
4	Bihar	24
5	Chattisgarh	2
6	Goa	15
7	Gujarat	19
8	Haryana	4
9	Himachal Pradesh	14
10	Jammu and Kashmir	2
11	Jharkhand	2
12	Karnataka	4
13	Kerala	2
14	Madhya Pradesh	2
15	Maharashtra	2
16	Manipur	2
	Foundation X
17	Meghalaya	29
18	Mizoram	10
19	Nagaland	4
20	Odisha	12
21	Puducherry	14
	NGO Z
22	Punjab	8
	Government
23	Rajasthan	16
	Research Institute A
24	Sikkim	4
25	Tamil Nadu	4
	Organization C
26	Tripura	8
27	Uttar Pradesh	15
28	Uttarakhand	1
29	West Bengal	12

This is now a tab delimited file, so it can easily be imported into any program that uses such files, or can be pasted into a word processor and converted into a table by using the “text to table” feature that any decent word processor should have.

Bonus session

Imagine (for whatever reason) you wanted the table from Example 5 to be used in R. Here’s how you can do it. First, count how many sites each organization is working in and make a note of that. In this case, it’s 16, 5, 1, 1, 2, 4. Then, do the following:

Copy whatever is in your text editor, switch over to R and enter the following:

1

> orgs = rep(scan("clipboard", what="character"), c(16, 5, 1, 1, 2, 4))

Switch back to your text editor, and revert the file to its original state (just do a couple of undos). Then, do what you did in Example 5 except:

• In the first step, instead of replacing with \t\2\t\r\n, replace with nothing.
• In the next step, instead of replacing with \1\t\2\t\3, replace with \2\t\3.

Again, copy whatever’s in your text file, switch back over to R, and do the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

> organizations = read.delim("clipboard", header=F)
> cbind(orgs, organizations)
                   orgs                V1 V2
1        Organization M    Andhra Pradesh  7
2        Organization M Arunachal Pradesh  8
3        Organization M             Assam  8
4        Organization M             Bihar 24
5        Organization M       Chattisgarh  2
6        Organization M               Goa 15
7        Organization M           Gujarat 19
8        Organization M           Haryana  4
9        Organization M  Himachal Pradesh 14
10       Organization M Jammu and Kashmir  2
11       Organization M         Jharkhand  2
12       Organization M         Karnataka  4
13       Organization M            Kerala  2
14       Organization M    Madhya Pradesh  2
15       Organization M       Maharashtra  2
16       Organization M           Manipur  2
17         Foundation X         Meghalaya 29
18         Foundation X           Mizoram 10
19         Foundation X          Nagaland  4
20         Foundation X            Odisha 12
21         Foundation X        Puducherry 14
22                NGO Z            Punjab  8
23           Government         Rajasthan 16
24 Research Institute A            Sikkim  4
25 Research Institute A        Tamil Nadu  4
26       Organization C           Tripura  8
27       Organization C     Uttar Pradesh 15
28       Organization C       Uttarakhand  1
29       Organization C       West Bengal 12

Whether or not we take a course in communication, there is considerable value in trying to refine our understanding of communication. To demonstrate, I will present two class exercises. In describing the exercises, hopefully some of the jargon common in the communications discipline (for example, encoding, decoding, channel, and congruence) will become clearer, and you will be at least a little more sensitive to trying to verify the effectiveness of your everyday communication approaches.

Exercise 1: Oral Instructions

The first exercise involved oral descriptions of drawings composed of basic geometric shapes. One student (we will call him the “sender”) was shown a simple drawing to describe to his classmates (the “receivers”) who were, in turn, supposed to try to recreate the drawing being described.

However, there were restrictions. There were encoding restrictions on the person giving the instructions: they could describe the image only by using basic terms relating to shapes, lengths, directions, distances, and so on, and they could not use gestures in their instructions. For instance, in the second source drawing, the sender could not say, “Draw a house sitting upon a mound…” but would have to say something like “Draw the top half of a circle. Above that semi-circle, draw a square…” and so on. In the decoding process by the receivers, the option to ask questions of the sender was removed. The channel was handicapped by imposing a five-minute time limit for the descriptions and the drawing.

As can be seen in Figure 1, the resulting drawings are quite varied, although in some cases (particularly the drawing of the cube with a six-pointed star on its face), they do come close to reproducing the source.

Figure 1: Three sets of drawings based on oral instructions. The first (outlined) drawing in each set is the source figure that the sender had to give instructions for. The remaining are the interpretation of the instructions by the receivers.

Exercise 2: Written Instructions

After a classroom discussion of the outcomes, one of the students strongly felt that things would be quite different if the instructions were given in written form. “By taking time to write the instructions,” he asserted, “we would be able to make sure that the instructions are clear enough that anyone reading it would be able to understand them.”

We decided to test his theory by writing instructions for drawing a cube with a six-pointed star on one face. Here are some of the instructions written by the students and the resulting drawings from various faculty and staff.

Drawing 1

(1) First, draw a square. (2) In the middle of the square, make a small triangle. (3) Make another triangle also in the middle of the square just opposite to the first triangle, but it is on the first triangle. (4) Above the square, draw a line parallel to the line of the square and join the sides and form a parallelogram. (5) From the left side, make another parallelogram with the help of one side of the square.

Drawing 2

(1) Draw a square (4 centimetre). (2) Make a six-point star inside square. (3)_Put the name of square ABCD. (4) Take line AB and draw two slanting lines of 1 centimetre from A and B leftward. (5) Give E above A; give F above B. (6) Match F and E. (7) Now take point A and D. Draw a slanting line of 1 centimetre upward. (8)_Give the name G. (9) Match F and G.

Drawing 3

Please make a diagram with the help of the following instructions. (1) First, make a square (which has all four sides at 90 degree angles). (2) Next, on the upper side and right side of this square, increase a line which will join. After joining, it should look like a cube, but only on upper and right sides. (3) Then, within the box, make a triangle which has 60 degree angles from both side lines; one line should be as a base line. (4) In the next step, make the same triangle but opposite on the first triangle, which should have five points.

What Went Wrong?

In the discussions following both exercises, there were a lot of attempts to explain why things did not go as expected. When discussing the experience giving oral instructions, some criticisms included the restrictions, which some argued were somewhat arbitrary. Yet, we deal with such restrictions regularly. Many of us, for instance, rely on text messages (or services like Twitter) for a considerable amount of our communication, and these messages are restricted to less than 200 characters. An organization preparing a television or radio advertisement has to work within a limited time frame, somewhere between 10 and 30 seconds. When writing a classified advertisement or a newspaper advertisement, you are likely to be restricted by the number of square inches of space your advertisement occupies.

In other instances, the receiver was simply confused. In the second drawing example for written instructions, the receiver got stuck trying to figure out how to draw a six-pointed star: around her paper there were many star scribbles, but they were all the more typical five-pointed star that we would be inclined to draw if someone had asked us to draw a star. In the end, she did not have enough time to complete her drawing. While the sender would have interpreted her message as being clear and direct, the receiver clearly needed some supplementary instruction.

There were also cases where the instructions were not congruent. In the third drawing example for written instructions, the sender tells us to draw two overlapping triangles, but then tells us that the resulting shape should have five points. Similarly, in the first drawing example, the receiver felt most confused by the fifth instruction. The fourth instruction did not specify which direction the parallelogram should be oriented. After she had already drawn what she interpreted as the fourth instruction, the fifth instruction did not seem to be in congruence with what she had already drawn.

It is also interesting to note that it is difficult to say whether people were right or wrong in what they drew. Most of the receivers would strongly assert that they drew according to their interpretation of the instructions that they were given. From that perspective, what they drew was correct. Many senders, on the other hand, did acknowledge that they had not tested their message to see whether it conveyed what they intended. A few of them also admitted to “cheating” a little by being close to the receiver and reinforcing the written statements by providing supplementary oral instructions.

What Can We Learn From This?

Perhaps the most important message from these experiences is that we should not assume that our communication is effective. Rather, we should try to build methods into our communication that try to confirm that the receiver has understood our message. (In teaching parlance, this is commonly referred to as a comprehension check.) Often, these are simple questions or brief activities presented to the audience to verify that their understanding of the message is correct.

These experiences also indicate that there is no single channel that is best for effective communication. Most likely, a combination of approaches would be most effective since different receivers have different learning or comprehension styles. Imagine, for instance, how much easier it would have been if we could have simply shown the audience the drawing and asked them to reproduce it. Or how helpful it would be if we could also use gestures along with our words to mimic the act of drawing, thus also helping the receiver to create an accurate drawing.

Keeping these lessons in mind, we should also try to identify what type of communicator we are. For some, their strengths will be in written communication; others might prefer speaking, demonstrating, illustrating, or employing one of the many communication reinforcement tools available to us. Once you have figured that out, practice—both to keep your strengths strong, and to improve upon your weaknesses.

Cross-posted at the Tata-Dhan Academy PDM blog and at Ananda Mahto.