/*
cls
cd "C:/Users/sttp/Documents/nhb/STATA/StataHacks/docs/regex"
do2md using regular_expressions, css("../../css/nhb.css") //show
*/
/***
---
title: Regular expressions in Stata
...


# Regular expressions in Stata


## Introduction

Regular expressions are a relatively easy, flexible method of searching strings.
You can use them to search any string (e.g. variables, macros).

In Stata, there are three functions that use regular expressions.

Regular expressions can be very effective in cleansing string data.


## Regular expression functions in Stata

Stata has the following regular expression functions:

* **regexm(s, re)** performs a match of a regular expression and evaluates to 
  1 if regular expression re (a string) is satisfied by the string s, 
  otherwise returns 0

* **regexr(s1,re,s2)** replaces the first substring within s1 that matches re 
  with s2 and returns the resulting string.  If s1 contains no substring that 
  matches re, the unaltered s1 is returned.

* **regexs(n)** returns subexpression n from a previous regexm() match, 
  where 0 < n < 10. Subexpression 0 is reserved for the entire string that 
  satisfied the regular expression.

So a regular expression is a string which is used as a filter for another string.


## Regular expressions rules

A regular expression is a string working as a string filter. The filter is based
on a set of characters: 

* The dash operator (-) in a-z  means "match a range of characters or numbers".
	The "a" and "z" are merely an example. 
	It could also be 0-9, 5-8, F-M, etc.

* Period (.) means "match any character"

* A backslash (/) is used as an escape character to match characters that 
	would otherwise be interpreted as a regular-expression operator

* The pipe character (|) signifies a logical "or" that is often used in 
  character sets (see square brackets just below)

* Square brackets ([]) denote a set of allowable characters/expressions to 
  use in matching, such as [a-zA-Z0-9] for all alphanumeric characters
  
The function =regexm(string, "[0-9]") below evaluates string acordingly to the 
regular expression "[0-9]", that says that a ciffer (from 0 to 9) is present in 
the text.
***/
//OFF
local test11 regexm("a", "[0-9]")
local test12 regexm("abc", "[0-9]")
local test13 regexm("4", "[0-9]")
local test14 regexm("4 ", "[0-9]")
local test15 regexm("44", "[0-9]")
//ON
/****/display `"* `test11' gives `=`test11'' since a character is not a ciffer"'
/****/display `"* `test12' gives `=`test12'' since text is not a ciffer"'
/****/display `"* `test13' gives `=`test13'' since 4 is a ciffer"'
/****/display `"* `test14' gives `=`test14'' since 4 + space contains a ciffer"'
/****/display `"* `test15' gives `=`test15'' since 44 has 2 ciffers"'
/***
More special regular expression characters are:

* Asterisk (*) means "match zero or more" of the preceding expression

* Plus sign (+) means "match one or more" of the preceding expression

* When caret (^) placed at the beginning of a regular expression, the caret 
  means "match expression at beginning of string". This character 
  can be thought of as an "anchor" character since it does not 
  directly match a character, only the location of the match

* When the dollar sign ($) is placed at the end of a regular expression, 
  it means "match expression at end of string". This is the other 
  anchor character

***/
//OFF
local test regexm("a", "^[0-9]+$")
local test regexm("abc", "^[0-9]+$")
local test regexm("4", "^[0-9]+$")
local test regexm("4 ", "^[0-9]+$")
local test regexm("44", "^[0-9]+$")
local test regexm("123.44", "^[0-9]+$")
//ON
/****/display `"* `test' gives `=`test'' since a character is not a ciffer"'
/****/display `"* `test' gives `=`test'' since text is not a ciffer"'
/****/display `"* `test' gives `=`test'' since 4 is a ciffer"'
/****/display `"* `test' gives `=`test'' since 4 + space is not one or more ciffers"'
/****/display `"* `test' gives `=`test'' since 44 has 2 ciffers"'
/****/display `"* `test' gives `=`test'' since dot "." is not a ciffer"'
/***
# Examples

## Howto test a regular expression in Stata

It is important to test regular expressions before full scale usage.

The easiest way to do so is to use the command -display-:
***/
display =regexm("This test will return a 1", "t[eo]")
display =regexm("This will return a 0", "t[eo]")
/***
##Howto use regexm and regexs to generate a grouping variable

We use the auto data:
***/
/**/sysuse auto, clear
/***
List the make of cars containing either of the strings Datsun, Pont or Toyota
***/
list make if regexm(make, "Datsun|Pont|Toyota")
/***
Define a grouping variable for the strings Datsun, Pont or Toyota.
Note that what is in soft brackets () can be extrated by the function regexs 
with a integer between 1 and 9 as argument:
***/
/**/generate grp = regexs(1) if regexm(make, "(Datsun|Pont|Toyota)")
/***
And the result is:
***/
list make grp if regexm(make, "Datsun|Pont|Toyota")
/***
The variable grp is set to missing if the make does not match one of the 3 
strings Datsun, Pont, or Toyota:
***/
codebook grp
/***
## Howto standardise strings by using the regex replace function regexr

***/
//OFF
clear
input str20 name age str6 sex
name       age      sex
"nh Bruun" 52 male
"Henrik Bruun" 52 male
"henrik Bruun" 52 male
end
//ON
/***
All variants of first name below must must be change to Niels Henrik
***/
list
/***
The solution:
***/
/**/replace name = regexr(name, "nh|[H|h]enrik", "Niels Henrik")  
/***
And the new data are:
***/
list
//OFF
clear
input str20 name age str6 sex
name       age      sex
"nh Bruun" 52 male
"Henrik Bruun" 52 male
"henrik Bruun" 52 male
end
//ON
/***
PS space matters!!!
***/
replace name = regexr(name, "nh | [H|h]enrik", "Niels Henrik")  
/***
And now the changes are:
***/
list
/***
## Grouping strings, education at Denmark Statistics, into years of education
***/
//OFF
clear
input str8 AFSP4E
0C525000
1C525000
2C525000
3A525000
3B525000
3C525000
4A525000
4B525000
4C525000
5A525000
5B525000
end
//ON
/***
The variable AFSP4E must be transformed into another variable edu_time 
by the rules:

* If AFSP4E starts with 0, 1, 2 it must be "<=10 year"
* If AFSP4E starts with 3, 4 or 5B it must be ">10 year & <=15 year"
* If AFSP4E starts with 5A it must be ">15 year"

Here are some example values:
***/
list
/***
And the code could be (capture added due to Stata version 12):
***/
/**/capture generate str edu_time = "<=10 year" * regexm(AFSP4E, "^[0-2]") ///
				+ ">10 year & <=15 year" * regexm(AFSP4E, "^[3-4]|^5B") ///
				+ ">15 year" * regexm(AFSP4E, "^5A")
/***
And the result:
***/
list
/**/capture drop edu_time
/***
Another version of the code could be:
***/
/**/generate edu_time = 1 * regexm(AFSP4E, "^[0-2]") ///
						+ 2 * regexm(AFSP4E, "^[3-4]|^5B") ///
						+ 3 * regexm(AFSP4E, "^5A")
/***
combined with:
***/
/**/label define edu_time 1 "<=10 year" 2 ">10 year & <=15 year" 3 ">15 year"
/**/label values edu_time edu_time
/***
And the result is the same (almost):
***/
list
/***
Now the variable test is labeled number just like it is prefered in Stata.

## Grouping numbers, social group at Denmark Statistics
***/
//OFF
clear
input SOCIO02
111
112
113
114
115
118
131
132
133
134
135
139
120
310
210
220
321
330
410
end
//ON
/***
Now assume that a number variable SOCIO02 has to be grouped into a new variable 
employment by the 3 leading digits. The grouping is:

	1. 111 112 113 114 120 131 132 133 134 135 139 310
	2. 210 220 321 330
	3. 410

First thing is that the variable must be a string variable in order to handled
by regexm.

Second the note that "^111-114" is not a regex for 111, 112, 113 and 114.
This is how one would formulate it for numbers, but we handle strings here.
So to get the string headings 111, 112, 113 and 114 note that they all start 
with "11" and are followed by 1, 2, 3, or 4 or in regex "^11[1-4]".

Thirdly note that the regex "^11[1-38]" selects strings starting with 111, 112, 
113 and 118 since square brackets means 1-3 or 8 and 1-3 means 1, 2 or 3.

A solution is shown below:
***/
/**/generate employment = 1 if regexm(string(SOCIO02), "^11[1-4]|^120|^13[1-59]|^310") 
/**/replace employment = 2 if regexm(string(SOCIO02), "^210|^220|^321|^330")
/**/replace employment = 3 if regexm(string(SOCIO02), "^410")
/***
And the result is:
***/
/**/sort employment SOCIO02
list employment SOCIO02
/***
## Getting the birthday from a danish social security number and more on testing

A danish social security number consist of 10 digits. The first 2 digits are 
day of birth, the next 2 digits are month of birth and the next 2 are the last
2 digits in the year of birth.

First generate a sample set to test the regular expressions:
***/
clear
input str10 dksecnum
2305123456
1210728998
121223
end
list
/***
To get birth dates from dksecnum (when it has proper values, ie 10 digits) 
simply do:
***/
/**/generate bday = mdy(real(regexs(2)), real(regexs(1)), 1900 + real(regexs(3))) ///
	if regexm(dksecnum, "^([0-9][0-9])([0-9][0-9])([0-9][0-9])[0-9][0-9][0-9][0-9]")
/**/format %tdCCYY-NN-DD bday
list
/***
## References

1. [Stata: What are regular expressions and how can I use them in Stata?](http://www.stata.com/support/faqs/data-management/regular-expressions/)
2. [UCLA: How can I extract a portion of a string variable using regular expressions?](http://www.ats.ucla.edu/stat/stata/faq/regex.htm)
3. [Rose Anne Medeiros: Using regular expressions for data management in Stata](http://repec.org/wcsug2007/medeiros_reg_ex.pdf)
***/