Skip to content

Instantly share code, notes, and snippets.

@djvanderlaan

djvanderlaan/#codelistdemo.md

Last active March 15, 2024 10:22
Show Gist options
  • Save djvanderlaan/f898bd8b4416dfe6157a7c45c616eecb to your computer and use it in GitHub Desktop.
Save djvanderlaan/f898bd8b4416dfe6157a7c45c616eecb to your computer and use it in GitHub Desktop.
Download ZIP
Working with codelists demo for datapackage
Raw
#codelistdemo.md

We have the Iris dataset saved into a data package. There are two resources: the data and the code list for the ‘Species’ variable:

> library(datapackage)
> dp <- opendatapackage("iris")
> dp
[iris] 

Location: <./>
Resources:
[iris] The Iris Dataset
[Species-codelist] Iris Species Codes

Get the data. We have the numeric codes in the ‘Species’ field:

> iris <- dpresource(dp, "iris") |> dpgetdata()
> head(iris)
  Sepal.Length Sepal.Width Petal.Length Petal.Width Species
1          5.1         3.5          1.4         0.2       1
2          4.9         3.0          1.4         0.2       1
3          4.7         3.2          1.3         0.2       1
4          4.6         3.1          1.5         0.2       1
5          5.0         3.6          1.4         0.2       1
6          5.4         3.9          1.7         0.4       1

We can see there is a codelist associated with ‘Species’

> iris$Species
  [1] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 [38] 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
 [75] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3
[112] 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
[149] 3 3
attr(,"fielddescriptor")
Field Descriptor:
name    :"Species"
type    :"integer"
codelist:"Species-codelist"

And also get that codelist. The field descriptor info in the column has a link back to the datapackage

> dpcodelist(iris$Species)
  code      label
1    1     setosa
2    2 versicolor
3    3  virginica

We can use this codelist to convert the field from its numeric values to factor:

> dptofactor(iris$Species) |> head()
[1] setosa setosa setosa setosa setosa setosa
Levels: setosa versicolor virginica

We can also specify that all fields with an associated codelist should be converted to factor:

> dpresource(dp, "iris") |> dpgetdata(to_factor = TRUE) |> head()
  Sepal.Length Sepal.Width Petal.Length Petal.Width Species
1          5.1         3.5          1.4         0.2  setosa
2          4.9         3.0          1.4         0.2  setosa
3          4.7         3.2          1.3         0.2  setosa
4          4.6         3.1          1.5         0.2  setosa
5          5.0         3.6          1.4         0.2  setosa
6          5.4         3.9          1.7         0.4  setosa

One annoying little thing in R is in some operations remove attributes. For example when subsetting, the link to the datapackage and codelist is lost and the following no longer works:

> tmp <- iris[c(1, 51, 101), ]
> dptofactor(tmp$Species)
Warning:  Field does not have an associated code list. Returning original vector. 
[1] 1 2 3

In that case we have to get the codelist ourselves when we want to convert to factor:

> codel <- dp |> dpresource("iris") |> dpfield("Species") |> dpcodelist()
> # or 
> # codel <- dp |> dpresource("Species-codelist") |> dpgetdata()
> dptofactor(tmp$Species, codelist = codel)
[1] setosa     versicolor virginica 
Levels: setosa versicolor virginica

And even if we don’t have any facilities for automatically use the codelist (.e.g. just the facilities offered by version 1 of the datapackage spec), we can use existing functionality to see what the codelist is, read the codelist and apply the codelist:

> dp |> dpresource("iris") |> dpfield("Species")
Field Descriptor:
name    :"Species"
type    :"integer"
codelist:"Species-codelist"
> codel <- dp |> dpresource("Species-codelist") |> dpgetdata()
> tmp$Species <- factor(tmp$Species, levels = codel[[1]], labels = codel[[2]])
> tmp
    Sepal.Length Sepal.Width Petal.Length Petal.Width    Species
1            5.1         3.5          1.4         0.2     setosa
51           7.0         3.2          4.7         1.4 versicolor
101          6.3         3.3          6.0         2.5  virginica
Raw
datapackage.json
{
"name": "iris",
"resources": [
{
"name": "iris",
"title": "The Iris Dataset",
"format": "csv",
"mediatype": "text/csv",
"path": "iris.csv",
"encoding": "utf-8",
"schema": {
"fields": [
{
"name": "Sepal.Length",
"type": "number"
},
{
"name": "Sepal.Width",
"type": "number"
},
{
"name": "Petal.Length",
"type": "number"
},
{
"name": "Petal.Width",
"type": "number"
},
{
"name": "Species",
"type": "integer",
"codelist": "Species-codelist"
}
]
}
},
{
"name": "Species-codelist",
"title": "Iris Species Codes",
"format": "csv",
"mediatype": "text/csv",
"path": "Species-codelist.csv",
"encoding": "utf-8",
"schema": {
"fields": [
{
"name": "code",
"type": "integer"
},
{
"name": "label",
"type": "string"
}
]
}
}
]
}
Raw
iris.csv
Sepal.Length Sepal.Width Petal.Length Petal.Width Species
5.1 3.5 1.4 0.2 1
4.9 3 1.4 0.2 1
4.7 3.2 1.3 0.2 1
4.6 3.1 1.5 0.2 1
5 3.6 1.4 0.2 1
5.4 3.9 1.7 0.4 1
4.6 3.4 1.4 0.3 1
5 3.4 1.5 0.2 1
4.4 2.9 1.4 0.2 1
4.9 3.1 1.5 0.1 1
5.4 3.7 1.5 0.2 1
4.8 3.4 1.6 0.2 1
4.8 3 1.4 0.1 1
4.3 3 1.1 0.1 1
5.8 4 1.2 0.2 1
5.7 4.4 1.5 0.4 1
5.4 3.9 1.3 0.4 1
5.1 3.5 1.4 0.3 1
5.7 3.8 1.7 0.3 1
5.1 3.8 1.5 0.3 1
5.4 3.4 1.7 0.2 1
5.1 3.7 1.5 0.4 1
4.6 3.6 1 0.2 1
5.1 3.3 1.7 0.5 1
4.8 3.4 1.9 0.2 1
5 3 1.6 0.2 1
5 3.4 1.6 0.4 1
5.2 3.5 1.5 0.2 1
5.2 3.4 1.4 0.2 1
4.7 3.2 1.6 0.2 1
4.8 3.1 1.6 0.2 1
5.4 3.4 1.5 0.4 1
5.2 4.1 1.5 0.1 1
5.5 4.2 1.4 0.2 1
4.9 3.1 1.5 0.2 1
5 3.2 1.2 0.2 1
5.5 3.5 1.3 0.2 1
4.9 3.6 1.4 0.1 1
4.4 3 1.3 0.2 1
5.1 3.4 1.5 0.2 1
5 3.5 1.3 0.3 1
4.5 2.3 1.3 0.3 1
4.4 3.2 1.3 0.2 1
5 3.5 1.6 0.6 1
5.1 3.8 1.9 0.4 1
4.8 3 1.4 0.3 1
5.1 3.8 1.6 0.2 1
4.6 3.2 1.4 0.2 1
5.3 3.7 1.5 0.2 1
5 3.3 1.4 0.2 1
7 3.2 4.7 1.4 2
6.4 3.2 4.5 1.5 2
6.9 3.1 4.9 1.5 2
5.5 2.3 4 1.3 2
6.5 2.8 4.6 1.5 2
5.7 2.8 4.5 1.3 2
6.3 3.3 4.7 1.6 2
4.9 2.4 3.3 1 2
6.6 2.9 4.6 1.3 2
5.2 2.7 3.9 1.4 2
5 2 3.5 1 2
5.9 3 4.2 1.5 2
6 2.2 4 1 2
6.1 2.9 4.7 1.4 2
5.6 2.9 3.6 1.3 2
6.7 3.1 4.4 1.4 2
5.6 3 4.5 1.5 2
5.8 2.7 4.1 1 2
6.2 2.2 4.5 1.5 2
5.6 2.5 3.9 1.1 2
5.9 3.2 4.8 1.8 2
6.1 2.8 4 1.3 2
6.3 2.5 4.9 1.5 2
6.1 2.8 4.7 1.2 2
6.4 2.9 4.3 1.3 2
6.6 3 4.4 1.4 2
6.8 2.8 4.8 1.4 2
6.7 3 5 1.7 2
6 2.9 4.5 1.5 2
5.7 2.6 3.5 1 2
5.5 2.4 3.8 1.1 2
5.5 2.4 3.7 1 2
5.8 2.7 3.9 1.2 2
6 2.7 5.1 1.6 2
5.4 3 4.5 1.5 2
6 3.4 4.5 1.6 2
6.7 3.1 4.7 1.5 2
6.3 2.3 4.4 1.3 2
5.6 3 4.1 1.3 2
5.5 2.5 4 1.3 2
5.5 2.6 4.4 1.2 2
6.1 3 4.6 1.4 2
5.8 2.6 4 1.2 2
5 2.3 3.3 1 2
5.6 2.7 4.2 1.3 2
5.7 3 4.2 1.2 2
5.7 2.9 4.2 1.3 2
6.2 2.9 4.3 1.3 2
5.1 2.5 3 1.1 2
5.7 2.8 4.1 1.3 2
6.3 3.3 6 2.5 3
5.8 2.7 5.1 1.9 3
7.1 3 5.9 2.1 3
6.3 2.9 5.6 1.8 3
6.5 3 5.8 2.2 3
7.6 3 6.6 2.1 3
4.9 2.5 4.5 1.7 3
7.3 2.9 6.3 1.8 3
6.7 2.5 5.8 1.8 3
7.2 3.6 6.1 2.5 3
6.5 3.2 5.1 2 3
6.4 2.7 5.3 1.9 3
6.8 3 5.5 2.1 3
5.7 2.5 5 2 3
5.8 2.8 5.1 2.4 3
6.4 3.2 5.3 2.3 3
6.5 3 5.5 1.8 3
7.7 3.8 6.7 2.2 3
7.7 2.6 6.9 2.3 3
6 2.2 5 1.5 3
6.9 3.2 5.7 2.3 3
5.6 2.8 4.9 2 3
7.7 2.8 6.7 2 3
6.3 2.7 4.9 1.8 3
6.7 3.3 5.7 2.1 3
7.2 3.2 6 1.8 3
6.2 2.8 4.8 1.8 3
6.1 3 4.9 1.8 3
6.4 2.8 5.6 2.1 3
7.2 3 5.8 1.6 3
7.4 2.8 6.1 1.9 3
7.9 3.8 6.4 2 3
6.4 2.8 5.6 2.2 3
6.3 2.8 5.1 1.5 3
6.1 2.6 5.6 1.4 3
7.7 3 6.1 2.3 3
6.3 3.4 5.6 2.4 3
6.4 3.1 5.5 1.8 3
6 3 4.8 1.8 3
6.9 3.1 5.4 2.1 3
6.7 3.1 5.6 2.4 3
6.9 3.1 5.1 2.3 3
5.8 2.7 5.1 1.9 3
6.8 3.2 5.9 2.3 3
6.7 3.3 5.7 2.5 3
6.7 3 5.2 2.3 3
6.3 2.5 5 1.9 3
6.5 3 5.2 2 3
6.2 3.4 5.4 2.3 3
5.9 3 5.1 1.8 3
Raw
Species-codelist.csv
code label
1 setosa
2 versicolor
3 virginica
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment