Skip to content

Instantly share code, notes, and snippets.

@miku

miku/Golab.md Secret

Last active March 11, 2019 23:26
Show Gist options
  • Save miku/39e4273d15abfd7e4297071338da3349 to your computer and use it in GitHub Desktop.
Save miku/39e4273d15abfd7e4297071338da3349 to your computer and use it in GitHub Desktop.
Download ZIP
Turn data into Go structs - Martin Czygan
Raw
Golab.md

Turn data into Go structs

Tools

Go encourages tools: gofmt, gorename, stringer and many more.

Go uses struct tags

Go offers struct tags which are discoverable via reflection. These enjoy a wide range of use in the standard library in the JSON/XML and other encoding packages.

JSON and XML

When working with data, JSON and XML are common serialization formats.

The /ˈtiːdɪəm/

Have you ever written JSON or XML struct tags by hand before?

I did. It is not hard, but can become tedious and error-prone, if the data structures become large.

Automation

Can we automate the step that takes us from raw JSON and XML data to a nice Go struct?

Yes, there are various options.

JSONGen

JSONGen is a tool for generating native Golang types from JSON objects.

Reduce effort by 99%

  • Replace 3208 keystrokes with 30.
$ curl -sL https://is.gd/xG6wk4 | JSONGen
$ type _ struct {
	Message struct {
		AlternativeId  []string `json:"alternative-id"`
		ContainerTitle []string `json:"container-title"`
		ContentDomain  struct {
			CrossmarkRestriction bool          `json:"crossmark-restriction"`
			Domain               []interface{} `json:"domain"`
		} `json:"content-domain"`
		Created struct {
			DateParts []int64 `json:"date-parts"`
			DateTime  string  `json:"date-time"`
			Timestamp int64   `json:"timestamp"`
		} `json:"created"`
		DOI       string
		Deposited struct {
			DateParts []int64 `json:"date-parts"`
			DateTime  string  `json:"date-time"`
			Timestamp int64   `json:"timestamp"`
		} `json:"deposited"`
		ISSN    []string
		Indexed struct {
			DateParts []int64 `json:"date-parts"`
			DateTime  string  `json:"date-time"`
			Timestamp int64   `json:"timestamp"`
		} `json:"indexed"`
		IsReferencedByCount int64 `json:"is-referenced-by-count"`
		IssnType            []struct {
			Type  string `json:"type"`
			Value string `json:"value"`
		} `json:"issn-type"`
		Issue  string `json:"issue"`
		Issued struct {
			DateParts []int64 `json:"date-parts"`
		} `json:"issued"`
		JournalIssue struct {
			Issue           string `json:"issue"`
			PublishedOnline struct {
				DateParts []int64 `json:"date-parts"`
			} `json:"published-online"`
			PublishedPrint struct {
				DateParts []int64 `json:"date-parts"`
			} `json:"published-print"`
		} `json:"journal-issue"`
		Language string `json:"language"`
		License  []struct {
			ContentVersion string `json:"content-version"`
			DelayInDays    int64  `json:"delay-in-days"`
			Start          struct {
				DateParts []int64 `json:"date-parts"`
				DateTime  string  `json:"date-time"`
				Timestamp int64   `json:"timestamp"`
			} `json:"start"`
			URL string
		} `json:"license"`
		Link []struct {
			ContentType         string `json:"content-type"`
			ContentVersion      string `json:"content-version"`
			IntendedApplication string `json:"intended-application"`
			URL                 string
		} `json:"link"`
		Member          string        `json:"member"`
		OriginalTitle   []interface{} `json:"original-title"`
		Page            string        `json:"page"`
		Prefix          string        `json:"prefix"`
		PublishedOnline struct {
			DateParts []int64 `json:"date-parts"`
		} `json:"published-online"`
		PublishedPrint struct {
			DateParts []int64 `json:"date-parts"`
		} `json:"published-print"`
		Publisher       string `json:"publisher"`
		ReferenceCount  int64  `json:"reference-count"`
		ReferencesCount int64  `json:"references-count"`
		Relation        struct {
		} `json:"relation"`
		Score               float64       `json:"score"`
		ShortContainerTitle []string      `json:"short-container-title"`
		ShortTitle          []interface{} `json:"short-title"`
		Source              string        `json:"source"`
		Subtitle            []interface{} `json:"subtitle"`
		Title               []string      `json:"title"`
		Type                string        `json:"type"`
		URL                 string
		Volume              string `json:"volume"`
	} `json:"message"`
	MessageType    string `json:"message-type"`
	MessageVersion string `json:"message-version"`
	Status         string `json:"status"`
}

XML

I was searching for something similar for XML, which is more complex than JSON. There are:

Mapping between XML elements and data structures is inherently flawed: an XML element is an order-dependent collection of anonymous values, while a data structure is an order-independent collection of named values.

XMLGen

Similar to JSONGen, very easy to use.

$ curl -s http://www.ibiblio.org/xml/examples/shakespeare/all_well.xml | XMLGen
type _ struct {
	PLAY struct {
		TITLE string `xml:"TITLE"`
		FM    struct {
			P []string `xml:"P"`
		} `xml:"FM"`
		PERSONAE struct {
			TITLE   string   `xml:"TITLE"`
			PERSONA []string `xml:"PERSONA"`
			PGROUP  struct {
				PERSONA  []string `xml:"PERSONA"`
				GRPDESCR string   `xml:"GRPDESCR"`
			} `xml:"PGROUP"`
			PERSONA []string `xml:"PERSONA"`
			PGROUP  struct {
				PERSONA  []string `xml:"PERSONA"`
				GRPDESCR string   `xml:"GRPDESCR"`
			} `xml:"PGROUP"`
			PERSONA string `xml:"PERSONA"`
		} `xml:"PERSONAE"`
		SCNDESCR string `xml:"SCNDESCR"`
		PLAYSUBT string `xml:"PLAYSUBT"`
		ACT      []struct {
			TITLE string `xml:"TITLE"`
			SCENE []struct {
				TITLE    string `xml:"TITLE"`
				STAGEDIR string `xml:"STAGEDIR"`
				SPEECH   []struct {
					SPEAKER  string `xml:"SPEAKER"`
					LINE     string `xml:"LINE"`
					STAGEDIR string `xml:"STAGEDIR"`
				} `xml:"SPEECH"`
				STAGEDIR string `xml:"STAGEDIR"`
				SPEECH   []struct {
					SPEAKER  string   `xml:"SPEAKER"`
					LINE     []string `xml:"LINE"`
					STAGEDIR string   `xml:"STAGEDIR"`
				} `xml:"SPEECH"`
				STAGEDIR string `xml:"STAGEDIR"`
				SPEECH   []struct {
					SPEAKER  string   `xml:"SPEAKER"`
					LINE     []string `xml:"LINE"`
					STAGEDIR []string `xml:"STAGEDIR"`
					LINE     []string `xml:"LINE"`
				} `xml:"SPEECH"`
				STAGEDIR string `xml:"STAGEDIR"`
				SPEECH   struct {
					SPEAKER  string `xml:"SPEAKER"`
					LINE     string `xml:"LINE"`
					STAGEDIR string `xml:"STAGEDIR"`
				} `xml:"SPEECH"`
				STAGEDIR string `xml:"STAGEDIR"`
				SPEECH   []struct {
					SPEAKER  string   `xml:"SPEAKER"`
					LINE     []string `xml:"LINE"`
					STAGEDIR string   `xml:"STAGEDIR"`
				} `xml:"SPEECH"`
				STAGEDIR string `xml:"STAGEDIR"`
				SPEECH   struct {
					SPEAKER  string   `xml:"SPEAKER"`
					LINE     []string `xml:"LINE"`
					STAGEDIR string   `xml:"STAGEDIR"`
				} `xml:"SPEECH"`
				STAGEDIR string `xml:"STAGEDIR"`
			} `xml:"SCENE"`
			EPILOGUE struct {
				TITLE  string `xml:"TITLE"`
				SPEECH struct {
					SPEAKER string   `xml:"SPEAKER"`
					LINE    []string `xml:"LINE"`
				} `xml:"SPEECH"`
				STAGEDIR string `xml:"STAGEDIR"`
			} `xml:"EPILOGUE"`
		} `xml:"ACT"`
	} `xml:"PLAY"`
}

zek

While in XML, order can matter, it does not necessary matter.

Plus:

  • example
  • can work with multiple files
$ curl -s http://www.ibiblio.org/xml/examples/shakespeare/all_well.xml | zek -e -c
// PLAY was generated 2018-10-21 23:22:49 by tir on sol.
type PLAY struct {
        XMLName xml.Name `xml:"PLAY"`
        Text    string   `xml:",chardata"`
        TITLE   string   `xml:"TITLE"` // All's Well That Ends Well...
        FM      struct {
                Text string   `xml:",chardata"`
                P    []string `xml:"P"` // Text placed in the public...
        } `xml:"FM"`
        PERSONAE struct {
                Text    string   `xml:",chardata"`
                TITLE   string   `xml:"TITLE"`   // Dramatis Personae
                PERSONA []string `xml:"PERSONA"` // KING OF FRANCE, DUKE OF F...
                PGROUP  []struct {
                        Text     string   `xml:",chardata"`
                        PERSONA  []string `xml:"PERSONA"`  // Steward, Clown, VIOLENTA,...
                        GRPDESCR string   `xml:"GRPDESCR"` // servants to the Countess ...
                } `xml:"PGROUP"`
        } `xml:"PERSONAE"`
        SCNDESCR string `xml:"SCNDESCR"` // SCENE  Rousillon; Paris; ...
        PLAYSUBT string `xml:"PLAYSUBT"` // ALL'S WELL THAT ENDS WELL...
        ACT      []struct {
                Text  string `xml:",chardata"`
                TITLE string `xml:"TITLE"` // ACT I, ACT II, ACT III, A...
                SCENE []struct {
                        Text     string   `xml:",chardata"`
                        TITLE    string   `xml:"TITLE"`    // SCENE I.  Rousillon. The ...
                        STAGEDIR []string `xml:"STAGEDIR"` // Enter BERTRAM, the COUNTE...
                        SPEECH   []struct {
                                Text    string `xml:",chardata"`
                                SPEAKER string `xml:"SPEAKER"` // COUNTESS, BERTRAM, LAFEU,...
                                LINE    []struct {
                                        Text     string `xml:",chardata"` // In delivering my son from...
                                        STAGEDIR string `xml:"STAGEDIR"`  // To HELENA, To BERTRAM, Kn...
                                } `xml:"LINE"`
                                STAGEDIR []string `xml:"STAGEDIR"` // Enter PAROLLES, Aside, Ex...
                        } `xml:"SPEECH"`
                } `xml:"SCENE"`
                EPILOGUE struct {
                        Text   string `xml:",chardata"`
                        TITLE  string `xml:"TITLE"` // EPILOGUE
                        SPEECH struct {
                                Text    string   `xml:",chardata"`
                                SPEAKER string   `xml:"SPEAKER"` // KING
                                LINE    []string `xml:"LINE"`    // The king's a beggar, now ...
                        } `xml:"SPEECH"`
                        STAGEDIR string `xml:"STAGEDIR"` // Exeunt
                } `xml:"EPILOGUE"`
        } `xml:"ACT"`
}

Tools FTW

Before you write a struct tag for existing JSON or XML data manually, you might want to try one of these utilities first.

Thanks

Display the source blob
Display the rendered blob
Raw
Golab_2018_LightningTalk_MartinCzygan.pdf
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment