Skip to content

Instantly share code, notes, and snippets.

@hermanbanken
Last active June 6, 2021 10:45
Show Gist options
  • Select an option

  • Save hermanbanken/f681592bce93f0d912be7726ddac8162 to your computer and use it in GitHub Desktop.

Select an option

Save hermanbanken/f681592bce93f0d912be7726ddac8162 to your computer and use it in GitHub Desktop.
Protobuf ramblings
[submodule "proto/googleapis"]
path = proto/vendor/googleapis
url = https://github.com/googleapis/googleapis.git
{
dependencies: {
+ "grpc-tools": "^1.9.1",
+ "grpc_tools_node_protoc_ts": "^4.1.3",
}
}

Protobuf

Example in Envoy control plane: github.com/envoyproxy/go-control-plane/envoy/type/http.pb.go. Protobuf outputs with some details about what generated it:

// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// 	protoc-gen-go v1.22.0
// 	protoc        v3.10.1
// source: envoy/type/http.proto

package envoy_type

Using correct protobuf repositories ("google.golang.org/protobuf").

import (
	_ "github.com/cncf/udpa/go/udpa/annotations"
	proto "github.com/golang/protobuf/proto"
	protoreflect "google.golang.org/protobuf/reflect/protoreflect"
	protoimpl "google.golang.org/protobuf/runtime/protoimpl"
	reflect "reflect"
)

With some validation of Protobuf versions baked in:

const (
	// Verify that this generated code is sufficiently up-to-date.
	_ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
	// Verify that runtime/protoimpl is sufficiently up-to-date.
	_ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
)

Dynamic fields

This is supported using Google's Any type: https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/any.proto

Example of packing this up:

package google.profile;
message Person {
  string first_name = 1;
  string last_name = 2;
}
foo := &pb.Person{...}
any, err := anypb.New(foo)
if err != nil {
  ...
}
...
foo := &pb.Person{}
if err := any.UnmarshalTo(foo); err != nil {
  ...
}

Example JSON representation:

{
  "@type": "type.googleapis.com/google.profile.Person",
  "firstName": <string>,
  "lastName": <string>
}

Javascript and Typescript client code generation

@google-cloud/datastore ships with a /build/protos file including .{js,d.ts} files and a /build/protos/protos.json manifest. This is generated using Googles GAPIC Toolkit, specifically gapic-generator-typescript.

Googles GAPIC Toolkit exposes macros with Bazel for Protobuf compilation, and specifically gapic.node_library:

# synth.py: https://github.com/googleapis/nodejs-datastore/pull/695/files#diff-3620890cb174a7f98c0722895e37bb79L12-R12
v1_library = gapic.node_library('datastore', version, proto_path=f'google/datastore/{version}')

Explainer in the repository: https://github.com/googleapis/gapic-generator#prerequisites-for-code-generation.

Easier is it to use the specific sub-generators that we need:

This repository contains common annotations to configure proto services:

For example rpc options with google.api.http and google.api.method_signature:

// Tests the specified permissions against the IAM access control policy
// for a [ServiceAccount][google.iam.admin.v1.ServiceAccount].
rpc TestIamPermissions(google.iam.v1.TestIamPermissionsRequest) returns (google.iam.v1.TestIamPermissionsResponse) {
  option (google.api.http) = {
    post: "/v1/{resource=projects/*/serviceAccounts/*}:testIamPermissions"
    body: "*"
  };
  option (google.api.method_signature) = "resource,permissions";
}

And [(google.api.field_behavior) = REQUIRED] for messages:

// Required. The account id that is used to generate the service account
// email address and a stable unique id. It is unique within a project,
// must be 6-30 characters long, and match the regular expression
// `[a-z]([-a-z0-9]*[a-z0-9])` to comply with RFC1035.
string account_id = 2 [(google.api.field_behavior) = REQUIRED];

// The [ServiceAccount][google.iam.admin.v1.ServiceAccount] resource to
// create. Currently, only the following values are user assignable:
// `display_name` and `description`.
ServiceAccount service_account = 3;
mkdir dist
mkdir dist/js && mkdir dist/go
# export PATH=$HOME/go/bin/protoc-gen-lint:$PATH
# protoc -I proto proto/*.proto -I proto/vendor/googleapis --lint_out=. proto/*.proto
files="proto/a.proto proto/b.proto" # proto/vendor/googleapis/google/rpc/status.proto"
# proto/*.proto proto/vendor/googleapis/google/rpc/*.proto
# Generate files for every proto file in the proto directory
protoc -I proto -I proto/vendor/googleapis $files \
--go_out=plugins=grpc:dist/go `# Generates go code for protobuf messages and grpc` \
--js_out=import_style=commonjs,binary:dist/js `# Commonjs imports are required by the ts generator` \
--grpc_out=dist/js --plugin=protoc-gen-grpc=`which grpc_tools_node_protoc_plugin` `# Ensures grpc files are generated` \
--plugin=protoc-gen-ts=./node_modules/.bin/protoc-gen-ts `# Custom plugin for generating ts declarations of generated grpc and message code` \
--ts_out=dist/js `# Save the output from protoc-gen-ts with the js files`
syntax = "proto3";
package test;
option go_package = ".;test";
import "google/protobuf/timestamp.proto";
import "google/rpc/status.proto";
import "shared.proto";
message TestResponse {
google.rpc.Status status = 1;
string user = 2;
}
message TestRequest {
string user = 2;
}
service SomeService {
rpc CreateSome(TestRequest) returns (TestResponse) {}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment