Skip to content

Instantly share code, notes, and snippets.

@hiroyuki-sato
hiroyuki-sato / embulk.md
Created May 3, 2021 14:48
embulk-filter-to_json error
org.embulk.exec.PartialExecutionException: org.embulk.config.ConfigException: Unexpected failure in ObjectNode rebuilt from org.embulk.config.ConfigSource.
	at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(BulkLoader.java:344)
	at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:584)
	at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:37)
	at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:357)
	at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:354)
	at org.embulk.spi.ExecInternal.doWith(ExecInternal.java:26)
	at org.embulk.exec.BulkLoader.run(BulkLoader.java:354)
	at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:324)
embulk gem install -V embulk-output-bigquery
2021-04-21 19:14:42.166 +0900: Embulk v0.9.23
Gem plugin path is: /Users/hsato/.embulk/lib/gems
HEAD https://api.rubygems.org/api/v1/dependencies
200 OK
GET https://api.rubygems.org/api/v1/dependencies?gems=embulk-output-bigquery
200 OK
Getting SRV record failed: DNS result has no information for _rubygems._tcp.api.rubygems.org
testRuntimeClasspath - Runtime classpath of source set 'test'.
+--- org.jruby.joni:joni:2.1.41
| \--- org.jruby.jcodings:jcodings:1.0.55
+--- org.jruby.jcodings:jcodings:1.0.55
+--- org.embulk:embulk-util-config:0.2.1
+--- org.embulk:embulk-util-timestamp:0.2.1
| \--- org.embulk:embulk-util-rubytime:0.3.2
+--- org.embulk:embulk-util-json:0.1.0
| \--- com.fasterxml.jackson.core:jackson-core:2.6.7
+--- org.embulk:embulk-util-text:0.1.0
@hiroyuki-sato
hiroyuki-sato / embulk_error.txt
Last active March 21, 2021 14:35
embulk error
2021-03-21 16:58:34.196 +0900: Embulk v0.9.23
2021-03-21 16:58:35.233 +0900 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected.
2021-03-21 16:58:38.277 +0900 [INFO] (main): Gem's home and path are set by default: "/Users/hsato/.embulk/lib/gems"
2021-03-21 16:58:39.609 +0900 [INFO] (main): Started Embulk v0.9.23
2021-03-21 16:58:39.768 +0900 [INFO] (0001:transaction): Loaded plugin embulk/parser/jsonpath from a load path
2021-03-21 16:58:39.817 +0900 [INFO] (0001:transaction): Listing local files at directory 'example' filtering filename by prefix 'input'
2021-03-21 16:58:39.819 +0900 [INFO] (0001:transaction): "follow_symlinks" is set false. Note that symbolic links to directories are skipped.
2021-03-21 16:58:39.822 +0900 [INFO] (0001:transaction): Loading files [example/input.json, example/input3.json, example/input2.json]
2021-03-21 16:58:39.908 +0900 [INFO] (0001:transaction): Using local thread executor with max_threads=8 / output tasks 6 = input tasks 3 * 2
2021-03
******************************************************************
*
*
* embulk v0.10.20
*
*
******************************************************************
%%%%% try: 1 %%%%%
2021-03-15 09:53:48.509 +0900 [INFO] (main): embulk_home is set from command-line: /Users/usr/.embulk-dev/0.10.20
2021-03-15 09:53:49.011 +0900 [INFO] (main): Started Embulk v0.10.20
org.embulk.exec.PartialExecutionException: java.lang.RuntimeException: java.lang.ClassFormatError: Truncated class file
at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(BulkLoader.java:344)
at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:584)
at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:37)
at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:357)
at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:354)
at org.embulk.spi.ExecInternal.doWith(ExecInternal.java:26)
at org.embulk.exec.BulkLoader.run(BulkLoader.java:354)
at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:324)
at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:288)
@hiroyuki-sato
hiroyuki-sato / cdata.tsv
Last active March 4, 2021 12:48
CData Drivers 20210303
No category name note
1 BIG DATA & NOSQL Amazon Athena
2 BIG DATA & NOSQL Amazon DynamoDB
3 BIG DATA & NOSQL Azure Data Catalog
4 BIG DATA & NOSQL Azure Data Lake Storage
5 BIG DATA & NOSQL BigQuery
6 BIG DATA & NOSQL Cassandra
7 BIG DATA & NOSQL Cloudant
8 BIG DATA & NOSQL CockroachDB
9 BIG DATA & NOSQL Cosmos DB
@hiroyuki-sato
hiroyuki-sato / epsg.java
Created February 26, 2021 00:52
EPSG:32654 -> EPSG:6677変換
/*
* This Java source file was generated by the Gradle 'init' task.
*/
package test;
import org.locationtech.proj4j.CRSFactory;
import org.locationtech.proj4j.CoordinateReferenceSystem;
import org.locationtech.proj4j.CoordinateTransform;
import org.locationtech.proj4j.CoordinateTransformFactory;
import org.locationtech.proj4j.ProjCoordinate;
@hiroyuki-sato
hiroyuki-sato / embulk.diff
Created February 16, 2021 13:58
embulk-filter-flatten_json patch
diff --git a/src/main/java/org/embulk/filter/flatten_json/FlattenJsonFilterPlugin.java b/src/main/java/org/embulk/filter/flatten_json/FlattenJsonFilterPlugin.java
index 8caaf97..7898bc5 100644
--- a/src/main/java/org/embulk/filter/flatten_json/FlattenJsonFilterPlugin.java
+++ b/src/main/java/org/embulk/filter/flatten_json/FlattenJsonFilterPlugin.java
@@ -85,11 +85,12 @@ public class FlattenJsonFilterPlugin
return new PageOutput()
{
private PageReader pageReader = new PageReader(inputSchema);
+ private PageBuilder pageBuilder = new PageBuilder(Exec.getBufferAllocator(), outputSchema, output);
@hiroyuki-sato
hiroyuki-sato / digdag.sql
Created February 15, 2021 11:24
digdag slow attempts check
select
p.name,
sa.attempt_name,
sa.created_at,
sa.finished_at,
(sa.finished_at - sa.created_at) as est
from
session_attempts sa,
projects p
where