Table of Contents
- https://github.com/ashigeru/asakusafw/tree/SCR-01/asakusa-dmdl-java (entry point)
- https://github.com/ashigeru/asakusafw/tree/SCR-01/asakusa-dmdl-core
Flow:
- Syntax Analysis
- Semantic Analysis
- Code Generation
Models:
- Source Code
- Abstract Syntax Tree
- Semantic Graph
- Target Code
- com.asakusafw.dmdl.java.Main (java)
- #main:L94
- L96 -> #configure
- L98 -> GenerateTask#process
- com.asakusafw.dmdl.java.GenerateTask (java)
- com.asakusafw.dmdl.util.AnalyzeTask (core)
- com.asakusafw.dmdl.util.AnalyzeTask (core)
- https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-core/src/main/java/com/asakusafw/dmdl/util/AnalyzeTask.java
- L108: ServiceLoader#load(TypeDriver.class)
- L109: ServiceLoader#load(AttributeDriver.class)
- ServiceLoader: http://download.oracle.com/javase/6/docs/api/java/util/ServiceLoader.html
- com.asakusafw.dmdl.util.AnalyzeTask (core)
- com.asakusafw.dmdl.parser.DmdlParser (core)
- src/main/javacc/DmdlParser.jj
- com.asakusafw.dmdl.util.AnalyzeTask (core)
- https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-core/src/main/java/com/asakusafw/dmdl/util/AnalyzeTask.java
- L122 -> DmdlAnalyzer#addModel
- L81 -> DmdlAnalyzer#resolve
- com.asakusafw.dmdl.analyzer.DmdlAnalyzer (core)
- https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-core/src/main/java/com/asakusafw/dmdl/analyzer/DmdlAnalyzer.java
- #addModel:L111
- L128 -> #computeDependencies:L128->L132
- #resolve:L148
- L150 -> #resolveSymbol:L165 (using topological sort)
- L152 -> #resolveAttribute:L573 (using AttributeDriver)
- com.asakusafw.dmdl.analyzer.DmdlAnalyzer (core)
- com.asakusafw.dmdl.spi.AttributeDriver
- src/main/resources/META-INF/services/com.asakusafw.dmdl.spi.AttributeDriver (core)
- plug-in: com.asakusafw.dmdl.analyzer.driver.NamespaceDriver (core)
- com.asakusafw.dmdl.java.GenerateTask (java)
- https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-java/src/main/java/com/asakusafw/dmdl/java/GenerateTask.java
- L76 -> JavaModelClassGenerator#emit
- com.asakusafw.dmdl.java.emitter.JavaModelClassGenerator (java)
- com.asakusafw.dmdl.java.emitter.ConcreteModelEmitter (java)
- javalang-tools
- https://github.com/ashigeru/javalang-tools/tree/master/java-dom
- Manipulates Java ASTs
- com.asakusafw.dmdl.java.emitter.ConcreteModelEmitter (java)
- https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-java/src/main/java/com/asakusafw/dmdl/java/emitter/ConcreteModelEmitter.java
- L112 -> JavaDataModelDriver#getTypeAnnotations
- L124 -> JavaDataModelDriver#getInterfaces
- L133 -> JavaDataModelDriver#getMethods
- L133 -> JavaDataModelDriver#getMemberAnnotations
- com.asakusafw.dmdl.java.spi.JavaDataModelDriver (java)
- src/main/resources/META-INF/services/com.asakusafw.dmdl.java.spi.JavaDataModelDriver (java)
- example: com.asakusafw.dmdl.java.emitter.driver.WritableDriver (java)
- https://github.com/ashigeru/asakusafw/tree/SCR-01/ashigel-compiler (entry point)
- https://github.com/ashigeru/asakusafw/tree/SCR-01/asakusa-dsl-vocabulary
- *.java
- javac (JDK)
- TypeElement (javax.lang.model.element)
- OperatorClassCollector
- OperatorClass
- OperatorClassEmitter
- *Factory.java, *Impl.java
- JSR 269: Pluggable Annotation Processing API
- src/main/resources/META-INF/services/javax.annotation.processing.Processor
- com.asakusafw.compiler.operator.OperatorCompiler (comp)
- https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/operator/OperatorCompiler.java
- #init:L70
- L77 -> #loadSubProcessors:L98
- #process:L178
- L188 -> #start:L198
- com.asakusafw.compiler.operator.OperatorProcessor (comp)
- https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/operator/OperatorProcessor.java
- #describe:L64
- #implement:L72
- https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/resources/META-INF/services/com.asakusafw.compiler.operator.OperatorProcessor
- example: com.asakusafw.compiler.operator.processor.UpdateOperatorProcessor (comp)
(by javac)
- com.asakusafw.compiler.operator.OperatorCompiler (comp)
- https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/operator/OperatorCompiler.java
- L204 -> OperatorClassCollector#add
- L206 -> OperatorClassCollector#collect
- com.asakusafw.compiler.operator.OperatorClassCollector (comp)
- com.asakusafw.compiler.operator.OperatorCompiler (comp)
- com.asakusafw.compiler.operator.OperatorClassEmitter (comp)
- https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/operator/OperatorClassEmitter.java
- L121 -> OperatorFactoryClassGenerator#generate
- L92 -> OperatorImplementationGenerator#generate
- com.asakusafw.compiler.operator.OperatorFactoryClassGenerator (comp)
- com.asakusafw.compiler.operator.OperatorImplementationClassGenerator (comp)
- com.asakusafw.compiler.operator.DataModelMirrorRepository
- https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/operator/DataModelMirrorRepository.java
- https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/resources/META-INF/services/com.asakusafw.compiler.operator.DataModelMirrorRepository
- https://github.com/ashigeru/asakusafw/tree/SCR-01/ashigel-compiler-bootstrap (entry point)
- https://github.com/ashigeru/asakusafw/tree/SCR-01/ashigel-compiler
- https://github.com/ashigeru/asakusafw/tree/SCR-01/asakusa-dsl-vocabulary
4-Layered:
- Batch
- Jobflow
- External I/O
- Stage (= MapReduce Job)
- Map/Reduce
- Shuffle
- Jobflow
Flow:
- Description Analysis (Batch DSL)
- Semantic Analysis (transaction planning)
- Description Analysis (Flow DSL)
- Semantic Analysis (jobflow: stage planning)
- Semantic Analysis (stage: shuffle planning)
- Code Generation (Map/Reduce)
- Code Generation (Shuffle)
- Code Generation (stage)
- Code Generation (external I/O)
- Semantic Analysis (stage: shuffle planning)
- Code Generation (jobflow)
- Code Generation (batch)
- com.asakusafw.compiler.bootstrap.BatchCompilerDriver (boot)
- com.asakusafw.compiler.testing.DirectBatchCompiler (comp)
- com.asakusafw.compiler.testing.DirectBatchCompiler (comp)
- com.asakusafw.compiler.batch.BatchDriver (comp)
- com.asakusafw.vocabulary.batch.BatchDescription (voc)
- https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dsl-vocabulary/src/main/java/com/asakusafw/vocabulary/batch/BatchDescription.java
- L73 -> #describe (DSL user code)
- L164 -> #getWorks
- com.asakusafw.compiler.testing.DirectBatchCompiler (comp)
- com.asakusafw.compiler.batch.BatchCompiler (comp)
- com.asakusafw.compiler.batch.BatchCompiler (comp)
- https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/batch/BatchCompiler.java
- L104 -> WorkDescriptionProcessor.process
- example: com.asakusafw.compiler.batch.processor.JobFlowWorkDescriptionProcessor (comp)
- com.asakusafw.compiler.batch.processor.JobFlowWorkDescriptionProcessor (comp)
- java.com.asakusafw.compiler.flow.JobFlowDriver (comp)
- https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/flow/JobFlowDriver.java
- L112 -> FlowDescriptionDriver#createFlowGraph
- com.asakusafw.compiler.flow.FlowDescriptionDriver (comp)
- com.asakusafw.vocabulary.flow.FlowDescription (voc)
- com.asakusafw.compiler.batch.processor.JobFlowWorkDescriptionProcessor (comp)
- com.asakusafw.compiler.flow.FlowCompiler (comp)
- com.asakusafw.compiler.flow.plan.StagePlanner (comp)
- https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/flow/plan/StagePlanner.java
- #validate:L801
- #validateConnection
- #validateAcyclic
- #normalizeFlowGraph:L652
- #inlineFlowParts
- #insertCheckpoints
- #buildStageGraph:L164
- #buildComputationBlocks
- #buildStageBlocks
- com.asakusafw.compiler.flow.FlowCompiler (comp)
- com.asakusafw.compiler.flow.stage.StageCompiler (comp)
- https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/flow/stage/StageCompiler.java
- L188 -> ShuffleAnalyzer#analyze
- L189 -> StageAnalyzer#analyze
- L204* -> Shuffle*Emiter#emit
- L154* -> *FragmentEmitter#emit
- L136* -> *erEmitter#emit
- package: com.asakusafw.compiler.flow.stage (comp)
- com.asakusafw.compiler.flow.FlowCompiler (comp)
- com.asakusafw.compiler.flow.jobflow.JobflowCompiler (comp)
- https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/flow/jobflow/JobflowCompiler.java
- L106 -> JobflowAnalyzer#analyze
- L136* -> ExternalIoDescriptionProcessor#emit*
- L276 -> StageClientEmitter#emit
- package: com.asakusafw.compiler.flow.jobflow (comp)
- com.asakusafw.compiler.batch.BatchCompiler (comp)
- example: com.asakusafw.compiler.batch.experimental.ExperimentalWorkflowProcessor (comp)
- com.asakusafw.compiler.batch.WorkflowProcessor (comp)
- com.asakusafw.compiler.flow.FlowElementProcessor (comp)
- com.asakusafw.compiler.flow.ExternalIoDescriptionProcessor (comp)
- https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/flow/ExternalIoDescriptionProcessor.java
- https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/resources/META-INF/services/com.asakusafw.compiler.flow.ExternalIoDescriptionProcessor
- com.asakusafw.compiler.flow.DataClassRepository (comp)
- com.asakusafw.compiler.flow.FlowGraphRewriter (comp)