Skip to content

Commit

Permalink
Merge branch 'feature_java-11'
Browse files Browse the repository at this point in the history
  • Loading branch information
@PastorGL
PastorGL committed Mar 7, 2023
2 parents bce693e + 9954a2e commit f94f5b6
Show file tree
Hide file tree
Showing 49 changed files with 1,620 additions and 112 deletions.
6 changes: 3 additions & 3 deletions TDL4.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -282,7 +282,7 @@ <h3>Language Operators</h3>
</table>

<h3>Parameter List Expressions</h3>
<p>Parameter lists are expressions composed from parameters (at least one), separated by commas.</p>
<p>Parameter lists are expressions composed of parameters (at least one), separated by commas.</p>
<div class="hor-scroll"><svg xmlns="http://www.w3.org/2000/svg" class="railroad-diagram" width="476" height="110" viewBox="0 0 476 110"><g transform="translate(.5 .5)"><path d="M 20 45 v 20 m 0 -10 h 20.5"></path><g><path d="M40 55h0"></path><path d="M436 55h0"></path><path d="M40 55h20"></path><g><path d="M60 55h0"></path><path d="M416 55h0"></path><g class="non-terminal"><path d="M60 55h0"></path><path d="M130 55h0"></path><rect x="60" y="44" width="70" height="22" rx="8" ry="8"></rect><text x="95" y="59">param</text></g><path d="M130 55h10"></path><g><path d="M140 55h0"></path><path d="M416 55h0"></path><path d="M140 55a10 10 0 0 0 10 -10v-15a10 10 0 0 1 10 -10"></path><g><path d="M160 20h236"></path></g><path d="M396 20a10 10 0 0 1 10 10v15a10 10 0 0 0 10 10"></path><path d="M140 55h20"></path><g><path d="M160 55h0"></path><path d="M396 55h0"></path><path d="M160 55h10"></path><g><path d="M170 55h0"></path><path d="M386 55h0"></path><path d="M170 55h20"></path><g><path d="M190 55h0"></path><path d="M366 55h0"></path><g class="terminal"><path d="M190 55h0"></path><path d="M276 55h0"></path><rect x="190" y="44" width="86" height="22" rx="0" ry="0"></rect><text x="233" y="59">,</text></g><path d="M276 55h10"></path><path d="M286 55h10"></path><g class="non-terminal"><path d="M296 55h0"></path><path d="M366 55h0"></path><rect x="296" y="44" width="70" height="22" rx="8" ry="8"></rect><text x="331" y="59">param</text></g></g><path d="M366 55h20"></path></g><path d="M386 55h10"></path><path d="M170 55a10 10 0 0 0 -10 10v15a10 10 0 0 0 10 10"></path><g><path d="M170 90h216"></path></g><path d="M386 90a10 10 0 0 0 10 -10v-15a10 10 0 0 0 -10 -10"></path></g><path d="M396 55h20"></path></g></g><path d="M416 55h20"></path></g><path d="M 436 55 h 20 m 0 -10 v 20"></path></g></svg></div>
<p>Each parameter is a name to value (which may be expression or array literal) pair.</p>
<div class="hor-scroll"><svg xmlns="http://www.w3.org/2000/svg" class="railroad-diagram" width="540" height="108" viewBox="0 0 540 108"><g transform="translate(.5 .5)"><path d="M 20 21 v 20 m 0 -10 h 20.5"></path><g><path d="M40 31h0"></path><path d="M500 31h0"></path><path d="M40 31h20"></path><g><path d="M60 31h0"></path><path d="M480 31h0"></path><g class="terminal"><path d="M60 31h0"></path><path d="M122 31h0"></path><rect x="60" y="20" width="62" height="22" rx="0" ry="0"></rect><text x="91" y="35">@</text></g><path d="M122 31h10"></path><path d="M132 31h10"></path><g class="terminal"><path d="M142 31h0"></path><path d="M268 31h0"></path><rect x="142" y="20" width="126" height="22" rx="0" ry="0"></rect><text x="205" y="35">IDENTIFIER</text></g><path d="M268 31h10"></path><path d="M278 31h10"></path><g class="terminal"><path d="M288 31h0"></path><path d="M350 31h0"></path><rect x="288" y="20" width="62" height="22" rx="0" ry="0"></rect><text x="319" y="35">=</text></g><path d="M350 31h10"></path><path d="M360 31h10"></path><g class="non-terminal"><path d="M370 31h0"></path><path d="M480 31h0"></path><rect x="370" y="20" width="110" height="22" rx="8" ry="8"></rect><text x="425" y="35">expression</text></g></g><path d="M480 31h20"></path><path d="M40 31a10 10 0 0 1 10 10v26a10 10 0 0 0 10 10"></path><g><path d="M60 77h20"></path><path d="M460 77h20"></path><g class="terminal"><path d="M80 77h0"></path><path d="M142 77h0"></path><rect x="80" y="66" width="62" height="22" rx="0" ry="0"></rect><text x="111" y="81">@</text></g><path d="M142 77h10"></path><path d="M152 77h10"></path><g class="terminal"><path d="M162 77h0"></path><path d="M288 77h0"></path><rect x="162" y="66" width="126" height="22" rx="0" ry="0"></rect><text x="225" y="81">IDENTIFIER</text></g><path d="M288 77h10"></path><path d="M298 77h10"></path><g class="terminal"><path d="M308 77h0"></path><path d="M370 77h0"></path><rect x="308" y="66" width="62" height="22" rx="0" ry="0"></rect><text x="339" y="81">=</text></g><path d="M370 77h10"></path><path d="M380 77h10"></path><g class="non-terminal"><path d="M390 77h0"></path><path d="M460 77h0"></path><rect x="390" y="66" width="70" height="22" rx="8" ry="8"></rect><text x="425" y="81">array</text></g></g><path d="M480 77a10 10 0 0 0 10 -10v-26a10 10 0 0 1 10 -10"></path></g><path d="M 500 31 h 20 m 0 -10 v 20"></path></g></svg></div>
Expand All @@ -294,12 +294,12 @@ <h2>Data Storage Interface Statements</h2>
<h3>CREATE Statement</h3>
<p>Creates a new Data Set from data source. Its name must be new and unique.</p>
<div class="hor-scroll"><svg xmlns="http://www.w3.org/2000/svg" class="railroad-diagram" width="828" height="86" viewBox="0 0 828 86"><g transform="translate(.5 .5)"><path d="M 20 45 v 20 m 0 -10 h 20.5"></path><g><path d="M40 55h0"></path><path d="M788 55h0"></path><path d="M40 55h20"></path><g><path d="M60 55h0"></path><path d="M768 55h0"></path><g class="terminal"><path d="M60 55h0"></path><path d="M154 55h0"></path><rect x="60" y="44" width="94" height="22" rx="0" ry="0"></rect><text x="107" y="59">CREATE</text></g><path d="M154 55h10"></path><g><path d="M164 55h0"></path><path d="M266 55h0"></path><path d="M164 55a10 10 0 0 0 10 -10v-15a10 10 0 0 1 10 -10"></path><g><path d="M184 20h62"></path></g><path d="M246 20a10 10 0 0 1 10 10v15a10 10 0 0 0 10 10"></path><path d="M164 55h20"></path><g class="terminal"><path d="M184 55h0"></path><path d="M246 55h0"></path><rect x="184" y="44" width="62" height="22" rx="0" ry="0"></rect><text x="215" y="59">DS</text></g><path d="M246 55h20"></path></g><path d="M266 55h10"></path><g class="non-terminal"><path d="M276 55h0"></path><path d="M362 55h0"></path><rect x="276" y="44" width="86" height="22" rx="8" ry="8"></rect><text x="319" y="59">ds_name</text></g><path d="M362 55h10"></path><path d="M372 55h10"></path><g class="terminal"><path d="M382 55h0"></path><path d="M492 55h0"></path><rect x="382" y="44" width="110" height="22" rx="0" ry="0"></rect><text x="437" y="59">(</text></g><path d="M492 55h10"></path><path d="M502 55h10"></path><g class="non-terminal"><path d="M512 55h0"></path><path d="M630 55h0"></path><rect x="512" y="44" width="118" height="22" rx="8" ry="8"></rect><text x="571" y="59">params_expr</text></g><path d="M630 55h10"></path><path d="M640 55h10"></path><g class="terminal"><path d="M650 55h0"></path><path d="M768 55h0"></path><rect x="650" y="44" width="118" height="22" rx="0" ry="0"></rect><text x="709" y="59">)</text></g></g><path d="M768 55h20"></path></g><path d="M 788 55 h 20 m 0 -10 v 20"></path></g></svg></div>
<p>There is one mandatory parameter named <b>@path</b>, which must contain a path (usually, URI) string to data source. There is common optional parameter <b>@part_count</b> (defaults to 1 if omitted) that specifies a number of parts in the Data Set.</p>
<p>There is one mandatory parameter named <b>@path</b>, which must contain a path (usually, a URI) to data source. There is common optional parameter <b>@part_count</b> (defaults to 1 if omitted) that specifies a number of parts in the Data Set. To use Storage Adapter apart from standard Hadoop, <b>@adapter</b> parameter could be used (a list of available Storages depends on the distribution).</p>

<h3>COPY Statement</h3>
<p>Saves existing Data Set to storage.</p>
<div class="hor-scroll"><svg xmlns="http://www.w3.org/2000/svg" class="railroad-diagram" width="930" height="86" viewBox="0 0 930 86"><g transform="translate(.5 .5)"><path d="M 20 45 v 20 m 0 -10 h 20.5"></path><g><path d="M40 55h0"></path><path d="M890 55h0"></path><path d="M40 55h20"></path><g><path d="M60 55h0"></path><path d="M870 55h0"></path><g class="terminal"><path d="M60 55h0"></path><path d="M138 55h0"></path><rect x="60" y="44" width="78" height="22" rx="0" ry="0"></rect><text x="99" y="59">COPY</text></g><path d="M138 55h10"></path><g><path d="M148 55h0"></path><path d="M250 55h0"></path><path d="M148 55a10 10 0 0 0 10 -10v-15a10 10 0 0 1 10 -10"></path><g><path d="M168 20h62"></path></g><path d="M230 20a10 10 0 0 1 10 10v15a10 10 0 0 0 10 10"></path><path d="M148 55h20"></path><g class="terminal"><path d="M168 55h0"></path><path d="M230 55h0"></path><rect x="168" y="44" width="62" height="22" rx="0" ry="0"></rect><text x="199" y="59">DS</text></g><path d="M230 55h20"></path></g><path d="M250 55h10"></path><g class="non-terminal"><path d="M260 55h0"></path><path d="M346 55h0"></path><rect x="260" y="44" width="86" height="22" rx="8" ry="8"></rect><text x="303" y="59">ds_name</text></g><path d="M346 55h10"></path><g><path d="M356 55h0"></path><path d="M474 55h0"></path><path d="M356 55a10 10 0 0 0 10 -10v-15a10 10 0 0 1 10 -10"></path><g><path d="M376 20h78"></path></g><path d="M454 20a10 10 0 0 1 10 10v15a10 10 0 0 0 10 10"></path><path d="M356 55h20"></path><g class="terminal"><path d="M376 55h0"></path><path d="M454 55h0"></path><rect x="376" y="44" width="78" height="22" rx="0" ry="0"></rect><text x="415" y="59">*</text></g><path d="M454 55h20"></path></g><path d="M474 55h10"></path><g class="terminal"><path d="M484 55h0"></path><path d="M594 55h0"></path><rect x="484" y="44" width="110" height="22" rx="0" ry="0"></rect><text x="539" y="59">(</text></g><path d="M594 55h10"></path><path d="M604 55h10"></path><g class="non-terminal"><path d="M614 55h0"></path><path d="M732 55h0"></path><rect x="614" y="44" width="118" height="22" rx="8" ry="8"></rect><text x="673" y="59">params_expr</text></g><path d="M732 55h10"></path><path d="M742 55h10"></path><g class="terminal"><path d="M752 55h0"></path><path d="M870 55h0"></path><rect x="752" y="44" width="118" height="22" rx="0" ry="0"></rect><text x="811" y="59">)</text></g></g><path d="M870 55h20"></path></g><path d="M 890 55 h 20 m 0 -10 v 20"></path></g></svg></div>
<p>There is one mandatory parameter named <b>@path</b>, which must contain a path (usually, URI) string to data storage.</p>
<p>There is one mandatory parameter named <b>@path</b>, which must contain a path (usually, a URI) string to data Storage. To use Storage Adapter apart from standard Hadoop, <b>@adapter</b> parameter could be used (a list of available Storages depends on the distribution).</p>
<p>If a star is used after Data Set name, it is treated as a prefix, and all matching Data Sets will be copied to storage (usually, into subfolders under specified <b>@path</b>).</p>

<h2>Data Transformation Statements</h2>
Expand Down
2 changes: 1 addition & 1 deletion cli/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,6 @@
</execution>
</executions>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
Expand Down Expand Up @@ -218,6 +217,7 @@
<mainClass>io.github.pastorgl.datacooker.doc.DocGen</mainClass>
<arguments>
<argument>${docs.output}</argument>
<argument>Data Cooker ETL Tool</argument>
</arguments>
</configuration>
</execution>
Expand Down
91 changes: 89 additions & 2 deletions cli/src/test/java/io/github/pastorgl/datacooker/doc/DocGen.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,11 @@
import io.github.pastorgl.datacooker.RegisteredPackages;
import io.github.pastorgl.datacooker.data.TransformInfo;
import io.github.pastorgl.datacooker.data.Transforms;
import io.github.pastorgl.datacooker.metadata.AdapterMeta;
import io.github.pastorgl.datacooker.scripting.OperationInfo;
import io.github.pastorgl.datacooker.scripting.Operations;
import io.github.pastorgl.datacooker.storage.AdapterInfo;
import io.github.pastorgl.datacooker.storage.Adapters;
import org.apache.commons.io.IOUtils;
import org.apache.velocity.VelocityContext;
import org.apache.velocity.app.Velocity;
Expand All @@ -27,7 +30,9 @@
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.HashMap;
import java.util.Map;
import java.util.stream.Collectors;

import static java.nio.charset.StandardCharsets.UTF_8;

Expand All @@ -44,6 +49,8 @@ public static void main(String[] args) {

}
Files.createDirectories(Paths.get(outputDirectory, "package"));
Files.createDirectories(Paths.get(outputDirectory, "input"));
Files.createDirectories(Paths.get(outputDirectory, "output"));
Files.createDirectories(Paths.get(outputDirectory, "operation"));
Files.createDirectories(Paths.get(outputDirectory, "transform"));

Expand All @@ -63,6 +70,7 @@ public static void main(String[] args) {
try (FileWriter writer = new FileWriter(outputDirectory + "/index.html"); StringWriter sw = new StringWriter()) {
VelocityContext ic = new VelocityContext();
ic.put("pkgs", pkgs);
ic.put("distro", args[1]);

Velocity.getTemplate("index.vm", UTF_8.name()).merge(ic, sw);

Expand All @@ -77,6 +85,8 @@ public static void main(String[] args) {
}

for (String pkgName : pkgs.keySet()) {
Map<String, AdapterInfo> ins = Adapters.packageInputs(pkgName);
Map<String, AdapterInfo> outs = Adapters.packageOutputs(pkgName);
Map<String, OperationInfo> ops = Operations.packageOperations(pkgName);
Map<String, TransformInfo> transforms = Transforms.packageTransforms(pkgName);

Expand All @@ -86,25 +96,85 @@ public static void main(String[] args) {
VelocityContext ic = new VelocityContext();
ic.put("name", pkgName);
ic.put("descr", descr);
ic.put("ins", ins);
ic.put("outs", outs);
ic.put("ops", ops);
ic.put("transforms", transforms);

Velocity.getTemplate("package.vm", UTF_8.name()).merge(ic, sw);

String pkg = sw.toString();

m.append(pkg.replace("href=\"transform/", "href=\"#transform/")
m.append(pkg.replace("href=\"input/", "href=\"#input/")
.replace("href=\"output/", "href=\"#output/")
.replace("href=\"transform/", "href=\"#transform/")
.replace("href=\"operation/", "href=\"#operation/")
.replace("href=\"index", "href=\"#index"));
writer.append(header);
writer.append(pkg.replace("href=\"transform/", "href=\"../transform/")
writer.append(pkg.replace("href=\"input/", "href=\"../input/")
.replace("href=\"output/", "href=\"../output/")
.replace("href=\"transform/", "href=\"../transform/")
.replace("href=\"operation/", "href=\"../operation/")
.replace("href=\"index", "href=\"../index"));
writer.append(footer);
} catch (Exception e) {
throw new Exception("Package '" + pkgName + "'", e);
}

for (Map.Entry<String, AdapterInfo> entry : ins.entrySet()) {
String verb = entry.getKey();
AdapterInfo opInfo = entry.getValue();

try (FileWriter writer = new FileWriter(outputDirectory + "/input/" + verb + ".html"); StringWriter sw = new StringWriter()) {
VelocityContext vc = new VelocityContext();
vc.put("op", opInfo.meta);
vc.put("pkgName", pkgName);

String example = genAdapterExample("source", opInfo.meta);
vc.put("example", example);

Velocity.getTemplate("input.vm", StandardCharsets.UTF_8.name()).merge(vc, sw);

String op = sw.toString();

m.append(op.replace("href=\"package/", "href=\"#package/")
.replace("href=\"index", "href=\"#index"));
writer.append(header);
writer.append(op.replace("href=\"package/", "href=\"../package/")
.replace("href=\"index", "href=\"../index"));
writer.append(footer);
} catch (Exception e) {
throw new Exception("Input adapter '" + verb + "'", e);
}
}

for (Map.Entry<String, AdapterInfo> entry : outs.entrySet()) {
String verb = entry.getKey();
AdapterInfo opInfo = entry.getValue();

try (FileWriter writer = new FileWriter(outputDirectory + "/output/" + verb + ".html"); StringWriter sw = new StringWriter()) {
VelocityContext vc = new VelocityContext();
vc.put("op", opInfo.meta);
vc.put("pkgName", pkgName);

String example = genAdapterExample("dest", opInfo.meta);
vc.put("example", example);

Velocity.getTemplate("output.vm", StandardCharsets.UTF_8.name()).merge(vc, sw);

String tr = sw.toString();

m.append(tr.replace("href=\"package/", "href=\"#package/")
.replace("href=\"index", "href=\"#index"));
writer.append(header);
writer.append(tr.replace("href=\"package/", "href=\"../package/")
.replace("href=\"index", "href=\"../index"));
writer.append(footer);
} catch (Exception e) {
throw new Exception("Output adapter '" + verb + "'", e);
}
}

for (Map.Entry<String, TransformInfo> entry : transforms.entrySet()) {
String verb = entry.getKey();
TransformInfo opInfo = entry.getValue();
Expand Down Expand Up @@ -203,4 +273,21 @@ public static void main(String[] args) {
System.exit(-7);
}
}

private static String genAdapterExample(String dir, AdapterMeta am) {
Map<String, Object> params = new HashMap<>();
am.definitions.forEach((name, meta) -> params.put(name, meta.defaults));
String example = "input".equals(dir) ? "CREATE" : "COPY";

return new SyntaxHighlighter(example + " example " + params.entrySet().stream()
.filter(e -> (e.getValue() != null))
.map(e -> {
String ret = "@" + e.getKey() + "=";
Object def = e.getValue();
ret += (def instanceof String) ? "'" + def + "'" : String.valueOf(def).toUpperCase();
return ret;
})
.collect(Collectors.joining(",\n", "(", ")")) + ";")
.highlight();
}
}
2 changes: 1 addition & 1 deletion cli/src/test/resources/index.vm
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<a id="index.html"></a>
<h1>Data Cooker ETL Tool Index</h1>
<h1>$distro Index</h1>

<p>This distribution has the following packages available</p>

Expand Down
Loading

0 comments on commit f94f5b6

Please sign in to comment.