Demonstrates the implementation of a scanner plugin.
Note
|
This tutorial is written for version 1.4.0 of jQAssistant. |
1. Overview
A scanner plugin allows extending the functionality of jQAssistant to read arbitrary structures (e.g. from files) and create the corresponding information in the database. The following example demonstrates this for CSV files using a Maven project.
2. Setup Maven Project
The plugin is an artifact which contains the scanner implementation, model classes and a plugin descriptor. A Maven project needs to declare the following dependencies:
<dependencies>
<dependency>
<groupId>com.buschmais.jqassistant.core</groupId>
<artifactId>scanner</artifactId>
<version>1.4.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.buschmais.jqassistant.plugin</groupId>
<artifactId>common</artifactId>
<version>1.4</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.opencsv</groupId>
<artifactId>opencsv</artifactId>
<version>4.1</version>
<scope>provided</scope>
</dependency>
<!-- Test dependencies -->
<dependency>
<groupId>com.buschmais.jqassistant.core</groupId>
<artifactId>plugin</artifactId>
<version>1.4.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.buschmais.jqassistant.neo4jserver</groupId>
<artifactId>neo4jv3</artifactId>
<version>1.4.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.buschmais.jqassistant.plugin</groupId>
<artifactId>common</artifactId>
<type>test-jar</type>
<version>1.4</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.7.13</version>
<scope>test</scope>
</dependency>
</dependencies>
The following artifacts dependencies are declared:
- com.buschmais.jqassistant.core:scanner
-
Provides the scanner plugin API
- com.buschmais.jqassistant.plugin:common
-
Contains common functionality shared by plugins,
- com.opencsv:opencsv
-
A library 'opencsv' for reading CSV files. This library is already included in the jQAssistant distribution, therefore the scope is
provided
.
There are also several dependencies defined for implementing tests, see Add A Test.
3. Define The Graph Model
The model to be stored in the database is defined using the approach provided by eXtended Objects. It is based on annotated interfaces declaring methods representing properties and relations.
First a label "CSV" is defined which is going to be used for all nodes created by the scanner:
/**
* Defines the label which is shared by all nodes representing CSV structures.
*/
@Label("CSV")
public interface CSVDescriptor extends Descriptor {
}
A CSV file is represented by CSVFileDescriptor which inherits from CSVDescriptor and FileDescriptor (provided by the common plugin). Thus a node of this type will carry the labels "CSV" and "File". Furthermore a list of "HAS_ROW" relations is defined by the property "rows".
/**
* Represents a CSV file. The labels are inherited from {@link CSVDescriptor}
* and {@link FileDescriptor}.
*/
public interface CSVFileDescriptor extends CSVDescriptor, FileDescriptor {
@Relation("HAS_ROW")
List<CSVRowDescriptor> getRows();
}
A single row is a node defined by the type CSVRowDescriptor which inherits the label "CSV" from CSVDescriptor and provides its own label "Row", a property "lineNumber" and a list of "HAS_COLUMN" relations.
/**
* Represents a row of a CSV file.
*/
@Label("Row")
public interface CSVRowDescriptor extends CSVDescriptor {
int getLineNumber();
void setLineNumber(int lineNumber);
@Relation("HAS_COLUMN")
List<CSVColumnDescriptor> getColumns();
}
CSVColumnDescriptor finally defines a column of a row following the principles explained above:
/**
* Represents a column within a row of a CSV file.
*/
@Label("Column")
public interface CSVColumnDescriptor extends CSVDescriptor {
String getValue();
void setValue(String value);
int getIndex();
void setIndex(int index);
}
4. Implement The Scanner Plugin
The implementation of the plugin itself inherits from AbstractScannerPlugin which requires generic type parameters for the item type it handles and the descriptor type it creates. In the example FileResource is used which represents a file contained in a directory or archive. This allows plugins to be independent of the source where files or directories are picked up by the scanner.
The method accepts is called by the scanner to determine if the plugin can handle the given item. The example matches the value of the parameter path against the file extension ".csv". The scope parameter may be used to further restrict executions of the plugin, e.g. by checking equality against JavaScope.CLASSPATH.
The scan method actually reads the CSV file and stores the gathered data into the database using the interface Store provided by the scanner context.
/**
* A CSV file scanner plugin.
*/
@Requires(FileDescriptor.class) // The file descriptor is created by the file scanner plugin and enriched by
// this one
public class CSVFileScannerPlugin extends AbstractScannerPlugin<FileResource, CSVFileDescriptor> {
@Override
public boolean accepts(FileResource item, String path, Scope scope) {
return path.toLowerCase().endsWith(".csv");
}
@Override
public CSVFileDescriptor scan(FileResource item, String path, Scope scope, Scanner scanner) throws IOException {
ScannerContext context = scanner.getContext();
final Store store = context.getStore();
// Open the input stream for reading the file.
try (InputStream stream = item.createStream()) {
// Retrieve the scanned file node from the scanner context.
final FileDescriptor fileDescriptor = context.getCurrentDescriptor();
// Add the CSV label.
final CSVFileDescriptor csvFileDescriptor = store.addDescriptorType(fileDescriptor, CSVFileDescriptor.class);
// Parse the stream using OpenCSV.
CSVReader csvReader = new CSVReader(new InputStreamReader(stream));
String[] columns;
int row = 0;
while ((columns = csvReader.readNext()) != null) {
// Create the node for a row
CSVRowDescriptor rowDescriptor = store.create(CSVRowDescriptor.class);
csvFileDescriptor.getRows().add(rowDescriptor);
rowDescriptor.setLineNumber(row);
for (int i = 0; i < columns.length; i++) {
// Create the node for a column
CSVColumnDescriptor columnDescriptor = store.create(CSVColumnDescriptor.class);
rowDescriptor.getColumns().add(columnDescriptor);
columnDescriptor.setIndex(i);
columnDescriptor.setValue(columns[i]);
}
row++;
}
return csvFileDescriptor;
}
}
}
Finally the model and the plugin implementation must be declared in the jQAssistant plugin descriptor:
<jqa-plugin:jqassistant-plugin
xmlns:jqa-plugin="http://www.buschmais.com/jqassistant/core/plugin/schema/v1.1" name="CSV Example">
<description>Provides a scanner for CSV file.</description>
<model>
<class>my.project.plugin.scanner.model.CSVFileDescriptor</class>
<class>my.project.plugin.scanner.model.CSVRowDescriptor</class>
<class>my.project.plugin.scanner.model.CSVColumnDescriptor</class>
</model>
<scanner>
<class>my.project.plugin.scanner.CSVFileScannerPlugin</class>
</scanner>
</jqa-plugin:jqassistant-plugin>
The plugin is automatically loaded by the scanner if it can be found on the classpath, e.g. by adding it as dependency to the Maven plugin.
5. Add A Test
jQAssistant comes with a test framework which provides convenience functionality for scanning and verifying data.
The CSV scanner plugin shall be tested using the following file:
Indiana,Jones
The test class CSVScannerPluginTest
extends from AbstractPluginIT
and
-
scans the CSV file which is located in the root of the test class path
-
executes a Cypher query to lookup the created file node with the labels
CSV
andTest
-
verifies the created graph representing the CSV structure
public class CSVScannerPluginTest extends AbstractPluginIT {
@Test
public void scanCSVFile() {
store.beginTransaction();
// Scan the test CSV file located as resource in the classpath
File testFile = new File(getClassesDirectory(CSVScannerPluginTest.class), "/test.csv");
// Scan the CSV file and assert that the returned descriptor is a CSVFileDescriptor
assertThat(getScanner().scan(testFile, "/test.csv", DefaultScope.NONE), CoreMatchers.<Descriptor>instanceOf(CSVFileDescriptor.class));
// Determine the CSVFileDescriptor by executing a Cypher query
TestResult testResult = query("MATCH (csvFile:CSV:File) RETURN csvFile");
List<CSVFileDescriptor> csvFiles = testResult.getColumn("csvFile");
assertThat(csvFiles.size(), equalTo(1));
CSVFileDescriptor csvFile = csvFiles.get(0);
assertThat(csvFile.getFileName(), equalTo("/test.csv"));
// Get rows and verify expected count
List<CSVRowDescriptor> rows = csvFile.getRows();
assertThat(rows.size(), equalTo(1));
// Verify first (and only) row
CSVRowDescriptor row0 = rows.get(0);
assertThat(row0.getLineNumber(), equalTo(0));
// Verify the columns of the first row
List<CSVColumnDescriptor> row0Columns = row0.getColumns();
assertThat(row0Columns.size(), equalTo(2));
CSVColumnDescriptor headerColumn0 = row0Columns.get(0);
assertThat(headerColumn0.getIndex(), equalTo(0));
assertThat(headerColumn0.getValue(), equalTo("Indiana"));
CSVColumnDescriptor headerColumn1 = row0Columns.get(1);
assertThat(headerColumn1.getIndex(), equalTo(1));
assertThat(headerColumn1.getValue(), equalTo("Jones"));
store.commitTransaction();
}
}