Demonstrates the implementation of a scanner plugin.

Note
 This tutorial is written for version 1.4.0 of jQAssistant.

1. Overview

A scanner plugin allows extending the functionality of jQAssistant to read arbitrary structures (e.g. from files) and create the corresponding information in the database. The following example demonstrates this for CSV files using a Maven project.

2. Setup Maven Project

The plugin is an artifact which contains the scanner implementation, model classes and a plugin descriptor. A Maven project needs to declare the following dependencies:

pom.xml
<dependencies>
    <dependency>
        <groupId>com.buschmais.jqassistant.core</groupId>
        <artifactId>scanner</artifactId>
        <version>1.4.0</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>com.buschmais.jqassistant.plugin</groupId>
        <artifactId>common</artifactId>
        <version>1.4</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>com.opencsv</groupId>
        <artifactId>opencsv</artifactId>
        <version>4.1</version>
        <scope>provided</scope>
    </dependency>
    <!-- Test dependencies -->
    <dependency>
        <groupId>com.buschmais.jqassistant.core</groupId>
        <artifactId>plugin</artifactId>
        <version>1.4.0</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>com.buschmais.jqassistant.neo4jserver</groupId>
        <artifactId>neo4jv3</artifactId>
        <version>1.4.0</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>com.buschmais.jqassistant.plugin</groupId>
        <artifactId>common</artifactId>
        <type>test-jar</type>
        <version>1.4</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.12</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-simple</artifactId>
        <version>1.7.13</version>
        <scope>test</scope>
    </dependency>
</dependencies>

The following artifacts dependencies are declared:

com.buschmais.jqassistant.core:scanner

Provides the scanner plugin API

com.buschmais.jqassistant.plugin:common

Contains common functionality shared by plugins,

com.opencsv:opencsv

A library 'opencsv' for reading CSV files. This library is already included in the jQAssistant distribution, therefore the scope is provided.

There are also several dependencies defined for implementing tests, see Add A Test.

3. Define The Graph Model

The model to be stored in the database is defined using the approach provided by eXtended Objects. It is based on annotated interfaces declaring methods representing properties and relations.

First a label "CSV" is defined which is going to be used for all nodes created by the scanner:

CSVDescriptor
/**
 * Defines the label which is shared by all nodes representing CSV structures.
 */
@Label("CSV")
public interface CSVDescriptor extends Descriptor {
}

A CSV file is represented by CSVFileDescriptor which inherits from CSVDescriptor and FileDescriptor (provided by the common plugin). Thus a node of this type will carry the labels "CSV" and "File". Furthermore a list of "HAS_ROW" relations is defined by the property "rows".

CSVFileDescriptor
/**
 * Represents a CSV file. The labels are inherited from {@link CSVDescriptor}
 * and {@link FileDescriptor}.
 */
public interface CSVFileDescriptor extends CSVDescriptor, FileDescriptor {

    @Relation("HAS_ROW")
    List<CSVRowDescriptor> getRows();

}

A single row is a node defined by the type CSVRowDescriptor which inherits the label "CSV" from CSVDescriptor and provides its own label "Row", a property "lineNumber" and a list of "HAS_COLUMN" relations.

CSVRowDescriptor
/**
 * Represents a row of a CSV file.
 */
@Label("Row")
public interface CSVRowDescriptor extends CSVDescriptor {

    int getLineNumber();

    void setLineNumber(int lineNumber);

    @Relation("HAS_COLUMN")
    List<CSVColumnDescriptor> getColumns();

}

CSVColumnDescriptor finally defines a column of a row following the principles explained above:

CSVColumnDescriptor
/**
 * Represents a column within a row of a CSV file.
 */
@Label("Column")
public interface CSVColumnDescriptor extends CSVDescriptor {

    String getValue();

    void setValue(String value);

    int getIndex();

    void setIndex(int index);

}

4. Implement The Scanner Plugin

The implementation of the plugin itself inherits from AbstractScannerPlugin which requires generic type parameters for the item type it handles and the descriptor type it creates. In the example FileResource is used which represents a file contained in a directory or archive. This allows plugins to be independent of the source where files or directories are picked up by the scanner.

The method accepts is called by the scanner to determine if the plugin can handle the given item. The example matches the value of the parameter path against the file extension ".csv". The scope parameter may be used to further restrict executions of the plugin, e.g. by checking equality against JavaScope.CLASSPATH.

The scan method actually reads the CSV file and stores the gathered data into the database using the interface Store provided by the scanner context.

CSVFileScannerPlugin
/**
 * A CSV file scanner plugin.
 */
@Requires(FileDescriptor.class) // The file descriptor is created by the file scanner plugin and enriched by
                                // this one
public class CSVFileScannerPlugin extends AbstractScannerPlugin<FileResource, CSVFileDescriptor> {

    @Override
    public boolean accepts(FileResource item, String path, Scope scope) {
        return path.toLowerCase().endsWith(".csv");
    }

    @Override
    public CSVFileDescriptor scan(FileResource item, String path, Scope scope, Scanner scanner) throws IOException {
        ScannerContext context = scanner.getContext();
        final Store store = context.getStore();
        // Open the input stream for reading the file.
        try (InputStream stream = item.createStream()) {
            // Retrieve the scanned file node from the scanner context.
            final FileDescriptor fileDescriptor = context.getCurrentDescriptor();
            // Add the CSV label.
            final CSVFileDescriptor csvFileDescriptor = store.addDescriptorType(fileDescriptor, CSVFileDescriptor.class);
            // Parse the stream using OpenCSV.
            CSVReader csvReader = new CSVReader(new InputStreamReader(stream));
            String[] columns;
            int row = 0;
            while ((columns = csvReader.readNext()) != null) {
                // Create the node for a row
                CSVRowDescriptor rowDescriptor = store.create(CSVRowDescriptor.class);
                csvFileDescriptor.getRows().add(rowDescriptor);
                rowDescriptor.setLineNumber(row);
                for (int i = 0; i < columns.length; i++) {
                    // Create the node for a column
                    CSVColumnDescriptor columnDescriptor = store.create(CSVColumnDescriptor.class);
                    rowDescriptor.getColumns().add(columnDescriptor);
                    columnDescriptor.setIndex(i);
                    columnDescriptor.setValue(columns[i]);
                }
                row++;
            }
            return csvFileDescriptor;
        }
    }
}

Finally the model and the plugin implementation must be declared in the jQAssistant plugin descriptor:

/META-INF/jqassistant-plugin.xml
<jqa-plugin:jqassistant-plugin
        xmlns:jqa-plugin="http://www.buschmais.com/jqassistant/core/plugin/schema/v1.1" name="CSV Example">
    <description>Provides a scanner for CSV file.</description>
    <model>
        <class>my.project.plugin.scanner.model.CSVFileDescriptor</class>
        <class>my.project.plugin.scanner.model.CSVRowDescriptor</class>
        <class>my.project.plugin.scanner.model.CSVColumnDescriptor</class>
    </model>
    <scanner>
        <class>my.project.plugin.scanner.CSVFileScannerPlugin</class>
    </scanner>
</jqa-plugin:jqassistant-plugin>

The plugin is automatically loaded by the scanner if it can be found on the classpath, e.g. by adding it as dependency to the Maven plugin.

5. Add A Test

jQAssistant comes with a test framework which provides convenience functionality for scanning and verifying data.

The CSV scanner plugin shall be tested using the following file:

src/test/resources/test.csv
Indiana,Jones
Resultset
Figure 1. Graph representation of the CSV file

The test class CSVScannerPluginTest extends from AbstractPluginIT and

  • scans the CSV file which is located in the root of the test class path

  • executes a Cypher query to lookup the created file node with the labels CSV and Test

  • verifies the created graph representing the CSV structure

CSVScannerPluginTest
public class CSVScannerPluginTest extends AbstractPluginIT {

    @Test
    public void scanCSVFile() {
        store.beginTransaction();
        // Scan the test CSV file located as resource in the classpath
        File testFile = new File(getClassesDirectory(CSVScannerPluginTest.class), "/test.csv");

        // Scan the CSV file and assert that the returned descriptor is a CSVFileDescriptor
        assertThat(getScanner().scan(testFile, "/test.csv", DefaultScope.NONE), CoreMatchers.<Descriptor>instanceOf(CSVFileDescriptor.class));

        // Determine the CSVFileDescriptor by executing a Cypher query
        TestResult testResult = query("MATCH (csvFile:CSV:File) RETURN csvFile");
        List<CSVFileDescriptor> csvFiles = testResult.getColumn("csvFile");
        assertThat(csvFiles.size(), equalTo(1));

        CSVFileDescriptor csvFile = csvFiles.get(0);
        assertThat(csvFile.getFileName(), equalTo("/test.csv"));

        // Get rows and verify expected count
        List<CSVRowDescriptor> rows = csvFile.getRows();
        assertThat(rows.size(), equalTo(1));

        // Verify first (and only) row
        CSVRowDescriptor row0 = rows.get(0);
        assertThat(row0.getLineNumber(), equalTo(0));
        // Verify the columns of the first row
        List<CSVColumnDescriptor> row0Columns = row0.getColumns();
        assertThat(row0Columns.size(), equalTo(2));

        CSVColumnDescriptor headerColumn0 = row0Columns.get(0);
        assertThat(headerColumn0.getIndex(), equalTo(0));
        assertThat(headerColumn0.getValue(), equalTo("Indiana"));

        CSVColumnDescriptor headerColumn1 = row0Columns.get(1);
        assertThat(headerColumn1.getIndex(), equalTo(1));
        assertThat(headerColumn1.getValue(), equalTo("Jones"));

        store.commitTransaction();
    }
}

6. Resources

  1. ZIP archive including the plugin