Untitled RSS Feed

Real World Example for Google Cloud Spanner - JPA - Hibernate

Knut Olav Løite — Sat, 27 May 2017 00:00:00 GMT

This post is a short description of a real-world example of an application using JPA/Hibernate and Spring Boot in combination with Google Cloud Spanner. As it is based on a real application, its code base is large and somewhat complex. If you are looking for a simple tutorial on how to set up Google Cloud Spanner with JPA/Hibernate, then have a look here: https://olavloite.github.io/2017/03/11/Google-Cloud-Spanner-with-Spring-Boot-JPA-and-Hibernate.html

Introduction

https://github.com/olavloite/github-monitor

https://github.com/olavloite/gaf

The combination of the two above projects show a real world example of how an application can be developed using JPA/Hibernate in combination with Google Cloud Spanner.

The project depends on two important libraries:

Google Cloud Spanner JDBC Driver

An open source JDBC Driver for Google Cloud Spanner that supports a number of features not supported by the official JDBC Driver from Google:

DML-statements (UPDATE, INSERT, DELETE)
DDL-statements (CREATE TABLE, ALTER TABLE, …)
Transactions

More information on this driver can be found here: https://github.com/olavloite/spanner-jdbc

Google Cloud Spanner Dialect

A Hibernate Dialect for Google Cloud Spanner. More information on this dialect can be found here: https://github.com/olavloite/spanner-hibernate

The dialect supports schema generation.

Example Project

The example project is a web application that reads GitHub Repositories from GitHub and stores these in a Google Cloud Spanner database. The main goal of this project is to show how you could develop a real world application with Google Cloud Spanner / JPA / Hibernate. The functionality of the example project itself is secondary.

Base Entity

The project uses a general application framework that has been used to develop other applications using PostgreSQL. The base of the framework has only been altered slightly in order to get it compatible with Cloud Spanner. The most important change is the way database id’s are generated. PostgreSQL supports sequences, Cloud Spanner does not. Id’s are therefore generated by the application framework instead of by the database.

The base entity is shown in the below code snippet (some methods have been removed for brevity)

@MappedSuperclass
@Cacheable(value = true)
@BatchSize(size = 20)
public abstract class FriggEntity implements Serializable
{
	private static final long serialVersionUID = 1L;

	private static final NoArgGenerator GENERATOR = Generators.randomBasedGenerator();

	/**
	 * ID is generated from the UUID
	 */
	@Id
	@FriggDomainDescriptor(excludedFromDataPanel = true)
	private Long id;

	@Column(nullable = false, unique = true, length = 20)
	@FriggDomainDescriptor(excludedFromDataPanel = true)
	private UUID uuid;

	@FriggDomainDescriptor(defaultVisibleInDataPanel = false, defaultVisibleInAutoForm = false)
	@Column(nullable = false)
	private Date created;

	@FriggDomainDescriptor(defaultVisibleInDataPanel = false, defaultVisibleInAutoForm = false)
	@Column(nullable = false)
	private Date updated;

	protected FriggEntity()
	{
	}

	public UUID getUuid()
	{
		if (uuid == null)
			uuid = GENERATOR.generate();
		return uuid;
	}

	public void setUuid(UUID uuid)
	{
		this.uuid = uuid;
	}

	@PrePersist
	protected void onCreate()
	{
		created = new Date();
		updated = new Date();
		if (uuid == null)
			uuid = GENERATOR.generate();
		if (id == null)
			id = uuid.getMostSignificantBits() ^ uuid.getLeastSignificantBits();
	}

	@PreUpdate
	protected void onUpdate()
	{
		updated = new Date();
	}

	@Override
	public boolean equals(Object other)
	{
		if (this == other)
			return true;
		if (!(other instanceof FriggEntity))
			return false;

		return ((FriggEntity) other).getUuid().equals(this.getUuid());
	}

	@Override
	public int hashCode()
	{
		return getUuid().hashCode();
	}

	@Override
	public boolean isSaved()
	{
		return getId() != null;
	}

}

The primary key of the base entity is a long without any functional meaning.
The entity contains a UUID that is generated by the application. This makes it possible to compare entities with each other that have not yet been saved. This UUID is also used to generate the primary key value for the entity.
The entity has created/updated timestamps that are automatically filled.
The entity uses the @PrePersist and @PreUpdate annotations of JPA to generate values for id, uuid, created and updated.

Account Entity

The Account entity is an extension of the base entity that represents a user account. Once again, large parts of the code has been left out for brevity.

@Entity
@Table(indexes = { @Index(unique = true, columnList = "username") })
public class Account extends FriggEntity
{
	private static final long serialVersionUID = 1L;

	@Column(length = 30, nullable = false)
	private String username;

	@FriggDomainDescriptor(header = "Password", description = "Password", passwordField = true)
	@Transient
	private String password;

	@FriggDomainDescriptor(excludedFromDataPanel = true, defaultVisibleInAutoForm = false)
	@Column(length = 200, nullable = true)
	private String passwordHash;

	@FriggDomainDescriptor(excludedFromDataPanel = true, defaultVisibleInAutoForm = false)
	@Column(length = 200, nullable = true)
	private String salt;

	@Column(length = 30, nullable = true)
	private String activeDirectoryDomain;

	@OneToMany(fetch = FetchType.LAZY, mappedBy = "account")
	private List roles = new ArrayList();

	public Account()
	{
	}

}

The Account entity extends the base entity and adds additional columns. The entity is stored in a table with a default name (ACCOUNT), and includes a unique index on the column username. The name of the index is generated.

RoleAccount Entity

RoleAccount stores relations between accounts and roles. I consider it good practice to explicitly define these many-to-many relations as a stand alone entity, and not using a many-to-many annotation with an automatically generated relations table. You gain more control over the relation by defining it as an entity.

@Entity
@Table(indexes = { @Index(name = "IDX_ROLEACCOUNT_ACCOUNT", columnList = "account"),
		@Index(name = "IDX_ROLEACCOUNT_ROLE", columnList = "role") })
public class RoleAccount extends FriggEntity
{
	private static final long serialVersionUID = 1L;

	@FriggDomainDescriptor(defaultEditableInAutoForm = false)
	@ManyToOne(fetch = FetchType.EAGER)
	@JoinColumn(name = "account", nullable = false)
	private Account account;

	@ManyToOne(fetch = FetchType.EAGER)
	@JoinColumn(name = "role", nullable = false)
	private Role role;

	public RoleAccount()
	{
	}

	public RoleAccount(Account account, Role role)
	{
		setAccount(account);
		setRole(role);
	}

}

The properties account and role have been annotated with @JoinColumn. Normally, this would lead to the generation of a table with two foreign key constraints. Google Cloud Spanner does however not support traditional foreign key constraints, and these are therefore also not generated.

Google Cloud Spanner does support Interleaved Tables (https://cloud.google.com/spanner/docs/schema-and-data-model#creating_interleaved_tables). Interleaved tables are never generated by the schema generation of the Google Cloud Spanner Hibernate dialect. If you want a schema using interleaved tables, you will have to create that part of the schema manually.

Getting the Project

The example project is a multi-module Maven project. It also depends on another multi-module Maven project (General Application Framework, gaf, https://github.com/olavloite/gaf). You should get both from GitHub and import them into your IDE.

https://github.com/olavloite/github-monitor

https://github.com/olavloite/gaf

Bulk Upload in Google Cloudspanner

Knut Olav Løite — Sat, 06 May 2017 00:00:00 GMT

Google Cloud Spanner is Google’s horizontally scalable, globally consistent, relational database service. Cloud Spanner claims to be the first and only relational database service that is both strongly consistent and horizontally scalable. Cloud Spanner supports transactions and SQL queries and could in theory be used with any application currently running on other relational databases. This tutorial shows how you could convert an existing relational database and bulk upload all data to a new Google Cloud Spanner database using this open source JDBC driver.

Topics

This tutorial contains the following topics: . Convert an existing relational database to Cloud Spanner . Bulk upload data to Cloud Spanner using multiple parallel workers . Connect to a Cloud Spanner database with JDBC and use the following features (not supported by the JDBC driver supplied by Google): .. Transactions .. DDL statements (CREATE TABLE) .. DML statements (INSERT INTO, DELETE FROM)

How it Works

This bulk upload example uses JDBC to connect to both the source (in this example: PostgreSQL) and destination Google Cloud Spanner database. The JDBC driver supplied by Google does not support DDL and DML statements, so we use this JDBC driver instead: https://github.com/olavloite/spanner-jdbc

Table conversion

The converter reads all tables in the source database and generates Google Cloud Spanner compliant DDL statements (CREATE TABLE…) for the creation of the tables. The converter can be run in both SkipExisting and DropAndRecreate mode. It defaults to SkipExisting. The table creation process runs as a single thread process in autocommit mode. The main code can be found in TableConverter .

Data conversion

The converter then iterates over all the tables in the source database and copies the data from the source to the destination. This is done by generating DML-statements (INSERT INTO…) for each row. Whether this process is executed single-threaded or multi-threaded using multiple workers, depends on the size of the table. It defaults to a threshold of 1,000 records. If the source table contains more records, the upload will be handled by multiple workers. This example project defaults to 10 upload workers running in parallel. Each worker opens a new connection to Cloud Spanner and runs its own transactions. Transactions are not supported by the JDBC driver supplied by Google, but the JDBC driver used by this example does.

Step 1: Create a Cloud Spanner test database

This tutorial assumes that you already have a Google Cloud Spanner account and an empty test database. If you do not, and you want to know how to get it, follow the steps in one of my other tutorials: https://olavloite.github.io/2017/03/11/Google-Cloud-Spanner-with-Spring-Boot-JPA-and-Hibernate.html

Step 2: Get and Run the Example Project

I have created a sample project that will be used for this tutorial. Clone the project from https://github.com/olavloite/spanner-jdbc-converter . It is a Maven project with a dependency on a JDBC Driver for Google Cloud Spanner. Then follow these steps:

Find the file converter.properties.example in the root of the project and make a copy of this and name the copy converter.properties.
Have a look at converter.properties. It contains three entries:
1. TableConverter.convertMode=SkipExisting: This indicates that existing tables in the Cloud Spanner database should not be dropped and recreated, but skipped during conversion. If you want to drop and recreate tables, change the value to DropAndRecreate
2. DataConverter.convertMode=SkipExisting: Same as TableConverter.convertMode, but now for the data in the table. If SkipExisting is set, no data will be converted if the destination table contains 1 or more records (no data comparison will be done). If set to DropAndRecreate, all data in the destination table will always be deleted and recreated from the source database.
3. TableConverter.specificColumnMapping.uuid=BYTES(16): Using this configuration you can specify a specific data type mapping for specific columns. The converter contains a default list of data type conversion which can be overridden with this property. This mapping specifies that all columns with the NAME (not data type) uuid should be converted to BYTES(16). If you want a specific mapping for one column of one table, you should specify it as TableConverter.specificColumnMapping.tableName.columnName
Run the Java application (the main class is Converter.class). It takes the following three arguments:
The JDBC connection string of the source database. In my case this is "jdbc:postgresql://localhost:5432/databaseName?user=username&password=secret"
The JDBC connection string of the destination (Cloud Spanner) database. In my case this is "jdbc:cloudspanner://localhost;Project=test-jdbc-161317;Instance=converted;Database=databaseName;PvtKeyPath=pathToKeyFile.json"
The path to the converter.properties file. If you run the test application in Eclipse and keep the converter.properties file in the root of the project, the value of this argument is simply "converter.properties"

The application will scan your source database for all tables and then try to create these in your Cloud Spanner database. After creating all tables, the application will convert the data of the converted tables. Data conversion is done using two strategies, depending on the size of the table: . Tables with 1,000 or less records will be converted using a simple single threaded conversion process. All records will be converted in one transaction. . Tables with more than 1,000 records are converted by 10 parallel workers. Each worker is assigned a proportionate number of the records in the table to convert. The workers then convert the data in batches of 1,000 records, each batch being a single transaction.

The code snippet below shows the main part of the worker. Note how the worker uses standard JDBC functionality to access Cloud Spanner. Both transactions and statements are standard JDBC calls.

@Override
public void run()
{
    try (Connection source = DriverManager.getConnection(urlSource);
            Connection destination = DriverManager.getConnection(urlDestination))
    {
        log.info(name + ": " + sourceTable + ": Starting copying " + numberOfRecordsToCopy + " records");

        destination.setAutoCommit(false);
        String sql = "INSERT INTO " + destinationTable + " (" + cols.getColumnNames() + ") VALUES \n";
        sql = sql + "(" + cols.getColumnParameters() + ")";
        PreparedStatement statement = destination.prepareStatement(sql);

        int lastRecord = beginOffset + numberOfRecordsToCopy;
        int recordCount = 0;
        int currentOffset = beginOffset;
        while (true)
        {
            int limit = Math.min(batchSize, lastRecord - currentOffset);
            String select = selectFormat.replace("$COLUMNS", cols.getColumnNames());
            select = select.replace("$TABLE", sourceTable);
            select = select.replace("$PRIMARY_KEY", cols.getPrimaryKeyColumns());
            select = select.replace("$BATCH_SIZE", String.valueOf(limit));
            select = select.replace("$OFFSET", String.valueOf(currentOffset));
            try (ResultSet rs = source.createStatement().executeQuery(select))
            {
                while (rs.next())
                {
                    int index = 1;
                    for (Integer type : cols.columnTypes)
                    {
                        Object object = rs.getObject(index);
                        statement.setObject(index, object, type);
                        index++;
                    }
                    if (useJdbcBatching)
                        statement.addBatch();
                    else
                        statement.executeUpdate();
                    recordCount++;
                }
                if (useJdbcBatching)
                    statement.executeBatch();
            }
            destination.commit();
            log.info(name + ": " + sourceTable + ": Records copied so far: " + recordCount + " of "
                    + numberOfRecordsToCopy);
            currentOffset = currentOffset + batchSize;
            if (recordCount >= numberOfRecordsToCopy)
                break;
        }
    }
    catch (SQLException e)
    {
        log.severe("Error during data copy: " + e.getMessage());
        throw new RuntimeException(e);
    }
    log.info(name + ": Finished copying");
}

Also note that transactions are committed after each batch and not after copying the entire table. This is not a programming error, but a necessity as Google Cloud Spanner does not allow transactions to contain more than 20,000 mutations. A mutation of one row with five columns counts as five mutations.

Summary

The Google Cloud Spanner JDBC driver allows you to work with Cloud Spanner as it was (almost) any other JDBC compliant relational database, including DDL- and DML-statements, (prepared) JDBC statements and transactions. Cloud Spanner itself has some limitations when it comes to bulk update statements. Insert and update statements can only operate on one row at a time. The JDBC driver can also be used to develop applications using JPA / Hibernate in combination with Google Cloud Spanner. An example can be found here: https://olavloite.github.io/2017/03/11/Google-Cloud-Spanner-with-Spring-Boot-JPA-and-Hibernate.html

Tutorial: Use Google Cloud Spanner with Spring Boot, JPA and Hibernate

Knut Olav Løite — Sat, 11 Mar 2017 00:00:00 GMT

Google Cloud Spanner is Google’s horizontally scalable, globally consistent, relational database service. Cloud Spanner claims to be the first and only relational database service that is both strongly consistent and horizontally scalable. Cloud Spanner supports transactions and SQL queries and could in theory be used with any application currently running on other relational databases. This tutorial will guide you through the steps needed to set up a test environment for developing a Spring Boot application with Cloud Spanner as its data source using JPA/Hibernate.

Step 1: Create a Cloud Spanner test database

Google kindly offers a $300 in free credit for trying out Cloud Spanner. Go to https://cloud.google.com/spanner/ and click on Try It Free to create a test instance. Go through the registration process and then follow these steps to create a Cloud Spanner instance and database:

If you register for the first time with Google Cloud, Google automatically creates a project for you with the title 'My first project' and a generated project id. Rename this project to something like cloudspanner-test. If you already have a Google Cloud account, you can add a new project.
Click on the main menu button in the top left corner next to the text Google Cloud Platform and select Spanner. Google will now initialize Cloud Spanner for your project.
Create an instance with these values:
- Instance name: Test Instance
- Instance id: test-instance
- Configuration: your choice
- Nodes: 1
Create a database with these values:
- Database name: test-database
- Tables: Do not create any tables now
Now we will create a service account that you can use to access the database remotely. Click on the main menu button in the top left corner and select IAM & Admin.
Click on Service accounts in the menu on the left.
Click on Create Service Account on top of the page.
Name the service account spanner and select the role Cloud Spanner | Cloud Spanner Admin. Turn on the option Furnish a private key and choose JSON. This key file will be downloaded to your computer and you will need it to access the database remotely. Click on Create.

That’s it! You have now created the Google Cloud Spanner database needed for this tutorial.

Step 2: Get and Run the Example Project

I have created a sample project that will be used for this tutorial. Clone the project from https://github.com/olavloite/spanner-jpa-example . It is a Maven project with a dependency on a JDBC Driver for Google Cloud Spanner and on a Hibernate Dialect library. Make sure to get these dependencies. They are available on Maven Central. Then follow these steps:

Find the file application.properties.example in the root of the project and make a copy of this and name the copy application.properties.
Open application.properties and change the value of spring.datasource.url into the following string. Make sure you change the placeholder [project-id] with the project id generated by Google (this is NOT the project name). Also change the placeholder [path-to-keyfile] with the actual location on your local disk of the JSON keyfile generated in the steps above:
```
jdbc:cloudspanner://localhost;Project=[project-id];Instance=test-instance;Database=test-database;PvtKeyPath=[path-to-keyfile]
```
Run the Java application (the main class is Application.class). The application is configured to automatically create the tables needed for the entities that are defined in the source code. This example project contains two entities (Customer and Invoice), and these tables will be created. Note that this process can take some time as Cloud Spanner is not very quick when it comes to DDL.
The application will then try to insert and then select some records into Cloud Spanner

You can try to experiment with additional entities and also try to update some of them by changing the code. Please note that the DDL statement support of Google Cloud Spanner is somewhat limited. Adding nullable columns to existing tables and adding new tables should work without any hassle. More complex changes, such as changing an existing column, might produce errors and need to be changed manually through the web interface of Google Cloud Spanner.