Tuesday, May 17, 2016

Spring Cloud with Turbine AMQP

I have previously blogged about using Spring Cloud with Turbine, a Netflix OSS library which provides a way to aggregate the information from Hystrix streams across a cluster.

The default aggregation flow is however pull-based, where Turbine requests the hystrix stream from each instance in the cluster and aggregates it together - this tends to be way more configuration heavy.


Spring Cloud Turbine AMQP offers a different model, where each application instance pushes the metrics from Hystrix commands to Turbine through a central RabbitMQ broker.


This blog post recreates the sample that I had configured previously using Spring Cloud support for AMQP - the entire sample is available at my github repo if you just want the code.

The changes are very minor for such a powerful feature, all the application which wants to feed the hystrix stream to an AMQP broker is to add these dependencies expressed in maven the following way:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-hystrix</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-netflix-hystrix-stream</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-stream-rabbit</artifactId>
</dependency>


These dependencies would now auto-configure all the connectivity details with RabbitMQ sample topic exchange and would start feeding in the hystrix stream data into this RabbitMQ topic.

Similarly on the Turbine end all that needs to be done is to specify the appropriate dependencies:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-turbine-stream</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-stream-rabbit</artifactId>
</dependency>

This would consume the hystrix messages from RabbitMQ and would in turn expose an aggregated stream over an http endpoint.

Using this aggregated stream a hystrix dashboard can be displayed along these lines:


The best way to try out the sample is using docker-compose and the README with the sample explains how to build the relevant docker containers and start it up using docker-compose.

Monday, May 2, 2016

Approaches to binding a Spring Boot application to a service in Cloud Foundry

If you want to try out Cloud Foundry the simplest way to do that is to download the excellent PCF Dev or to create a trial account at the Pivotal Web Services site.

The rest of the post assumes that you have an installation of Cloud Foundry available to you and that you have a high level understanding of Cloud Foundry. The objective of this post is to list out of the options you have in integrating your Java application to a service instance - this demo uses mysql as a sample service to integrate with but the approach is generic enough.

Overview of the Application

The application is fairly simple Spring-Boot app, it is a REST service exposing three domain types and their relationships, representing a university - Course, Teacher and Student. The domain instances are persisted to a MySQL database. The entire source code and the approaches are available at this github location if you want to jump ahead.

To try the application locally, first install a local mysql server database, on a Mac OSX box with homebrew available, the following set of commands can be run:

brew install mysql

mysql.server start
mysql -u root
# on the mysql prompt: 

CREATE USER 'univadmin'@'localhost' IDENTIFIED BY 'univadmin';
CREATE DATABASE univdb;
GRANT ALL ON univdb.* TO 'univadmin'@'localhost';

Bring up the Spring-Boot under cf-db-services-sample-auto:

mvn spring-boot:run

and an endpoint with a sample data will be available at http://localhost:8080/courses.

Trying this application on Cloud Foundry

If you have an installation of PCF Dev running locally, you can try out a deployment of the application the following way:

cf api api.local.pcfdev.io --skip-ssl-validation
cf login # login with admin/admin credentials

Create a Mysql service instance:
cf create-service p-mysql 512mb mydb

and push the app! (manifest.yml provides the binding of the app to the service instance)
cf push

An endpoint should be available at http://cf-db-services-sample-auto.local.pcfdev.io/courses

Approaches to service connectivity


Now that we have an application that works locally and on a sample local Cloud Foundry, these are the approaches to connecting to a service instance.

Approach 1 - Do nothing, let the Java buildpack handle the connectivity details


This approach is demonstrated in the cf-db-services-sample-auto project. Here the connectivity to the local database has been specified using Spring Boot and looks like this:

---

spring:
  jpa:
    show-sql: true
    hibernate.ddl-auto: none
    database: MYSQL

  datasource:
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://localhost/univdb?autoReconnect=true&useSSL=false
    username: univadmin
    password: univadmin

When this application is pushed to Cloud Foundry using the Java Buildpack, a component called the java-buildpack-auto-reconfiguration is injected into the application which reconfigures the connectivity to the service based on the runtime service binding.


Approach 2 - Disable Auto reconfiguration and use runtime properties

This approach is demonstrated in the cf-db-services-sample-props project. When a service is bound to an application, there is a set of environment properties injected into the application under the key "VCAP_SERVICES". For this specific service the entry looks something along these lines:

"VCAP_SERVICES": {
  "p-mysql": [
   {
    "credentials": {
     "hostname": "mysql.local.pcfdev.io",
     "jdbcUrl": "jdbc:mysql://mysql.local.pcfdev.io:3306/cf_456d9e1e_e31e_43bc_8e94_f8793dffdad5?user=**\u0026password=***",
     "name": "cf_456d9e1e_e31e_43bc_8e94_f8793dffdad5",
     "password": "***",
     "port": 3306,
     "uri": "mysql://***:***@mysql.local.pcfdev.io:3306/cf_456d9e1e_e31e_43bc_8e94_f8793dffdad5?reconnect=true",
     "username": "***"
    },
    "label": "p-mysql",
    "name": "mydb",
    "plan": "512mb",
    "provider": null,
    "syslog_drain_url": null,
    "tags": [
     "mysql"
    ]
   }
  ]
 }

The raw json is a little unwieldy to consume, however Spring Boot automatically converts this data into a flat set of properties that looks like this:

"vcap.services.mydb.plan": "512mb",
"vcap.services.mydb.credentials.username": "******",
"vcap.services.mydb.credentials.port": "******",
"vcap.services.mydb.credentials.jdbcUrl": "******",
"vcap.services.mydb.credentials.hostname": "******",
"vcap.services.mydb.tags[0]": "mysql",
"vcap.services.mydb.credentials.uri": "******",
"vcap.services.mydb.tags": "mysql",
"vcap.services.mydb.credentials.name": "******",
"vcap.services.mydb.label": "p-mysql",
"vcap.services.mydb.syslog_drain_url": "",
"vcap.services.mydb.provider": "",
"vcap.services.mydb.credentials.password": "******",
"vcap.services.mydb.name": "mydb",

Given this, the connectivity to the database can be specified in a Spring Boot application the following way - in a application.yml file:

spring:
  datasource:
    url: ${vcap.services.mydb.credentials.jdbcUrl}
    username: ${vcap.services.mydb.credentials.username}
    password: ${vcap.services.mydb.credentials.password}

One small catch though is that since I am now explicitly taking control of specifying the service connectivity, the runtime java-buildpack-auto-reconfiguration has to be disabled, which can done by a manifest metadata:
---
applications:
  - name: cf-db-services-sample-props
    path: target/cf-db-services-sample-props-1.0.0.RELEASE.jar
    memory: 512M
    env:
      JAVA_OPTS: -Djava.security.egd=file:/dev/./urandom
      SPRING_PROFILES_ACTIVE: cloud
    services:
      - mydb

buildpack: https://github.com/cloudfoundry/java-buildpack.git

env:
    JBP_CONFIG_SPRING_AUTO_RECONFIGURATION: '{enabled: false}'

Approach 3 - Using Spring Cloud Connectors

The third approach is to use the excellent Spring Cloud Connectors project and a configuration which specifies a service connectivity looks like this and is demonstrated in the cf-db-services-sample-connector sub-project:

@Configuration
@Profile("cloud")
public  class CloudFoundryDatabaseConfig {

    @Bean
    public Cloud cloud() {
        return new CloudFactory().getCloud();
    }

    @Bean
    public DataSource dataSource() {
        DataSource dataSource = cloud().getServiceConnector("mydb", DataSource.class, null);
        return dataSource;
    }
}

Pros and Cons



These are the Pros and Cons with each of these approaches:

Approaches Pros Cons
Approach 1 - Let Buildpack handle it
1. Simple, the application that works locally will work without any changes on the cloud

1. Magical - the auto-reconfiguration may appear magical to someone who does not understand the underlying flow
2. The number of service types supported is fairly limited -
say for eg, if a connectivity is required to Cassandra then Auto-reconfiguration will not work
Approach 2 - Explicit Properties 1. Fairly straightforward.
2. Follows the Spring Boot approach and uses some of the best practices of Boot based applications - for eg, there is a certain order in which datasource connection pools are created, all those best practices just flow in using this approach.
1. The Auto-reconfiguration will have to be explicitly disabled
2. Need to know what the flattened properties look like
3. A "cloud" profile may have to be manually injected through environment properties to differentiate local development and cloud deployment
4. Difficult to encapsulate reusability of connectivity to newer service types - say Cassandra or DynamoDB.
Approach 3 - Spring Cloud Connectors 1. Simple to integrate
2. Easy to add in re-usable integration to newer service types
1. Bypasses the optimizations of Spring Boot connection pool logic.

Conclusion


My personal preference is to go with Approach 2 as it most closely matches the Spring Boot defaults, not withstanding the cons of the approach. If more complicated connectivity to a service is required I will likely go with approach 3. Your mileage may vary though


References

1. Scott Frederick's spring-music has been a constant guide.
2. I have generously borrowed from Ben Hale's pong_matcher_spring sample.

Tuesday, April 26, 2016

Scatter-Gather using Spring Reactor Core

I have a good working experience in using the Netflix Rx-Java libraries and have previously blogged about using Rx-Java and Java 8 CompletableFuture for a scatter-gather kind of problems. Here I want to explore applying the same pattern using the Spring Reactor Core library.

tldr - If you are familiar with Netflix Rx-Java, you already know Spring Reactor Core, the API's map beautifully and I was thrilled to see that the Spring Reactor team has diligently used Marble diagrams in their Javadoc API's

Another quick point is that rx.Observable maps to Flux or Mono based on whether many items are being emitted or whether one or none is being emitted.

With this let me directly jump into the sample - I have a simple task(simulated using a delay) that is spawned a few times, I need to execute these tasks concurrently and then collect back the results, represented the following way using a rx.Observable code:

@Test
public void testScatterGather() throws Exception {
    ExecutorService executors = Executors.newFixedThreadPool(5);

    List<Observable<String>> obs =
            IntStream.range(0, 10)
                .boxed()
                .map(i -> generateTask(i, executors)).collect(Collectors.toList());


    Observable<List<String>> merged = Observable.merge(obs).toList();
    List<String> result = merged.toBlocking().first();

    logger.info(result.toString());

}

private Observable<String> generateTask(int i, ExecutorService executorService) {
    return Observable
            .<String>create(s -> {
                Util.delay(2000);
                s.onNext( i + "-test");
                s.onCompleted();
            }).subscribeOn(Schedulers.from(executorService));
}


Note that I am blocking purely for the test.

Now, a similar code using Spring Reactor Core translates to the following:

@Test
public void testScatterGather() {
    ExecutorService executors = Executors.newFixedThreadPool(5);

    List<Flux<String>> fluxList = IntStream.range(0, 10)
            .boxed()
            .map(i -> generateTask(executors, i)).collect(Collectors.toList());

    Mono<List<String>> merged = Flux.merge(fluxList).toList();

    List<String> list = merged.get();

    logger.info(list.toString());


}

public Flux<String> generateTask(ExecutorService executorService, int i) {
    return Flux.<String>create(s -> {
        Util.delay(2000);
        s.onNext(i + "-test");
        s.onComplete();
    }).subscribeOn(executorService);
}

It more or less maps one to one. A small difference is in the Mono type, I personally felt that this type was a nice introduction to the reactive library as it makes it very clear whether more than 1 item is being emitted vs only a single item which I have made use of in the sample.

These are still early explorations for me and I look forward to getting far more familiar with this excellent library.

Monday, April 18, 2016

First steps to Spring Boot Cassandra

If you want to start using Cassandra NoSQL database with Spring Boot, the best resource is likely the Cassandra samples available here and the Spring data Cassandra documentation.

Here I will take a little more roundabout way, by actually installing Cassandra locally and running a basic test against it and I aim to develop this sample into a more comprehensive example with the next blog post.

Setting up a local Cassandra instance

Your mileage may vary, but the simplest way to get a local install of Cassandra running is to use the Cassandra cluster manager(ccm) utility, available here.

ccm create test -v 2.2.5 -n 3 -s

Or a more traditional approach may simply be to download it from the Apache site. If you are following along, the version of Cassandra that worked best for me is the 2.2.5 one.

With either of the above, start up Cassandra, using ccm:

ccm start test

or with the download from the Apache site:

bin/cassandra -f

The -f flag will keep the process in the foreground, this way stopping the process will be very easy once you are done with the samples.

Now connect to this Cassandra instance:

bin/cqlsh

and create a sample Cassandra keyspace:

CREATE KEYSPACE IF NOT EXISTS sample WITH replication = {'class':'SimpleStrategy', 'replication_factor':1};

Using Spring Boot Cassandra


Along the lines of anything Spring Boot related, there is a starter available for pulling in all the relevant dependencies of Cassandra, specified as a gradle dependency here:

compile('org.springframework.boot:spring-boot-starter-data-cassandra')

This will pull in the dependencies that trigger the Auto-configuration for Cassandra related instances - a Cassandra session mainly.

For the sample I have defined an entity called the Hotel defined the following way:

package cass.domain;

import org.springframework.data.cassandra.mapping.PrimaryKey;
import org.springframework.data.cassandra.mapping.Table;

import java.io.Serializable;
import java.util.UUID;

@Table("hotels")
public class Hotel implements Serializable {

    private static final long serialVersionUID = 1L;

    @PrimaryKey
    private UUID id;

    private String name;

    private String address;

    private String zip;

    private Integer version;

    public Hotel() {
    }

    public Hotel(String name) {
        this.name = name;
    }

    public UUID getId() {
        return id;
    }

    public String getName() {
        return this.name;
    }

    public String getAddress() {
        return this.address;
    }

    public String getZip() {
        return this.zip;
    }

    public void setId(UUID id) {
        this.id = id;
    }

    public void setName(String name) {
        this.name = name;
    }

    public void setAddress(String address) {
        this.address = address;
    }

    public void setZip(String zip) {
        this.zip = zip;
    }

    public Integer getVersion() {
        return version;
    }

    public void setVersion(Integer version) {
        this.version = version;
    }

}

and the Spring data repository to manage this entity:

import cass.domain.Hotel;
import org.springframework.data.repository.CrudRepository;

import java.util.UUID;

public interface HotelRepository extends CrudRepository<Hotel, UUID>{}


A corresponding cql table is required to hold this entity:

CREATE TABLE IF NOT EXISTS  sample.hotels (
    id UUID,
    name varchar,
    address varchar,
    zip varchar,
    version int,
    primary key((id))
);


That is essentially it, Spring data support for Cassandra would now manage all the CRUD operations of this entity and a test looks like this:

import cass.domain.Hotel;
import cass.repository.HotelRepository;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.SpringApplicationConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

import java.util.UUID;

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.equalTo;

@RunWith(SpringJUnit4ClassRunner.class)
@SpringApplicationConfiguration(classes = SampleCassandraApplication.class)
public class SampleCassandraApplicationTest {

 @Autowired
 private HotelRepository hotelRepository;

 @Test
 public void repositoryCrudOperations() {
  Hotel sample = sampleHotel();
  this.hotelRepository.save(sample);

  Hotel savedHotel = this.hotelRepository.findOne(sample.getId());

  assertThat(savedHotel.getName(), equalTo("Sample Hotel"));

  this.hotelRepository.delete(savedHotel);
 }

 private Hotel sampleHotel() {
  Hotel hotel = new Hotel();
  hotel.setId(UUID.randomUUID());
  hotel.setName("Sample Hotel");
  hotel.setAddress("Sample Address");
  hotel.setZip("8764");
  return hotel;
 }

}

Here is the github repo with this sample. There is not much to this sample yet, in the next blog post I will enhance this sample to account for the fact that it is very important to understand the distribution of data across a cluster in a NoSQL system and how the entity like Hotel here can be modeled for efficient CRUD operations.

Saturday, March 26, 2016

Single Page Angularjs application with Spring Boot and Yeoman

I am very thankful for tools like yeoman which provide a very quick way to combine different javascript libraries together into a coherent application. Yeoman provides the UI tier, if you needed to develop the services tier and a web layer for the static assets a good way to package it is to use Spring Boot. I know there are tools like JHipster which make this easy but if you are just looking just a basic template what I am outlining here should just suffice.

So this is what I do, let us start by getting a basic Spring boot web template in place, the following way:


spring init --dependencies=web spring-boot-static-sample

This assumes that you have the command line application for Spring Boot available in your machine, if you don't then just follow the instructions here

There should be a folder called spring-boot-static-sample with all the Spring Boot generated code in there, now to layer in the static content in there, I have used the yeoman gulp angular generator to generate the code, the following way inside the spring-boot-static-sample folder:

npm install -g yo gulp bower
npm install -g generator-gulp-angular
yo gulp-angular

Almost there, just modify one of the gulp configurations - instead of creating the packaged javascript distribution to dist folder, let the folder be src/main/resources/static instead. In gulp/conf.js:


This is the folder that Spring boot uses to serve out static content by default.

And that's it, when you are developing the single page apps this can be done very quickly using the convenient gulp commands

gulp serve

and when you are ready to package the application just run

gulp build

which would get the static content into a location that Spring boot understands and then run the app:

mvn spring-boot:run

and the Single page app UI should show up.



Simple and clean!

Here is a sample project with these steps already executed - https://github.com/bijukunjummen/spring-boot-static-sample


Saturday, March 12, 2016

STEM project programming Language Choice - Scala

My daughter Sara who is in 3rd grade and I recently worked together on a school initiated STEM (Science, Technology, Engineering and Math) project and decided to do a programming challenge. I had recently got the excellent Elements book for her  and we wanted to connect the project to the Chemical Elements in some way. An idea that came to us was to try and combine the symbol of the elements and form words from them. So for eg. if you combine Phosphorus(P), Indium(In), Carbon(C), Hydrogen(H) you get PInCH. On an initial whim this sounded like an interesting and at the same time challenging enough problem for us to attempt.

Now our attention turned to the choice of programming language. I tried all the usual Educational programming environments and somehow programming pictorially or graphically did not appeal to me and I felt that it would be good to expose her to some basic programming skills. We together looked at Javascript as a possible choice, Python, Ruby, finally even Java and Scala. I am not used to programming in Python and Ruby, so that left Javascript, Java and Scala, after a little bit of more deliberation we decided to go with Scala, mainly because of the "Worksheet" support in IntelliJ which allows for small snippets of the program to be tried out without too much ceremony needing to be put in place. I would have been way more comfortable with Java as I use Java professionally, however I felt that it would be easier to explain the concepts with less verbosity to a third grader with Scala.



With the choice of language in place I was able to show her some basics of programming with Scala - declaring variables, simple functions, some basics of data types, basics of collections and simple ways to map collections.

We then jumped into the program itself, the approach we outlined and wrote was simple enough - we needed a dictionary of words, we needed a list of elements and something to generate the words by combining elements and filter them using the dictionary. The final code is fairly easy to navigate. We have avoided concepts like recursion, any complicated data structures to validate words and stuck to simple iteration to generate the words.


We found about 12000 words altogether, these are some that caught our eye:



The revelation for me though has been in how easily my daughter picked up on the Programming language concepts with a seemingly difficult language like Scala. I know there are more difficult concepts along the way once we get past the basics, however my choice of Scala was to ensure that the foundation is strong enough. I will let her explore more esoteric features by herself when she works on her STEM project for next year !

Here is the code that we came up with.


Monday, February 22, 2016

Spring Boot with Scala

A while back I had tried out a small Spring Boot based sample with Scala as the language and found that the combination works out quite nicely - no big surprises there actually as Scala programs ultimately run in the JVM. I have now updated the sample with the latest version of Spring Boot and some of the supporting libraries.

To very quickly revisit the sample, it is a very simple web application with a UI to manage a "Hotel" domain object managed via JPA, represented in scala the following way:

import javax.persistence.Id
import javax.persistence.GeneratedValue
import java.lang.Long
import javax.persistence.Entity
import scala.beans.BeanProperty
import org.hibernate.validator.constraints.NotEmpty

@Entity
class Hotel {

  @Id
  @GeneratedValue
  @BeanProperty
  var id: Long = _

  @BeanProperty
  @NotEmpty
  var name: String = _

  @BeanProperty
  @NotEmpty
  var address: String = _

  @BeanProperty
  @NotEmpty
  var zip: String = _
}

JPA annotations carry over quite well, one wrinkle may be the additional @BeanProperty annotation though, this is required for JPA implementations as this makes the scala compiler generate the normal Java Beans type getters and setters instead of the scala default getters and setters which don't follow the Java Bean conventions.

Spring Data makes it ridiculously simple to manage this domain type, all it requires is a marker interface and it generates a runtime implementation:

import org.springframework.data.repository.CrudRepository
import mvctest.domain.Hotel
import java.lang.Long

trait HotelRepository extends CrudRepository[Hotel, Long]

Now I have a toolkit available for managing the Hotel domain:

//save or update a hotel
hotelRepository.save(hotel)

//find one hotel
hotelRepository.findOne(id)

//find all hotels
val hotels = hotelRepository.findAll()

//delete a hotel
hotelRepository.delete(id)


And finally a controller to manage the UI flow with this repository:

@Controller
@RequestMapping(Array("/hotels"))
class HotelController @Autowired()(private val hotelRepository: HotelRepository) {

  @RequestMapping(method = Array(RequestMethod.GET))
  def list(model: Model) = {
    val hotels = hotelRepository.findAll()
    model.addAttribute("hotels", hotels)
    "hotels/list"
  }

  @RequestMapping(Array("/edit/{id}"))
  def edit(@PathVariable("id") id: Long, model: Model) = {
    model.addAttribute("hotel", hotelRepository.findOne(id))
    "hotels/edit"
  }

  @RequestMapping(method = Array(RequestMethod.GET), params = Array("form"))
  def createForm(model: Model) = {
    model.addAttribute("hotel", new Hotel())
    "hotels/create"
  }

  @RequestMapping(method = Array(RequestMethod.POST))
  def create(@Valid hotel: Hotel, bindingResult: BindingResult) = {
    if (bindingResult.hasErrors()) {
      "hotels/create"
    } else {
      hotelRepository.save(hotel)
      "redirect:/hotels"
    }
  }

  @RequestMapping(value = Array("/update"), method = Array(RequestMethod.POST))
  def update(@Valid hotel: Hotel, bindingResult: BindingResult) = {
    if (bindingResult.hasErrors()) {
      "hotels/edit"
    } else {
      hotelRepository.save(hotel)
      "redirect:/hotels"
    }
  }

  @RequestMapping(value = Array("/delete/{id}"))
  def delete(@PathVariable("id") id: Long) = {
    hotelRepository.delete(id)
    "redirect:/hotels"
  }
}

There are some wrinkles here too but should mostly make sense, the way the repository is autowired is a little non-intuitive and the way an explicit Array type has to be provided for request mapping paths and methods may be confusing.

Beyond these small concerns the code just works, do play with it, I would love any feedback on ways to improve this sample. Here is the git location of this sample - https://github.com/bijukunjummen/spring-boot-scala-web