Setup Jenkins and DevTools on Ubuntu on AWS EC2


Install Git, Java7, Java 8 and Maven

sudo apt-get install git

sudo apt-add-repository ppa:webupd8team/java

sudo apt-get update

sudo apt-get install oracle-java8-installer

sudo apt-get install oracle-java7-installer

sudo apt-get install oracle-java8-set-default

sudo apt-get install maven


Install Jenkins

sudo wget -q -O – | sudo apt-key add –
sudo sh -c ‘echo deb binary/ > /etc/apt/sources.list.d/jenkins.list’
sudo apt-get update
sudo apt-get install jenkins


Install MongoDB

sudo apt-key adv –keyserver hkp:// –recv 7F0CEB10

echo “deb “$(lsb_release -sc)”/mongodb-org/3.0 multiverse” | sudo tee /etc/apt/sources.list.d/mongodb-org-3.0.list

sudo apt-get update

sudo apt-get install -y mongodb-org

sudo apt-get install -y mongodb-org=3.0.4 mongodb-org-server=3.0.4 mongodb-org-shell=3.0.4 mongodb-org-mongos=3.0.4 mongodb-org-tools=3.0.4

sudo service mongod start

sudo tail -5000 /var/log/mongodb/mongod.log


Install Postgresql

sudo apt-get update
sudo apt-get install postgresql postgresql-contrib


Install Redis

sudo apt-get install build-essential

sudo apt-get install tcl8.5

sudo wget

sudo tar xzf redis-stable.tar.gz

cd redis-stable


make test

sudo make install

cd utils

sudo ./

sudo service redis_6379 start


Install Scala

sudo wget

sudo dpkg -i scala-2.11.7.deb


Install SBT and Gradle

echo “deb /” | sudo tee -a /etc/apt/sources.list.d/sbt.list
sudo apt-get update
sudo apt-get install sbt

install typesafe activator

sudo wget


sudo apt-get install unzip

curl -s | bash

source “/root/.gvm/bin/”

gvm install gradle



Setup back for jenkins




Add 8080 to all same ports as 80




AWS Summit 2015 Slides of Interest

AWS Security

Cost Optimization


AWS EC2 Contianers (DOCKER)

Introduction to EC2



AWS Elasticache

AWS Lambda


Microservices update

NOSQL (MongoDB) Injection


Streaming Large JSON Files with Java 8

As expected, the main options are the two big guys in JSON parsing,  GSON from Google and FasterXML Jackson 2.   Both are very solid and fast, but there’s also a couple of other libraries worth trying.

If you use Jackson, use the latest, FasterXML Jackson 2.  Not Codehaus.

Spring Cloud Configuration

Monday Cool Links

Monitoring and Metrics



Servers and Services

DevOps Techniques, Tools and Processes



Big Data

Microservices Top 20 Links – 2015


Why 12 Factor Application Patterns, Microservices and CloudFoundry Matter? Part 2 – What to change and how.

I first described a difficult scenario that I faced a few years ago.   This article is my recommendations to the current custodians of that project on what to change and how.

With the help of open source frameworks like Spring Cloud and open source platforms like Cloudfoundry, you aren’t forced into having a fragile, manual SDLC.   The first step is to embrace change and join a vibrant community that is succeeding in software development.



  • Increase agility through process, tools and culture
  • Examine the code and depedencies, determine modules based on business domain.
  • Refactor the monolithic EAR into microservices living in JARs guided by the 12 Factor Application patterns and microservices recommendations.
  • 12FA – III – Store configuration in the environment (or even better use a configuration server)
  • Change the Development Process, Frameworks, Libraries and Tools.
  • Change the Build Process and Tools.  Embrace Gradle, Maven, JDK 8.
  • Change the Delivery Process.   Enact Continuous delivery.
  • Change the Database modifications and migration Process and Tools.
  • Change the run-time platform from legacy application server to a PaaS.
  • Centralize the logging.    Using PCF Loggregator and tools like ELK and PaperTrail.
  • Embrace industry standard open source tools, libraries and frameworks
  • Good code fails when you don’t have good process and a platform to help you.
  • Good teams fail when you don’t have a good culture that embraces DevOps, microservices and not giant monoliths.


So how could we have improved our process?   The first step is to admin you have a problem and seek help.   By looking to the community, open source and successful startups that have faced similar struggles with monolithic J2EE code bases (Ebay, Netflix), we can determine some functional patterns.


Patterns and Principles for Change

  • Use Jenkins to deploy your app with no downtime to CloudFoundry.
  • Use CloudFoundry deployments to do A/B testing with no downtime or difficulty.
  • Bind to database, using the same process in development, test, Q/A and production.   No manual configuration or configuration changes.   Operators make available the correct database settings through the marketplace.  Same manifest and bindings will point to different databases in different spaces.
  • To migrate database changes use flywaydb or liquibase.   These are very well integrated into Spring Boot.
  • Look at NoSQL for part of your database workload, polyglot persistence makes sense in many cases.   For MongoDB development, you handle change management with Mongeez, which is automatically done with JHipster.
  • Smoke tests and automated tests tied to Jenkins.
  • Test against CloudFoundry development for local development, fast deployments, run unit and integration tests locally.
  • Use Maven or Gradle for builds, not ANT.
  • Use Circuit breakers in your web calls to avoid breaking systems and fail safely.
  • Use APM and JMX for deep checks using Pivotal Ops Metrics for CloudFoundry.
  • Have automated scripts check one log stream.  Use tools like ELK (ElasticSearch+Logstash+Kibana) to examine logs.
  • Use CloudFoundry containers that don’t make you SSH to many machines, removing a big manual step that could be a security issue.
  • Move to using smaller, lighterweight Spring Boot Jars and Wars; not giant EAR files.
  • Use lightweight web containers like Jetty, Tomcat, tcServer, or Undertow.
  • Use lightweight frameworks, remember YAGNI and KISS.
  • Read and embrace:  Release It!
  • Break up the apps into smaller, logical components that model real business functionality.
  • Refactor existing code guided by tests, enhancing and adding more tests as the code evolves.  Remembering to use code coverage tools like Jacoco.
  • Design clean RESTful APIs guided by SRP (Single Responsibility Principle).
  • Use smart, modern opinionated frameworks for development like Spring Boot, DropWizard and RatPack.
  • Install GVM and use it to install SpringBoot, Gradle, Groovy, LazyBones and other great tools.  example:   gvm install springboot
  • Use Spring Cloud Connectors, Spring Cloud Netflix projects for Service Discovery and other useful services to enable microservices orchestration and development.
  • Use a very helpful application generator like JHipster.
  • Use Domain Driven Design.
  • Keep to the basics of OOD for existing code and Functional Reactive Programming for new code.
  • Document your API with Spring REST Docs or Swagger.  There is a great presentation on that here.
  • Start your green field projects with one button click or use the Spring CLI (spring init -d=web,data-jpa,actuator,data-rest –build=maven newappdir/).



Service Discovery

Scala and more… Links

Scala + Mongo

Scala + redis

Scala + Postgresql



Rx-Netty, Netflix Karyon 2

Slides from 2015 NE Scala

Hot Links

SecurityScorecard Partners with Sequoia in 12.5 Million Series A Funding Round

Tools for Crushing High Availability and Scalability, Part I

One tip that seems obvious, look at what the Internet innovators, large scale startups are doing.   They have open sourced many of their tools.   Square, Netflix, Google, Twitter, Facebook, LinkedIn,

I.  Data in Memory

Persistent Key-Value Store for Java from the excellent Java Advent calendar for 2014.

NoSQL Key-Value stores that run in-memory like Redis are incredibly helpful for scalability.   Memcached is also an option.   Gemfire is insanely powerful and scales across WANS, it runs some huge transaction sites and is awesome with Java.   India Railways is an incredibly interesting example of scaling on in-memory data grids.

II.  Microservices / 12 Factor Apps on a PaaS

It’s hard to scale a lumbering beast, easier when you have a swarm of agile services.    With each service having an API that can be shared internally or externally you easily open your architecture to extension and usage from outside sources which is key in growing them.   This has worked well for Uber, Google, Facebook, Twitter and a host of others.

The Platform to Enable Microservices:   An Open PaaS.   With the Linux Foundation running it, CloudFoundry is the open choice, it’s the Tomcat of the PaaS.  Extensible, Standard, Fast, Flexible, Elastic and Open — CloudFoundry Rocks.

III.  Reactive Programming

Using one of the reactive frameworks and following the Reactive Manifesto really help drive your scalable apps.  Take a look at:


Part II – Draft

IV.  Rapid Data Ingest

A key piece of the HA puzzle is having the data you need instantly available and

V.  Functional

Whether it’s Scala, Map Reduce, Apache Spark, Groovy or Java 8 with Lambdas – functional programming is bringing new ways of increasing performance and solving big data and real-time programming issues.

VI.   Message Driven

Disconnected services and running asynchronously greatly improves scalability and keeps one weak link from breaking your data chain.

VII.  Netflix Architecture / Spring Cloud Netflix

Internet pioneer Netflix has created a number of amazing tools that keep applications scaling, failing fast and recovering.   These tools have been augmented with Spring and are coming to the PaaS.   These have with critical portions like Circuit Breakers and discovery.   These are very important to manage all the microservices in your architecture.   There are also great tools from running on AWS.

VIII.  DevOps

VIV.  Containerization

X.  Continuous Delivery

Having a tool that builds your applications from Git and runs automated tests and static analysis is very critical.

Canary Deploys

Blue-Green Deploys

A/B Testing

Circuit Breaker for Inter-Service Communication and External Service Access

Reactive Streams / Asynchronous Streams

XI.  Responsive Front-Ends / Single Page Applications

WebSockets with SockJS, Stomp.  HTML5.  AngularJS / Backbone / Ember / …

XII.  Polyglot Programming

Play, MEAN, Spring Boot, DropWizard, Ratpack, Sinatra, Rails.   There are many different frameworks for building modern web applications.

XIII.  Front-End Tooling

Gulp, Grunt, Bower, …

XIV.  Clustering

Yarn, CloudFoundry, Kubernetes.

XV.  Native Applications

IoS and Android applications have more sensors, better user experience and superior look and feel.   There are awesome tools to enable doing this development faster and cleaner.

XVI.  Using Netty

XVIV.  API Documentation with Swagger


Critical Tools

There’s a ton of great tools I recommend for all developers to install.   Here is a sampling of ones that get frequent usage:   Java, Groovy, Typesafe Activator, CF CLI, Spring Boot CLI, Maven, Gradle, GVM, Watch, HTML2Text, Boot2Docker, VirtualBox, CURL, WGET, BREW, Vagrant, SOAPUI, IntelliJ, NetBeans, Android Studio, Redis Desktop Manager, SQuirrel SQL, Sublime, pgAdmin 3, git, and JMeter.

CF CLI – A Command Line Tool for working with CloudFoundry, Pivotal CF and Pivotal PWS.   See some more information at CloudFoundry Rocks.

HTML2Text – convert HTML dumps to Text for reading/piping into logs and easy command line viewing.  Works well with Curl and WGET.

Watch – cool UNIX/MacOSX command line tool for checking on an app, great with cf app <MYAPP>.

Spring Boot CLI – A Command Line Tool for working/running Spring Boot.    This is installed by the excellent GVM.

Maven – Default build tool.

Gradle – The other awesome build tool.

GVM Tool – great tool for installing and managing Groovy, Gradle, Vert.x, LazyBones, SpringBoot and more.

gvm current
crash: 1.3.0
gradle: 2.2.1
grails: 2.4.3
groovy: 2.3.8
lazybones: 0.7.1
springboot: 1.2.0.RELEASE

Using Spring-XD to Load Files into GemFire XD

Spring XD Scripts

My General Setup Script (I save it in setup.xd and load it via  script –file setup.xd)

had config fs –namenode hdfs://pivhdsne:8020
admin config server http://localhost:9393
hadoop fs ls /
stream list

The Script for Loading a File into GemFireXD via Spring-XD

stream create --name fileload --definition "file --dir=/tmp/xd/input/load --outputType=text/plain |  jdbc --tableName=APP.filetest --columns=id,name" --deploy


Spring XD Configuration for GemFire XD

Copy the GemFire XD JDBC Driver to Spring-XD (might need tools.jar as well)

cp /usr/lib/gphd/Pivotal_GemFireXD_10/lib/gemfirexd-client.jar /opt/pivotal/spring-xd/xd/lib/

Modify the Sink’s JDBC properties to point to your Gemfire XD, if you are using the Pivotal HD VM and install Spring-XD with Yum (sudo yum update spring-xd), this is the location:

url = jdbc:gemfirexd://localhost:1527
username = gfxd
password = gfxd
driverClassName = com.pivotal.gemfirexd.jdbc.ClientDriver

For Peer Client Driver you need more files from GemFireXD Lib (the .so binaries), linking is probably a good idea.


GemFire XD Setup

connect client 'localhost:1527';

create table filetest (id int, name varchar(100)) REPLICATE PERSISTENT;
select id, kind, netservers from sys.members; 
select * from filetest;

Spring XD Commands

stream list

show your jobs




Quick Look: Spring XD


Spring XD:  A really quick way to batch process data from an easy to run shell.   It’s very easy to setup to run a one node version of XD and use it on Windows, Mac or Linux.   Looks like a great tool.




Quick Loop: JHipster and Spring Boot Tools

Install Yeoman and other JavaScript Tools

npm install -g yo
npm install -g bower
npm install -g grunt-cli
npm install -g gulp
npm install -g karma
npm install jshint
npm install -g ngmin

Install JHipster

npm install -g generator-jhipster

Update your version of JHipster with:

npm update -g generator-jhipster

And then you can update your project when you run again.

yo jhipster
yo jhipster:entity foo
yo jhipster:service bar
grunt server
bower install
bower update
grunt build
java -jar jhipster-0.1-SNAPSHOT.war
mvn -Pprod package
-javaagent:spring_loaded/springloaded-jhipster.jar -noverify -Dspringloaded=plugins=io.github.jhipster.loaded.instrument.JHipsterLoadtimeInstrumentationPlugin
mvn -Pprod spring-boot:run
mvn spring-boot:run

The application will be available on http://localhost:8080.

Quick Look: Mocking Options

JMock is cool, because of this awesome book!


can test private and static
doesn’t need to worry about injections, works through class loader
mocks statics, 3rd party library, …




Quick Tip: Spring REST Utility for Current HTTP Request

Utility Method for Spring REST

 public static HttpServletRequest getCurrentRequest() {
     RequestAttributes requestAttributes = RequestContextHolder.getRequestAttributes();
     Assert.state(requestAttributes != null, "Could not find current request via RequestContextHolder");
     Assert.isInstanceOf(ServletRequestAttributes.class, requestAttributes);
     HttpServletRequest servletRequest = ((ServletRequestAttributes) requestAttributes).getRequest();
     Assert.state(servletRequest != null, "Could not find current HttpServletRequest");
     return servletRequest;

Sometimes it’s easier to get the underlying Servlet request to get some headers or variables.

 final String userIpAddress = getCurrentRequest().getRemoteAddr();
 final String userAgent = getCurrentRequest().getHeader("user-agent");

This is used in the simple REST service using HTTP Post verb @ the awesome CloudFoundry:


Tool for Creating Your Test JSON.

Spring Boot Documentation


Quick Tip: Maven Cleaning

Purge Cache

Or you can manually delete all or some of the directories under your .m2 directory.

We find if anything majorly changes in a few related modules, this is the safest course of action.   Delete all directories, delete .m2 directories, fresh clone from Git, mvn clean and enjoy.

Quick Links: Modern Tools for Modern Big Data Applications

Modern Front-End Tools (Front-End Build Wars)

A ton of great tools for running tons of tasks and building modern front-ends.   These are to the front what maven and gradle are to the back.
Build Wars: Gulp vs Grunt
Spring-cleaning Unused CSS With Grunt, Gulp, Broccoli or Brunch
Yeoman – Modern workflows for modern webapps
Environment-specific Builds With Grunt, Gulp or Broccoli
Gulp, Grunt, Whatever – Pony Foo
Brunch | ultra-fast HTML5 build tool
Modern Web Apps (Java 8 + AngularJS + Spring 4 + Gradle)
trackr: An AngularJS app with a Java 8 backend – Part I | techdev Solutions
trackr: An AngularJS app with a Java 8 backend – Part II | techdev Solutions
Pagination with Spring Data and HATEOAS in an AngularJS app | Patrick Grimard’s Java Blog
Modern Rapid Development and Production Deployment Tools (Spring Boot, Spring 4)
RestController in Spring 4.0
Spring 4 Tutorials
Spring Transaction Management
Spring Framework 4 on Java 8 // Speaker Deck
Pivotal presentations channel
Big Data (Spring with Hadoop)
SpringOne2GX 2013 Replay: Real Time Analytics with Spring
SpringOne2GX 2013 Replay: Hadoop – Just the Basics for Big Data Rookies – Seven Databases in Seven Weeks: Hbase, Day 2
Apache Solr real-time live index updates at scale with Apache Hadoop | Java Code Geeks
Modern Cloud Development and Hosting (Cloud Foundry)
Run Your Java Code on Cloud Foundry – Andy Piper (Pivotal)
Run the site on Cloud Foundry · spring-io/sagan Wiki
Pivotal Resources 
A real-time architecture using Hadoop and Storm @ JAX London
Spring Boot with Groovy and Friends
HyperLogLog – Wikipedia, the free encyclopedia
The lost outpost | a weblog by Andy Piper about technology, photography, and life
Installing a Hadoop Cluster with three Commands | codecentric Blogcodecentric Blog
Using Ambari Blueprints to automatically provision and install the Lambda Architecture | codecentric Blogcodecentric Blog
3 lessons in database design from the team behind Twitter’s Manhattan — Tech News and Analysis
Bloom filter – Wikipedia, the free encyclopedia
Hadoop Tips: Bloom Filters in HBase and Chrome
Enterprise-ready production-ready Java batch applications powered by Spring Boot | codecentric Blogcodecentric Blog
MapReduce Introduction – Tutorial
Scaling SQL with Redis – David Cramer’s Blog
HBase – Apache HBase™ Home
Work with Hadoop and NoSQL Databases with Toad for Cloud – ReadWrite
Google Research Publication: MapReduce
Overview | Postgres-XL
In 45 Min, Set Up Hadoop (Pivotal HD) on a Multi-VM Cluster & Run Test Data | Pivotal P.O.V.
BOSH Components | Cloud Foundry Docs
Pivotal Open Source Hub (San Francisco, CA) – Meetup
WebScaleSQL | “We’re Gonna Need A Bigger Database”
Eight Terminal Utilities Every OS X Command Line User Should Know ·
Report: NoSQL Databases – Providing Extreme Scale and Flexibility — Gigaom Research
HDFS Architecture
PostgreSQL: PostgreSQL 9.4 Beta 1 Released
Using Redis at Pinterest for Billions of Relationships | Pivotal P.O.V.
Replication, Clustering, and Connection Pooling – PostgreSQL wiki
Hadoop Content on InfoQ – JSON on Hadoop Example for Extending HAWQ Data Formats Using Pivotal eXtension Framework (PXF)
When should I use Greenplum Database versus HAWQ? | PivotalGuru – In 45 Min, Set Up Hadoop (Pivotal HD) on a Multi-VM Cluster & Run Test Data
PostgreSQL: Documentation: 9.3: High Availability, Load Balancing, and Replication
PostgreSQL – Wikipedia, the free encyclopedia
The Hadoop Ecosystem Table
Pivotal Hadoop Distribution and HAWQ Realtime Query Engine | Architects Zone – EMC World 2014: Pivotal and Isilon Take Hadoop Prime Time in the Enterprise – Pivotal ship Hadoop distro complete with ‘world’s fastest’ SQL query engine – Pivotal Hadoop Distribution and HAWQ Realtime Query Engine
HAWQ | Pivotal P.O.V.
Cloud Foundry Eclipse Plugin | Pivotal Docs – Transform Your Skills: Simple Steps to Set Up SQL on Hadoop – What Makes Hadoop So Important & How To Gain Business Value From It
Search Results For: hadoop pivotal
Cloud Foundry Environment Variables | Cloud Foundry Docs
PCC Installation Checklist | Pivotal Docs
Pivotal Web Services Documentation | Pivotal CF Docs
Hadoop Tutorials from Hortonworks
Tips for Java Developers | Cloud Foundry Docs
Spring for Apache Hadoop
Spring Framework Reference Documentation
16. Web MVC framework
Getting Started with Pivotal Web Services | Pivotal CF Docs
6 Easy Steps: Deploy Pivotal’s Hadoop on Docker | Pivotal P.O.V.
OpenTSDB – A Distributed, Scalable Monitoring System
Big Data Benchmark
Pivotal Web Services