2016 Predictions

There have been a lot of interesting predictions for what may happen in technology for 2016.   Some friends at MongoDB told me their predictions.   The Kafka and Container Orchestration ones seem dead on.   For me I would add microservices will continue to be important and the Apache Spark will explode even more.   I feel Scala, Go, Python will gain steam and Java will remain steady.   A lot of cool new projects and technologies will come out that work in big data, cloud and devops.   We’ll have to see.

 

  1. Data gets its seat on the board: CDO becomes a must-have title for Fortune 500

CIOs are focused on the infrastructure to process data. CDOs are tasked with making the organization see data as an asset: making that data accessible, managed and governed, and finding the balance between extracting value from data and mitigating the risk of breaches.

 

  1. The ever-growing rise of Kafka: Data streams join databases to power modern business apps

Kafka will become an essential integration point in enterprise data infrastructure, facilitating the creation of intelligent, distributed systems. With the growth of IoT, global deployments, and microservices, the need to capture and control in-flight data before it’s stored in a database is becoming more important. Kafka and other streaming systems like Spark and Storm will complement databases as critical pieces of the enterprise stack for managing data across applications and data centers.

 

  1. The Container Wars are over: the battle is on for Container Orchestration

Docker has clearly won the Container War, and everyone is now standardizing on it. Now the battle has shifted to container orchestration, and control of the data center is what’s at stake. There will be winners and losers, and some surprises. Interestingly, the winners will be the incumbent vendors – Red Hat and VMWare – who have figured out how to take the open source gifts from Docker and Google (Kubernetes), build them into their commercial products, and sell them to their existing buyers.

 

  1. Application performance bottlenecks shift from storage to the CPU and network

For more than a decade, performance bottlenecks have overwhelmingly been located in the storage tier. During that time CPU and network speeds have remained flat, which hasn’t been much of a performance issue since they tended to be waiting on the storage tier. With key innovations in storage – Intel’s 3D XPoint, a new form of NAND flash memory that increases density, endurance and performance, and NVME SSD’s superior and faster interface that bypasses the file system – the enterprise architect will now need to focus their efforts on optimizing applications at the CPU and network tier. As a result, application code will need to be optimized, and deployment architectures – like in-memory computing, as well as keeping copies of data near users across the globe – will become critical.

Open Source Java Development Tools

For testing, SQL access, development, static code analysis and more.

— tools
http://www.ibm.com/developerworks/java/jdk/tools/memoryanalyzer/
http://www.ibm.com/developerworks/java/jdk/tools/gcmv/
http://www.ibm.com/developerworks/java/jdk/tools/dumpanalyzer/
http://java-source.net/open-source/web-testing-tools

http://htmlunit.sourceforge.net/
http://maxq.tigris.org/
http://jwebunit.sourceforge.net/
http://httpunit.sourceforge.net
TCP/Mon
http://sahi.sourceforge.net
http://watij.com/
http://imprimatur.sourceforge.net/
http://loadsim.sourceforge.net/
http://jmeter.apache.org/
http://webtest.canoo.com/webtest/manual/WebTestHome.html
http://java-source.net/open-source/testing-tools
http://www.clarkware.com/software/JUnitPerf.html
http://mockrunner.sourceforge.net/index.html
http://grinder.sourceforge.net/
http://xmlunit.sourceforge.net/
http://mockcreator.sourceforge.net/
http://quintanasoft.com/dumbster/
http://p-unit.sourceforge.net/
http://springunit.sourceforge.net/
http://mactor.sourceforge.net/
http://sourceforge.net/projects/dbhammer/
http://j-hawk.sourceforge.net/
http://databene.org/databene-benerator
http://sqlminus.sourceforge.net/
http://sqladmin.sourceforge.net/
http://sourceforge.net/projects/sqlshell
http://henplus.sourceforge.net/
http://memoranda.sourceforge.net/
http://jrat.sourceforge.net/
http://ejp.sourceforge.net/
http://jamonapi.sourceforge.net/
http://infrared.sourceforge.net/
http://profiler4j.sourceforge.net/
http://jstopwatch.sourceforge.net/
http://eurekaj.haagen.name/
https://code.google.com/p/allmon/downloads/list
http://homepages.mcs.vuw.ac.nz/~djp/djprof/
http://jfig.sourceforge.net/
http://trove4j.sourceforge.net/
http://quilt.sourceforge.net/
http://nounit.sourceforge.net/
http://insectj.sourceforge.net/
http://jester.sourceforge.net/
http://jvmdicover.sourceforge.net/
http://groboutils.sourceforge.net/codecoverage/index.html
http://coverlipse.sourceforge.net/
http://hansel.sourceforge.net/
http://pitest.org/
http://www.icesoft.org/java/home.jsf
http://www.csg.ci.i.u-tokyo.ac.jp/~chiba/javassist/
http://source.valtech.com/display/dpm/Dependometer
http://www.ucdetector.org/
http://qjpro.sourceforge.net/
http://condenser.sourceforge.net/
http://www.java2html.de/
http://tattletale.jboss.org/
http://artho.com/jlint/

http://docs.seleniumhq.org/
http://docs.seleniumhq.org/projects/webdriver/
http://code.tutsplus.com/tutorials/headless-functional-testing-with-selenium-and-phantomjs–net-30545
http://phantomjs.org/
http://casperjs.org/
http://www.slimerjs.org/
http://www.chrisle.me/2013/08/5-reasons-i-chose-selenium-over-phantomjs/
http://zombie.labnotes.org/

— Testing
http://zeroturnaround.com/rebellabs/developers-guide-static-code-analysis-findbugs-checkstyle-pmd-coverity-sonarqube/
http://java.dzone.com/articles/java-tools-source-code
http://xradar.sourceforge.net/
https://netbeans.org/kb/docs/java/code-inspect.html

xmlunit, jacoco, sqlunit, cdiunit, dbunit, arquillian, jmeter, soapui, etc…

google CodePro Analytix

https://developers.google.com/java-dev-tools/codepro/doc/features/codecoverage/code_coverage
https://developers.google.com/java-dev-tools/codepro/doc/features/metrics/metrics

https://github.com/springtestdbunit/spring-test-dbunit
http://springtestdbunit.github.io/spring-test-dbunit/
http://www.dbunit.org/
http://sqlunit.sourceforge.net/
https://code.google.com/p/jpa-unit/
http://fat-controller.sourceforge.net/

http://www.genericdtoassembler.org/

pl/sql
http://utplsql.sourceforge.net/
http://plunit.com/
https://code.google.com/p/pluto-test-framework/

http://plsqlutils.sourceforge.net/
http://log4plsql.sourceforge.net/

Performance

http://apmblog.compuware.com/2014/02/04/when-it-really-is-the-database-to-blame-for-bad-performance-a-story-about-slow-statements-and-resulting-connection-pool-issues/

http://apmblog.compuware.com/2010/09/14/week-38-transactions-in-a-jpa-world/

http://apmblog.compuware.com/2009/11/12/w-jax-jpa-under-the-hood/

http://apmblog.compuware.com/2014/01/07/hunting-a-oracle-jdbc-memory-leak-crashing-an-80jvm-websphere-cluster/

http://apmblog.compuware.com/2013/08/01/the-devops-way-to-solving-jvm-memory-issues/

http://apmblog.compuware.com/2013/05/07/fix-memory-leaks-in-java-production-applications/

http://apmblog.compuware.com/2011/12/15/the-top-java-memory-problems-part-2/

http://apmblog.compuware.com/2012/10/25/3-steps-to-diagnose-stuck-transactions-in-minutes/

http://apmblog.compuware.com/2012/10/17/how-to-triple-throughput-and-improve-application-performance-through-end-to-end-testing/

Oracle PL/SQL Tutorial

Getting IBM WS MQ Queue Count

 

hawt

http://hawt.io/
hawtio is a lightweight and modular HTML5 web console with lots of plugins for managing your Java stuff
plugins for camel, git, jboss, jmx, logs, maven, wiki
http://www.davsclaus.com/2013/04/apache-camel-web-dashboard-with-hawtio.html

http://hawt.io/plugins/index.html
http://macstrac.blogspot.co.uk/2013/06/introducing-apache-camel-based-open.html
http://bushorn.com/monitoring-camel-application/

https://github.com/Netflix/feign

https://github.com/Netflix/recipes-rss

http:// hawtdb.fusesource.org
http://www.eecs.harvard.edu/~mdw/proj/seda/
http://sourceforge.net/projects/seda/

Press Release: Cubes

Introducing Cubes

The new iOS app creates visual collections of the content that matters to users

 

October 15, 2015 — New York Techweek– Cubes (cubesapp.com), creators of the mobile app bringing elegance and order to email and cloud-based services, announced the launch of its beta product today at New York Techweek. Available on iOS, Cubes automatically collects, organizes and visually displays files and links from a multitude of services. With everything in one place, users can quickly discover content by type, author, topic or service.

 

Free on the App Store worldwide, Cubes supports Gmail, Yahoo, iCloud, Outlook.com, Exchange, Office365 and Dropbox, with additional cloud storage and social media services coming soon.

 

Highlights of Cubes

 

  • Visual collections: All your important files, links, photos and more are beautifully displayed in one place for easy access.

  • Content, automatically organized: All your email attachments, email links and cloud-based content are collected and arranged into intelligent Cubes.

  • Create your own Cubes: Curate your content based on interest, project, topic or anything else you’d like. It’s an efficient way to pull together content from different services.

  • Contact Cubes: Create “contact cubes” to easily see content shared with your most important contacts.

 

With more businesses adopting SaaS models, and some offices eschewing paper records altogether, cleaning up the cloud has never been more important. Digital content is spread throughout the cloud, making it difficult to find the right content on the go. Cubes not only catalogs files across multiple services, it caters to our innate preference to process information visually.

 

Citing the growing popularity of Instagram and Pinterest, Cubes CEO and co-founder Kalpesh Rathod noted, “Both science and the marketplace show consumers prefer to consume content visually. We process visuals 60,000 times faster than text and 90 percent of the information transmitted to the brain is visual.”

 

“Our users don’t have to waste time switching between different services or scrolling through lists. Visual collections makes it a breeze to quickly see everything that matters,” Rathod said.

 

A graduate of engineering and MBA programs, Kalpesh is a serial entrepreneur with experience launching companies. Cubes has already attracted investor backing from former senior executives at Microsoft and Oracle.

 

Former president of Microsoft Canada and investor Simon Witts said, “Kalpesh and the team have found a sweet spot between elegance and efficiency. Cubes will undoubtedly help mobile professionals be more productive.”

 

About Cubes

 

Cubes is a mobile app that creates visual collections to make it easy to find the content that matters. From the Toronto-based team behind Inbox Cube, Cubes automatically collects, organizes and visually displays content from email and cloud-based services. Finally, mobile professionals can view attachments, shared links, photos and files in one place. Find, read and share content more easily with Cubes. Learn more at cubesapp.com.

Quick Tip: Java 8 Quick Output

For Java 8, I have an ArrayList of Strings I want to display.   Couldn't be easier.   Using SLF4J for logging and running in Spring Boot. 

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.List;

        list.stream().forEach(value -> log.debug("Value: [{}]", value));

 

 

The Drive to Open APIs and Open Source

Cities are publishing data and using open source

http://cityofphiladelphia.github.io/slash-data/
http://cloudtweaks.com/2014/09/using-big-data-make-cities-smarter/

Governments are doing open source

https://github.com/alphagov/content-store

Hexagonal Architecture

http://www.infoq.com/news/2014/10/exploring-hexagonal-architecture
http://alistair.cockburn.us/Hexagonal+architecture

http://readwrite.com/2014/10/24/open-source-recruiting-facebook-netflix-twitter?imm_mid=0c6188&cmp=em-prog-na-na-newsltr_20141101

Reactive Programming, Reactive Streams, Lean, Unit Tests, DDD, Microservices, 12 Factor Apps

http://domainlanguage.com/ddd/
http://bravenewgeek.com/stream-processing-and-probabilistic-methods/
http://flexibleboss.com/2015/04/20/lean-and-flexible/
http://theory.stanford.edu/~aiken/publications/papers/asplos06.pdf
http://www.odata.org/

http://www.cnbc.com/id/102620215
http://softwaregarden.io/common-mistakes-junior-developers-writing-unit-tests/
https://github.com/light/JOps/blob/master/README.md
https://twitter.com/hashtag/craftconf%20slides?s=03

https://github.com/agilemobiledev/email-service-akka/blob/master/src/main/java/com/email/EmailService.java

Resume 2015

 

Timothy Spann

Background

I am a senior application developer and tech lead with extensive experience designing, coding, testing, running agile teams and supporting next-generation web based solutions in various Linux, Unix and Windows environments. I am a Senior Field Engineer at Pivotal Software. I have several blogs that I post about various technologies I am exploring including: Android, JDK 8, Scala, HTML 5 with WebSockets, Node.JS, Spring 4, Spring Boot, Redis, RabbitMQ, MongoDB, Groovy, SpringXD, Cloud, Apache Spark, Tachyon, Big Data, Hadoop, NoSQL, Microservices, 12 Factor Applications, Lambda architecture, reactive programming and PaaS. I am passionate about learning new technologies and how best to utilize them for real world applications.

 

 

 

Scala/Akka/Play

Scala/Play

Controlling Multiple Linux Boxes at Once

For running a few commands to multiple boxes

https://github.com/denis-sokolov/multi-ssh

http://sourceforge.net/projects/clusterssh/

https://github.com/duncs/clusterssh

https://code.google.com/p/parallel-ssh/

http://rundeck.org/

http://web.taranis.org/shmux/

find . -iname “*myfile*”

grep -iR “mytext” *

Of course, Puppet or Chef is a better option.    Or just use AWS ElasticBeanStalk or CloudFoundry.

Microservices, Devops, Scala, Akka, Spring Boot and more…

Spring Boot Microservices

RXJava/Reactive

 Microservices

New Open Source From IBM

NodeJS Metrics

Agentless System Crawler

DevOps Report

Scala / AKKA

Everything about Tor

Containers

Java 8

NOSQL (MongoDB) Injection

https://www.owasp.org/index.php/Testing_for_NoSQL_injection

http://blog.websecurify.com/2014/08/hacking-nodejs-and-mongodb.html

http://www.nosqlmap.net/

https://media.blackhat.com/bh-us-11/Sullivan/BH_US_11_Sullivan_Server_Side_WP.pdf

http://2012.zeronights.org/includes/docs/Firstov%20-%20Attacking%20MongoDB.pdf

http://ieee-security.org/TC/SPW2015/W2SP/papers/W2SP_2015_submission_11.pdf

http://tinple.io/tech/mongo-security-part-two

http://blog.websecurify.com/2014/08/attacks-nodejs-and-mongodb-part-to.html

Using https://en.wikipedia.org/wiki/Rainbow_table

http://www.reddit.com/r/netsec/

https://thecodebarbarian.wordpress.com/2014/09/04/defending-against-query-selector-injection-attacks/

http://docs.mongodb.org/manual/administration/security-checklist/

http://sakurity.com/blog/2015/06/04/mongo_ruby_regexp.html

http://software-talk.org/blog/2015/02/mongodb-nosql-injection-security/

https://github.com/shirishp/NoSQLInjectionDemo

https://blogs.adobe.com/security/files/2011/04/NoSQL-But-Even-Less-Security.pdf

http://www.slideshare.net/44Con/pentesting-nosql-dbs-with-nosql-exploitation-framework

Streaming Large JSON Files with Java 8

As expected, the main options are the two big guys in JSON parsing,  GSON from Google and FasterXML Jackson 2.   Both are very solid and fast, but there’s also a couple of other libraries worth trying.

If you use Jackson, use the latest, FasterXML Jackson 2.  Not Codehaus.

Spring Cloud Configuration

Hot Links

Critical Tools

There’s a ton of great tools I recommend for all developers to install.   Here is a sampling of ones that get frequent usage:   Java, Groovy, Typesafe Activator, CF CLI, Spring Boot CLI, Maven, Gradle, GVM, Watch, HTML2Text, Boot2Docker, VirtualBox, CURL, WGET, BREW, Vagrant, SOAPUI, IntelliJ, NetBeans, Android Studio, Redis Desktop Manager, SQuirrel SQL, Sublime, pgAdmin 3, git, and JMeter.

CF CLI – A Command Line Tool for working with CloudFoundry, Pivotal CF and Pivotal PWS.   See some more information at CloudFoundry Rocks.

HTML2Text – convert HTML dumps to Text for reading/piping into logs and easy command line viewing.  Works well with Curl and WGET.

Watch – cool UNIX/MacOSX command line tool for checking on an app, great with cf app <MYAPP>.

Spring Boot CLI – A Command Line Tool for working/running Spring Boot.    This is installed by the excellent GVM.

Maven – Default build tool.

Gradle – The other awesome build tool.

GVM Tool – great tool for installing and managing Groovy, Gradle, Vert.x, LazyBones, SpringBoot and more.

gvm current
Using:
crash: 1.3.0
gradle: 2.2.1
grails: 2.4.3
groovy: 2.3.8
lazybones: 0.7.1
springboot: 1.2.0.RELEASE






Using Spring-XD to Load Files into GemFire XD

Spring XD Scripts

My General Setup Script (I save it in setup.xd and load it via  script –file setup.xd)

had config fs –namenode hdfs://pivhdsne:8020
admin config server http://localhost:9393
hadoop fs ls /
stream list

The Script for Loading a File into GemFireXD via Spring-XD

stream create --name fileload --definition "file --dir=/tmp/xd/input/load --outputType=text/plain |  jdbc --tableName=APP.filetest --columns=id,name" --deploy

 

Spring XD Configuration for GemFire XD

Copy the GemFire XD JDBC Driver to Spring-XD (might need tools.jar as well)

cp /usr/lib/gphd/Pivotal_GemFireXD_10/lib/gemfirexd-client.jar /opt/pivotal/spring-xd/xd/lib/

Modify the Sink’s JDBC properties to point to your Gemfire XD, if you are using the Pivotal HD VM and install Spring-XD with Yum (sudo yum update spring-xd), this is the location:

/opt/pivotal/spring-xd/xd/config/modules/sink/jdbc/jdbc.properties
url = jdbc:gemfirexd://localhost:1527
username = gfxd
password = gfxd
driverClassName = com.pivotal.gemfirexd.jdbc.ClientDriver

For Peer Client Driver you need more files from GemFireXD Lib (the .so binaries), linking is probably a good idea.

 

GemFire XD Setup

gfxd
connect client 'localhost:1527';

create table filetest (id int, name varchar(100)) REPLICATE PERSISTENT;
select id, kind, netservers from sys.members; 
select * from filetest;

Spring XD Commands

stream list

show your jobs

 

Reference:

 

Quick Look: Spring XD

 

Spring XD:  A really quick way to batch process data from an easy to run shell.   It’s very easy to setup to run a one node version of XD and use it on Windows, Mac or Linux.   Looks like a great tool.

 

 

 

Quick Loop: JHipster and Spring Boot Tools

Install Yeoman and other JavaScript Tools

npm install -g yo
npm install -g bower
npm install -g grunt-cli
npm install -g gulp
npm install -g karma
npm install jshint
npm install -g ngmin

Install JHipster

npm install -g generator-jhipster

Update your version of JHipster with:

npm update -g generator-jhipster

And then you can update your project when you run again.

yo jhipster
yo jhipster:entity foo
yo jhipster:service bar
grunt server
bower install
bower update
grunt build
java -jar jhipster-0.1-SNAPSHOT.war --spring.profiles.active=prod
mvn -Pprod package
-javaagent:spring_loaded/springloaded-jhipster.jar -noverify -Dspringloaded=plugins=io.github.jhipster.loaded.instrument.JHipsterLoadtimeInstrumentationPlugin
mvn -Pprod spring-boot:run
mvn spring-boot:run

The application will be available on http://localhost:8080.

Quick Look: Mocking Options

JMock is cool, because of this awesome book!

JMockIt

can test private and static
doesn’t need to worry about injections, works through class loader
mocks statics, 3rd party library, …