2016 Predictions

There have been a lot of interesting predictions for what may happen in technology for 2016.   Some friends at MongoDB told me their predictions.   The Kafka and Container Orchestration ones seem dead on.   For me I would add microservices will continue to be important and the Apache Spark will explode even more.   I feel Scala, Go, Python will gain steam and Java will remain steady.   A lot of cool new projects and technologies will come out that work in big data, cloud and devops.   We’ll have to see.

 

  1. Data gets its seat on the board: CDO becomes a must-have title for Fortune 500

CIOs are focused on the infrastructure to process data. CDOs are tasked with making the organization see data as an asset: making that data accessible, managed and governed, and finding the balance between extracting value from data and mitigating the risk of breaches.

 

  1. The ever-growing rise of Kafka: Data streams join databases to power modern business apps

Kafka will become an essential integration point in enterprise data infrastructure, facilitating the creation of intelligent, distributed systems. With the growth of IoT, global deployments, and microservices, the need to capture and control in-flight data before it’s stored in a database is becoming more important. Kafka and other streaming systems like Spark and Storm will complement databases as critical pieces of the enterprise stack for managing data across applications and data centers.

 

  1. The Container Wars are over: the battle is on for Container Orchestration

Docker has clearly won the Container War, and everyone is now standardizing on it. Now the battle has shifted to container orchestration, and control of the data center is what’s at stake. There will be winners and losers, and some surprises. Interestingly, the winners will be the incumbent vendors – Red Hat and VMWare – who have figured out how to take the open source gifts from Docker and Google (Kubernetes), build them into their commercial products, and sell them to their existing buyers.

 

  1. Application performance bottlenecks shift from storage to the CPU and network

For more than a decade, performance bottlenecks have overwhelmingly been located in the storage tier. During that time CPU and network speeds have remained flat, which hasn’t been much of a performance issue since they tended to be waiting on the storage tier. With key innovations in storage – Intel’s 3D XPoint, a new form of NAND flash memory that increases density, endurance and performance, and NVME SSD’s superior and faster interface that bypasses the file system – the enterprise architect will now need to focus their efforts on optimizing applications at the CPU and network tier. As a result, application code will need to be optimized, and deployment architectures – like in-memory computing, as well as keeping copies of data near users across the globe – will become critical.

Open Source Java Development Tools

For testing, SQL access, development, static code analysis and more.

— tools
http://www.ibm.com/developerworks/java/jdk/tools/memoryanalyzer/
http://www.ibm.com/developerworks/java/jdk/tools/gcmv/
http://www.ibm.com/developerworks/java/jdk/tools/dumpanalyzer/
http://java-source.net/open-source/web-testing-tools

http://htmlunit.sourceforge.net/
http://maxq.tigris.org/
http://jwebunit.sourceforge.net/
http://httpunit.sourceforge.net
TCP/Mon
http://sahi.sourceforge.net
http://watij.com/
http://imprimatur.sourceforge.net/
http://loadsim.sourceforge.net/
http://jmeter.apache.org/
http://webtest.canoo.com/webtest/manual/WebTestHome.html
http://java-source.net/open-source/testing-tools
http://www.clarkware.com/software/JUnitPerf.html
http://mockrunner.sourceforge.net/index.html
http://grinder.sourceforge.net/
http://xmlunit.sourceforge.net/
http://mockcreator.sourceforge.net/
http://quintanasoft.com/dumbster/
http://p-unit.sourceforge.net/
http://springunit.sourceforge.net/
http://mactor.sourceforge.net/
http://sourceforge.net/projects/dbhammer/
http://j-hawk.sourceforge.net/
http://databene.org/databene-benerator
http://sqlminus.sourceforge.net/
http://sqladmin.sourceforge.net/
http://sourceforge.net/projects/sqlshell
http://henplus.sourceforge.net/
http://memoranda.sourceforge.net/
http://jrat.sourceforge.net/
http://ejp.sourceforge.net/
http://jamonapi.sourceforge.net/
http://infrared.sourceforge.net/
http://profiler4j.sourceforge.net/
http://jstopwatch.sourceforge.net/
http://eurekaj.haagen.name/
https://code.google.com/p/allmon/downloads/list
http://homepages.mcs.vuw.ac.nz/~djp/djprof/
http://jfig.sourceforge.net/
http://trove4j.sourceforge.net/
http://quilt.sourceforge.net/
http://nounit.sourceforge.net/
http://insectj.sourceforge.net/
http://jester.sourceforge.net/
http://jvmdicover.sourceforge.net/
http://groboutils.sourceforge.net/codecoverage/index.html
http://coverlipse.sourceforge.net/
http://hansel.sourceforge.net/
http://pitest.org/
http://www.icesoft.org/java/home.jsf
http://www.csg.ci.i.u-tokyo.ac.jp/~chiba/javassist/
http://source.valtech.com/display/dpm/Dependometer
http://www.ucdetector.org/
http://qjpro.sourceforge.net/
http://condenser.sourceforge.net/
http://www.java2html.de/
http://tattletale.jboss.org/
http://artho.com/jlint/

http://docs.seleniumhq.org/
http://docs.seleniumhq.org/projects/webdriver/
http://code.tutsplus.com/tutorials/headless-functional-testing-with-selenium-and-phantomjs–net-30545
http://phantomjs.org/
http://casperjs.org/
http://www.slimerjs.org/
http://www.chrisle.me/2013/08/5-reasons-i-chose-selenium-over-phantomjs/
http://zombie.labnotes.org/

— Testing
http://zeroturnaround.com/rebellabs/developers-guide-static-code-analysis-findbugs-checkstyle-pmd-coverity-sonarqube/
http://java.dzone.com/articles/java-tools-source-code
http://xradar.sourceforge.net/
https://netbeans.org/kb/docs/java/code-inspect.html

xmlunit, jacoco, sqlunit, cdiunit, dbunit, arquillian, jmeter, soapui, etc…

google CodePro Analytix

https://developers.google.com/java-dev-tools/codepro/doc/features/codecoverage/code_coverage
https://developers.google.com/java-dev-tools/codepro/doc/features/metrics/metrics

https://github.com/springtestdbunit/spring-test-dbunit
http://springtestdbunit.github.io/spring-test-dbunit/
http://www.dbunit.org/
http://sqlunit.sourceforge.net/
https://code.google.com/p/jpa-unit/
http://fat-controller.sourceforge.net/

http://www.genericdtoassembler.org/

pl/sql
http://utplsql.sourceforge.net/
http://plunit.com/
https://code.google.com/p/pluto-test-framework/

http://plsqlutils.sourceforge.net/
http://log4plsql.sourceforge.net/

Performance

http://apmblog.compuware.com/2014/02/04/when-it-really-is-the-database-to-blame-for-bad-performance-a-story-about-slow-statements-and-resulting-connection-pool-issues/

http://apmblog.compuware.com/2010/09/14/week-38-transactions-in-a-jpa-world/

http://apmblog.compuware.com/2009/11/12/w-jax-jpa-under-the-hood/

http://apmblog.compuware.com/2014/01/07/hunting-a-oracle-jdbc-memory-leak-crashing-an-80jvm-websphere-cluster/

http://apmblog.compuware.com/2013/08/01/the-devops-way-to-solving-jvm-memory-issues/

http://apmblog.compuware.com/2013/05/07/fix-memory-leaks-in-java-production-applications/

http://apmblog.compuware.com/2011/12/15/the-top-java-memory-problems-part-2/

http://apmblog.compuware.com/2012/10/25/3-steps-to-diagnose-stuck-transactions-in-minutes/

http://apmblog.compuware.com/2012/10/17/how-to-triple-throughput-and-improve-application-performance-through-end-to-end-testing/

Oracle PL/SQL Tutorial

Getting IBM WS MQ Queue Count

 

hawt

http://hawt.io/
hawtio is a lightweight and modular HTML5 web console with lots of plugins for managing your Java stuff
plugins for camel, git, jboss, jmx, logs, maven, wiki
http://www.davsclaus.com/2013/04/apache-camel-web-dashboard-with-hawtio.html

http://hawt.io/plugins/index.html
http://macstrac.blogspot.co.uk/2013/06/introducing-apache-camel-based-open.html
http://bushorn.com/monitoring-camel-application/

https://github.com/Netflix/feign

https://github.com/Netflix/recipes-rss

http:// hawtdb.fusesource.org
http://www.eecs.harvard.edu/~mdw/proj/seda/
http://sourceforge.net/projects/seda/

Hikari Connection Pool

DataSource Configuration in Java:

HikariConfig config = new HikariConfig();
config.setDriverClassName(“org.postgresql.Driver”);
config.setJdbcUrl(postgresqlConfig.getUrl());
config.setUsername(postgresqlConfig.getUser());
config.setPassword(postgresqlConfig.getPassword());
config.setMinimumIdle(postgresqlConfig.getMaxIdle());
config.setMaximumPoolSize(postgresqlConfig.getMaxActive());
config.addDataSourceProperty(“cachePrepStmts”, “true”);
config.addDataSourceProperty(“prepStmtCacheSize”, “250”);
config.addDataSourceProperty(“prepStmtCacheSqlLimit”, “2048”);
return new HikariDataSource(config);

gradle dependency:

compile ‘com.zaxxer:HikariCP:2.4.1’

Press Release: Cubes

Introducing Cubes

The new iOS app creates visual collections of the content that matters to users

 

October 15, 2015 — New York Techweek– Cubes (cubesapp.com), creators of the mobile app bringing elegance and order to email and cloud-based services, announced the launch of its beta product today at New York Techweek. Available on iOS, Cubes automatically collects, organizes and visually displays files and links from a multitude of services. With everything in one place, users can quickly discover content by type, author, topic or service.

 

Free on the App Store worldwide, Cubes supports Gmail, Yahoo, iCloud, Outlook.com, Exchange, Office365 and Dropbox, with additional cloud storage and social media services coming soon.

 

Highlights of Cubes

 

  • Visual collections: All your important files, links, photos and more are beautifully displayed in one place for easy access.

  • Content, automatically organized: All your email attachments, email links and cloud-based content are collected and arranged into intelligent Cubes.

  • Create your own Cubes: Curate your content based on interest, project, topic or anything else you’d like. It’s an efficient way to pull together content from different services.

  • Contact Cubes: Create “contact cubes” to easily see content shared with your most important contacts.

 

With more businesses adopting SaaS models, and some offices eschewing paper records altogether, cleaning up the cloud has never been more important. Digital content is spread throughout the cloud, making it difficult to find the right content on the go. Cubes not only catalogs files across multiple services, it caters to our innate preference to process information visually.

 

Citing the growing popularity of Instagram and Pinterest, Cubes CEO and co-founder Kalpesh Rathod noted, “Both science and the marketplace show consumers prefer to consume content visually. We process visuals 60,000 times faster than text and 90 percent of the information transmitted to the brain is visual.”

 

“Our users don’t have to waste time switching between different services or scrolling through lists. Visual collections makes it a breeze to quickly see everything that matters,” Rathod said.

 

A graduate of engineering and MBA programs, Kalpesh is a serial entrepreneur with experience launching companies. Cubes has already attracted investor backing from former senior executives at Microsoft and Oracle.

 

Former president of Microsoft Canada and investor Simon Witts said, “Kalpesh and the team have found a sweet spot between elegance and efficiency. Cubes will undoubtedly help mobile professionals be more productive.”

 

About Cubes

 

Cubes is a mobile app that creates visual collections to make it easy to find the content that matters. From the Toronto-based team behind Inbox Cube, Cubes automatically collects, organizes and visually displays content from email and cloud-based services. Finally, mobile professionals can view attachments, shared links, photos and files in one place. Find, read and share content more easily with Cubes. Learn more at cubesapp.com.

Top InfoSec OSINT Tools and Tutorials

http://resources.infosecinstitute.com/analyzing-javascript/
https://code.google.com/p/jsunpack-n/
http://www.relentless-coding.com/projects/jsdetox/
https://github.com/svent/jsdetox
http://www.pandasecurity.com/mediacenter/malware/deobfuscating-malicious-code-layer-by-layer/

Security Tools for Pentesting, Malware Analysis, Infosec
https://fedorahosted.org/scap-security-guide/
https://github.com/NationalSecurityAgency/SIMP
https://github.com/enaqx/awesome-pentest
https://github.com/NationalSecurityAgency
http://www.infoworld.com/article/2606779/security/163151-11-open-source-security-tools-catching-fire-on-GitHub.html
https://github.com/rshipp/awesome-malware-analysis
https://github.com/onlurking/awesome-infosec
https://coderwall.com/p/sr1olq/information-security-open-courses-documentation
http://darkmatters.norsecorp.com/2015/03/13/mozilla-releases-open-source-forensics-tool-on-github/
http://mig.mozilla.org/
http://resources.infosecinstitute.com/14-popular-web-application-vulnerability-scanners/
https://github.com/neuroo/grabber
https://subgraph.com/vega/
http://code.google.com/p/zaproxy/
http://wapiti.sourceforge.net/
https://github.com/andresriancho/w3af/
http://w3af.org/
https://github.com/OWASP/OWASP-WebScarab
http://code.google.com/p/skipfish/
http://code.google.com/p/ratproxy/
http://code.google.com/p/wfuzz/
http://sourceforge.net/projects/grendel/
http://websecuritytool.codeplex.com/
http://xss.codeplex.com/
http://www.arachni-scanner.com/
http://cyber-research.github.io/
https://github.com/jeffbryner/MozDef
https://github.com/jipegit/OSXAuditor
https://github.com/ossec/ossec-hids
https://github.com/nabla-c0d3/sslyze
https://github.com/aol/moloch
https://code.google.com/p/reaver-wps/downloads/list
http://www.sleuthkit.org/autopsy/v2/
https://github.com/gamelinux/passivedns
https://coderwall.com/p/sr1olq/information-security-open-courses-documentation
http://mig.mozilla.org/
https://ant.isi.edu/software/index.html
https://predict.org/Default.aspx?tabid=104
http://www.caida.org/data/overview/
http://www.isi.edu/ant/traces/dataset_list.html
https://www.bro.org/
https://ant.isi.edu/blog/?p=745
http://www.cs.fsu.edu/~lawrence/OffNetSec/lectures.html
http://www.isi.edu/~johnh/PAPERS/Fukuda15a.pdf

http://www.slideshare.net/mimeframe/ruxcon-2012-15195589
http://www.slideshare.net/zanelackey/attackdriven-defense
https://github.com/isislab/Hack-Night/
https://www.owasp.org/index.php/Secure_Coding_Principles
http://www.caida.org/data/overview/

Quick Tip: Java 8 Quick Output

For Java 8, I have an ArrayList of Strings I want to display.   Couldn't be easier.   Using SLF4J for logging and running in Spring Boot. 

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.List;

        list.stream().forEach(value -> log.debug("Value: [{}]", value));

 

 

Top 25 Reactive Big Data (JVM) Resources and Articles

Ingestion system in Scala/Akka/Spray/… seems like a pumped up super Flume
Smack Stack (Mesos + )
Avro Resources
Kite works with avro http://kitesdk.org/docs/1.0.0/
Key Features:
  • Row based with schema
  • schema in the file
  • schema is json
  • block compression, splittable files
  • schema evolution
Spark : @databricksMesos : @mesosphere Akka : @typesafe Cassandra : @DataStax Kafka : @ConfluentInc
Akka/Avro Persistence?  https://github.com/wandoulabs/chana
Kafka-Pgsql
Kafka-Avro-REST
Akka / Event Sourcing
Apache Gora
Java 8 Streams
Do we store our Avro in HBase?
Apache Flume for Avro Consumption
Do we store and process Avro files on HDFS?
AKKA
hacked
Replace Chron and Quartz
RDS Postgres or EC2 Postgres
jHiccup (JVM check)

The Drive to Open APIs and Open Source

Cities are publishing data and using open source

http://cityofphiladelphia.github.io/slash-data/
http://cloudtweaks.com/2014/09/using-big-data-make-cities-smarter/

Governments are doing open source

https://github.com/alphagov/content-store

Hexagonal Architecture

http://www.infoq.com/news/2014/10/exploring-hexagonal-architecture
http://alistair.cockburn.us/Hexagonal+architecture

http://readwrite.com/2014/10/24/open-source-recruiting-facebook-netflix-twitter?imm_mid=0c6188&cmp=em-prog-na-na-newsltr_20141101

JVM DevOps with Immutable Infrastructure

PalletOps for JVM
Immutable Infrastructure

Reactive Programming, Reactive Streams, Lean, Unit Tests, DDD, Microservices, 12 Factor Apps

http://domainlanguage.com/ddd/
http://bravenewgeek.com/stream-processing-and-probabilistic-methods/
http://flexibleboss.com/2015/04/20/lean-and-flexible/
http://theory.stanford.edu/~aiken/publications/papers/asplos06.pdf
http://www.odata.org/

http://www.cnbc.com/id/102620215
http://softwaregarden.io/common-mistakes-junior-developers-writing-unit-tests/
https://github.com/light/JOps/blob/master/README.md
https://twitter.com/hashtag/craftconf%20slides?s=03

https://github.com/agilemobiledev/email-service-akka/blob/master/src/main/java/com/email/EmailService.java

Resume 2015

 

Timothy Spann

Background

I am a senior application developer and tech lead with extensive experience designing, coding, testing, running agile teams and supporting next-generation web based solutions in various Linux, Unix and Windows environments. I am a Senior Field Engineer at Pivotal Software. I have several blogs that I post about various technologies I am exploring including: Android, JDK 8, Scala, HTML 5 with WebSockets, Node.JS, Spring 4, Spring Boot, Redis, RabbitMQ, MongoDB, Groovy, SpringXD, Cloud, Apache Spark, Tachyon, Big Data, Hadoop, NoSQL, Microservices, 12 Factor Applications, Lambda architecture, reactive programming and PaaS. I am passionate about learning new technologies and how best to utilize them for real world applications.

 

 

 

Rebuilding your infrastructure

http://engineeringblog.yelp.com/2015/03/using-services-to-break-down-monoliths.html

Marathon and Mesos
HAProxy, SmartStack, Zookeeper
Alerting with ElasticSearch (https://elastalert.readthedocs.org/en/latest/)
Postgresql
Redis Driver – Lettuce – has some nice features
Java Redis Orm
Advanced Java Data Structures
Redis Meetup – NY 04-August-2015
SSL Tunnel with Redis
Java with Redis
Redis Proxy
Testing Points
Linux Performance Tools
Docker/Microservices issues
Consul and Puppet for Service Discovery & Mgmt
Drill for Mongo query
API
Moving to Scala

Scala/Akka/Play

Scala/Play

Apache Drill with AVRO and MongoDB

Drill with REST and JDBC -> AVRO, MONGO
Set up Zookeeper based cluster
!quit
firewall was blocking it < watch these ports
select * from dfs.`/Users/timothyspann/Downloads/apache-drill-1.1.0/sample-data/region.parquet`
SELECT * FROM cp.`employee.json` LIMIT 20
show databases;
use mongo.`security-mydatabase`;
show tables;
 
Mongo Issue:   I have similar observations. At the moment I can only query elements that are strings. Numbers and dates do not work.
This is Drill 1.1 with Mongo 3.0 issue
bin/sqlline –u jdbc:drill:schema=dfs;zk=local

Microservices Weekly Report

Free GEO IP
Microservices Security
Security
Microservices
Interesting, may be helpful
Stateless authentication server
Big Data
Security
AWS ElasticBeanStalk with Docker

Controlling Multiple Linux Boxes at Once

For running a few commands to multiple boxes

https://github.com/denis-sokolov/multi-ssh

http://sourceforge.net/projects/clusterssh/

https://github.com/duncs/clusterssh

https://code.google.com/p/parallel-ssh/

http://rundeck.org/

http://web.taranis.org/shmux/

find . -iname “*myfile*”

grep -iR “mytext” *

Of course, Puppet or Chef is a better option.    Or just use AWS ElasticBeanStalk or CloudFoundry.

Microservices, Devops, Scala, Akka, Spring Boot and more…

Spring Boot Microservices

RXJava/Reactive

 Microservices

New Open Source From IBM

NodeJS Metrics

Agentless System Crawler

DevOps Report

Scala / AKKA

Everything about Tor

Containers

Java 8

Redis Speed Burn

Redis commands per second

Improve performance with Twemproxy or dynomite

https://github.com/twitter/twemproxy

https://github.com/Netflix/dynomite

https://blog.recurly.com/2014/05/clustering-redis-maximize-uptime-scale

https://github.com/areina/smitty

https://github.com/wandoulabs/codis

http://redis.io/topics/clients

http://redis.io/commands/MONITOR

http://redis.io/topics/benchmarks

http://skipperkongen.dk/2013/08/27/how-many-requests-per-second-can-i-get-out-of-redis/

http://redis.io/topics/partitioning

https://www.datadoghq.com/wp-content/uploads/2013/09/Understanding-the-Top-5-Redis-Performance-Metrics.pdf

http://objectrocket.com/blog/how-to/10-quick-tips-about-redis

http://redis.io/topics/cluster-tutorial

Mac Ulimit

http://unix.stackexchange.com/questions/108174/how-to-persist-ulimit-settings-in-osx-mavericks

 

Test Data and Generated Data

http://www.fakenamegenerator.com/advanced.php?t=country&n%5B%5D=us&c%5B%5D=us&gen=50&age-min=19&age-max=85

http://www.wordgenerator.net/fake-word-generator.php

http://www.json-generator.com/    <–  nice one

http://www.generatedata.com/

http://www.agilemobiledeveloper.com/2014/01/06/data-generator-and-data-migration/

https://www.mockaroo.com/

http://www.fakemailgenerator.com/   <- really but they receive email

http://www.databasetestdata.com/generated-data

http://www.convertcsv.com/generate-test-data.htm

http://c2.com/cgi/wiki?TestDataGenerator

https://github.com/mvisonneau/docker-generatedata

http://finraos.github.io/DataGenerator/

https://randomapi.com/

https://randomuser.me/

http://www.freedomaingenerator.com/

QuickHit: Check Mail Checking from AWS EC2 with JavaMail

To avoid having to install maven or gradle or specialized tools, we just put in a simple class and shell script.    First download the mail jars you need from here.   A simple wget will work on the links.  Keep them in the same dir as the .java and you can use the shell script below.   Real quick and dirty, but works for testing if your local EC2 instance can send email and there’s no firewall or other issues.

import javax.mail.Address;
import javax.mail.Message;
import javax.mail.Session;
import javax.mail.Transport;
import javax.mail.internet.InternetAddress;
import javax.mail.internet.MimeMessage;
import java.util.Properties;

try { Properties props = System.getProperties();
                props.put("mail.smtp.host", "localhost");
                props.put("mail.debug", "true");
                Session session = Session.getDefaultInstance(props, null);
                session.setDebug(true);
                Transport transport = session.getTransport("smtp");
                MimeMessage message = new MimeMessage(session);
                Address fromAddress = new InternetAddress("root@localhost.localdomain");
                message.setFrom(fromAddress);
                InternetAddress to = new InternetAddress("tspann@COOLDOMAIN.com");
                message.addRecipient(Message.RecipientType.TO, to);
                message.setSubject("Test Mail");
                message.setText("Test Body");
                transport.connect(host, from);
                message.saveChanges();
                Transport.send(message);
                transport.close();
            } catch (Exception x) {
                x.printStackTrace();
            }
    }
}

mailcheck.sh

javac -cp imap-1.5.4.jar:javax.mail.jar:smtp-1.5.4.jar MailCheck.java
java -cp imap-1.5.4.jar:javax.mail.jar:smtp-1.5.4.jar:. MailCheck