ted.neward@newardassociates.com | Blog: http://blogs.newardassociates.com | Github: tedneward | LinkedIn: tedneward
"Your code broke: fix it!"
How do I identify the problem?
How do I find the problem?
How do I fix the problem?
"the art of identifying and fixing a problem after it has been spotted"
Crashes & Hangs
Poor Performance/Scalability
Incorrect Results
Security Exploits
Inconsistent User Interfaces
Or, succinctly, "Bugs are unmet expectations"
Short/impossible deadlines
"Code first, think later"
Misunderstood requirements
Developer ignorance
Lack of commitment to quality
Know your project
anyone should be able to draw the whiteboard view of the project on demand
Know your language(s)
Know your tools & technologies
Know your operating system & environment
Schedule debugging time into the project
agile or not, debugging always happens
Always assume responsibility for the bug
this means not "copping out" and blaming the compiler, runtime, container, libraries, tools, or anything else
Focus on the bug, not the people
the more time you spend blaming people, you're not only burning valuable time, you're burning your bridges with the very folks who can help
Think creatively!
Keep a clear head
if you haven't found a solution in twenty minutes, take a break and walk away from the machine
if you haven't found a solution in two hours, go bother a colleague and have a Bug Chat
Most languages have some concept of "exceptions"
objects/instances that, once "thrown", will propagate up the thread call stack until "caught" (usually by a filter set to match certain criteria of the exception, such as its class) or until the thread reaches the topmost stack frame, at which point the thread terminates
these exceptions can be either programmer-generated or a signal from the underlying platform/OS/runtime
programmers can "catch" these exceptions by establishing exception handlers and a code block to execute when an exception is caught
Using exceptions effectively is not simple or straightforward
most programmers, in fact, just ignore them
... until they can't anymore
... and then they just shunt them off to a log
... until they can't anymore
... and then they do the absolute minimum to handle the error
... until they can't anymore...
Exception discipline crosses many languages
as the name implies, its a programmer discipline
moments of weakness here will cost you a fortune later
1: Distinguish between different kinds of exceptions
Programmer failures
off-by-one errors, null-reference usage, etc
User failures
attempting to open a nonexistent file, improper security credentials, etc
Resource failures
network connection failures, out-of-memory errors, etc
Programmer failures should never be caught/handled
there is no real way to "recover" from this kind of error
we need to see that failure quickly
that is, during development/QA
at most, preserve work before termination
Resource failures should never be ignored
they are a fact of computing life
particularly network failures
but there is not much we can do except alert the user
preserve work, but do not terminate
alert the user, expect to retry in human timescales (seconds/minutes)
User failures are commonplace
MUST be handled appropriately
retry logic will be absolutely necessary
developer must ask, "What can the user do here?"
2: Layer exceptions at component boundaries
exceptions are designed to cross component boundaries
differentiate between those that a caller needs to know about, and those that offer no benefit
consider exceptions as part of the API contract between caller and target
avoid using the "generic" exception types as part of the contract
document exception possibilities
3: Arrange exceptions carefully in logical groupings
for most languages, this means inheritance
Exception (base class)
IOException inherits from Exception
SQLException inherits from Exception
avoid putting too many layers/permutations in unless useful
FileNotFoundException : IOException
InaccessibleFileException : IOException
ZeroByteSizeFileException : IOException
4: Keep exceptions relevant to the component
create custom exception types to each component
BusinessLogicException : Exception
DataAccessException : Exception
... and so on
do not expose exceptions from dependencies
violates encapsulation
what on earth would the caller do with them?
5: Never ignore an exception
there must always be a mitigation strategy
choosing not to act is not ignoring
but it had better be documented why
for those exceptions "that will never happen", terminate the program
if it'll truly never happen, then there's nothing to worry about, right?
6: Only throw exceptions when the caller needs to know
... and include all relevant information when you throw
imagine you have to debug from only what's in the exception
preserve encapsulation
some systems will allow "nested exceptions" to permit layering of information
"all relevant information" != "all information"
remember, stack traces will often get logged to disk; too much information could be yielding secrets
7: Remember that exceptions represent another flow of control
exceptions can happen any time, any place, for any reason
methods that work with finite resources should always guard against or around exceptions
clean up after yourself
JDK command-line tools
jdb: standard console debugger
extcheck: detects extension conflicts
jconsole: sample JMX view of JVM
visualvm: successor to jconsole
JDK command-line tools
jps: List instrumented JVMs on system
jstat: Collects & logs perf stats on JVM instance
jstatd: RMI server app that monitors creation & termination of JVM instances to allow attaching
jinfo: configuration info for running JVM, core file, or remote debug server
JDK command-line tools
jhat: browsable heap dump of HPROF file info
jmap: prints shared object memory maps or heap memory details of JVM, core file, or remote debug server
jsadebugd: attaches to a process or core file and acts as a debug server
jstack: prints a stack trace of threads for JVM, core file or remote debug server
JVM diagnostic flags
"verbose" flag (class, gc, jni)
provides first-level view of what's going on inside VM
java.security.debug property
Set to "access", "jar", "policy" or "scl" to debug/diagnose platform security behavior
JVM "XX" diagnostic flags
varies by JVM release--use with caution
Common flags
-XX:-PrintGC: Print messages at garbage collection.
-XX:-PrintGCDetails: Print more details at garbage collection.
-XX:-PrintGCTimeStamps: Print timestamps at garbage collection.
-XX:-TraceClassLoading: Trace loading of classes.
-XX:-TraceClassUnloading: Trace unloading of classes.
-XX:-PrintCompilation: Print message when a method is compiled.
OS-specific tools
Win32: windbg, userdump, dumpchk, Sysinternals, ...
Solaris: cpustat, cputrack, dtrace, iostat, ...
Linux: libnjamd, lsstack, ltrace, vmstat, ...
FindBugs
static code analysis tool scans compiled code for dangerous code patterns that usually lead to bugs
bugs fit into one of the following categories: bad practice, correctness, malicious code vulnerability, multithreaded correctness, performance and dodgy
run this as part of the build to avoid bugs
or, more accurately, helps spot the bugs in the code before you have to execute the code (whether through unit test or by hand)
Goal: zero FindBugs warnings
Diagnostic logging
log4j or java.util.logging are essentially equivalent
create a hierarchy of logger names to segregate log information in a manageable fashion
remember that loggers are configured hierarchically, so it's possible to turn on as much or as little info as desired by "turning up" the right node in the hierarchy
create a TelnetAppender for a real-time view
TelnetAppender (log4j) and SocketHandler (j.u.logging) provided out of the box; use an appropriate format (not XML or HTML)
ensure logging levels are dynamically adjustable
java.util.logging logs are automatically adjustable thanks to the JMX LoggingMBean in Java5; for log4j, you may need to create this yourself
Data access
debugging database issues can be tricky given the multiple layers involved in a standard Java app
all these layers of indirection leave the developer with an unclear view of what's really being sent to the database and back
turn on data-access layer logging to view details
use P6Spy JDBC driver to "Decorate" your driver
P6Spy logs all incoming JDBC calls to a log stream, giving you a clear indication of what's being called when against your JDBC driver, including fetch size, scrolling, in-place updates and transactions
Java Platform Debugging Architecture (JPDA)
target JVM talks over pipeline to debugger agent
uses Java Debug Interface (JDI) and Connectors to separate details of connection from debugging
JVMTI: agent inside the debuggee VM
gathers debug events for analysis & manipulation
JDWP: Java Debug Wire Protocol
transmits debug information across a channel
JDI: Java Debug Interface
used by debugger front-end
JPDA options
transport (required): name of transport
dt_socket,address=
dt_shmem,address=
server: whether to listen for debugger ("y") (def. "n")
suspend: suspend the entire VM ("y") (default "n")
onthrow: delay initialization of debugger until a package-qualified class exception is thrown
oncaught: delay initialization of debugger until an uncaught exception is thrown ("y")
launch: launch process when debugger initializes
used in conjunction with onthrow/oncaught
agentlib:jdwp=transport=dt_socket,server=y,address=8000
Listen for a socket connection on port 8000. Suspend this VM before main class loads (suspend=y by default)
agentlib:jdwp=transport=dt_shmem,server=y,suspend=n
Choose an available shared memory transport address and print it to stdout. Listen for a shared memory connection at that address. Allow the VM to begin executing before the debugger application attaches.
agentlib:jdwp=transport=dt_socket,server=y,address=8000,onthrow=java.io.IOException,launch=/usr/local/bin/app
Wait for an instance of java.io.IOException to be thrown in this VM. Suspend the VM (suspend=y by default). Listen for a socket connection on port 8000. Execute the following: "/usr/local/bin/debugstub dt_socket myhost:8000". This program can launch a debugger process in a separate window which will attach to this VM and begin debugging it.
agentlib:jdwp=transport=dt_shmem,server=y,onuncaught=y,launch=d:\bin\debugstub.exe
Wait for an uncaught exception to be thrown in this VM. Suspend the VM. Select a shared memory transport address and listen for a connection at that address. Execute the following: "d:\bin\debugstub.exe dt_shmem <address>", where <address> is the selected shared memory address. This program can launch a debugger process in a separate window which will attach to this VM and begin debugging it.
JVMTI: Java Virtual Machine Tools Interface
low-level C++ API exposing JVM events
implementing code is loaded into target JVM via the "agentlib" command-line parameter, and registers with the JVM via a named exported C function and block of function pointers passed in
useful for building general debuggers/profilers, or for building special-purpose tools
see the samples in the JMVTI directory in the JDK
Online resources
http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/TSG-VM.pdf
Troubleshooting Guide
http://java.sun.com/javase/6/webnotes/trouble/TSG-Desktop/TSG-Desktop.pdf
Troubleshooting Guide for Desktop Applications
http://java.sun.com/javase/6/webnotes/trouble/other/matrix6-Windows.html
Quick Troubleshooting Guide
http://java.sun.com/products/hotspot/whitepaper.html
The Java HotSpot Performance Engine Architecture
http://java.sun.com/docs/performance/
Java Performance Documentation
http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp
Java HotSpot VM Options
Debugging is equal parts attitude, process, tools, and knowledge
hone your skills before it becomes necessary
find out what your options are in Production
nobody ever stayed at work at night because "Honey, I'd love to come home, but we're just swamped under getting these UML diagrams correct now that we're about to ship!"
but lots of developers have called home to say, "Honey, I'd love to come home, but we have a monstrous bug here that nobody knows how to fix, and we're supposed to ship tomorrow morning..."
Architect, Engineering Manager/Leader, "force multiplier"
http://www.newardassociates.com
http://blogs.newardassociates.com
Sr Distinguished Engineer, Capital One
Educative (http://educative.io) Author
Performance Management for Engineering Managers
Books
Developer Relations Activity Patterns (w/Woodruff, et al; APress, forthcoming)
Professional F# 2.0 (w/Erickson, et al; Wrox, 2010)
Effective Enterprise Java (Addison-Wesley, 2004)
SSCLI Essentials (w/Stutz, et al; OReilly, 2003)
Server-Based Java Programming (Manning, 2000)