Lifelong Learning

Reading Code - An Example. Spring 2.0.

With a couple of AWK scripts, called from an ANT script, you can - in about an hour - perform the first step of reading a large body of code. The "take inventory step". Here is a pretty quick analysis of the Spring 2.0 source code.

    * 1437 Classes and interfaces.
    * 178 Unique suffixes.
    * Most classes are around 30 lines long. Very few above a 100. Longest class is 269 lines.
    * Class names are long and descriptive.
    * Longest class name is 56 characters: DelegatePerTargetObjectDelegatingIntroductionInterceptor

Reading the Spring class names is very instructive. Extremely expressive names. A great model to follow as a naming standard.

The suffixes that occur more than 5 times with their frequency counts (studying the suffixes give you a good idea of the "concepts" in the code :

150 Exception
79 Bean
67 Utils
63 Resolver
51 Factory
49 Source
48 Interceptor
34 Support
28 View
28 Context
26 Tag
25 Controller
24 Editor
24 Adapter
22 Handler
21 Processor
20 Callback
19 Accessor
17 Manager
17 Listener
16 Parser
16 Extractor
16 Creator
15 Holder
15 Filter
15 Advisor
14 Pointcut
14 Definition
13 Mapping
13 Executor
12 Template
12 Resource
12 Configurer
12 Aware
11 Type
11 Proxy
11 Operations
11 Advice
10 Loader
10 Exporter
10 Assembler
9 Comparator
8 Result
8 Incrementer
8 Attribute
7 Translator
7 Strategy
7 Servlet
7 Reader
7 Operation
7 Matcher
7 Enum
6 Value
6 Request
6 Provider
6 Lookup
6 Event
5 Values
5 Synchronization
5 Status
5 Setter
5 Registry
5 Parameter
5 Locator
5 Helper
5 Attributes

The suffixes alphabetically, with their frequency counts are:

19 Accessor
24 Adapter
1 Adapter102
11 Advice
1 Advised
15 Advisor
1 Advisors
10 Assembler
1 Assert
8 Attribute
5 Attributes
12 Aware
79 Bean
4 Binder
1 Bootstrap
2 Builder
1 Bundle
1 Call
20 Callback
2 Chain
1 Coded
1 Codes
9 Comparator
4 Config
1 Configuration
12 Configurer
1 Constants
4 Container
3 Container102
28 Context
25 Controller
1 Conventions
3 Converter
1 Converter102
16 Creator
2 Data
1 Decoder
4 Decorator
14 Definition
1 Delegate
2 Dialect
1 Discover
3 Discoverer
1 Disposer
24 Editor
3 Entry
7 Enum
2 Error
2 Errors
6 Event
150 Exception
13 Executor
10 Exporter
16 Extractor
51 Factory
1 Factory102
2 File
15 Filter
1 Filters
1 Flow
1 Form
1 Function
4 Generator
2 Handle
22 Handler
5 Helper
15 Holder
4 Impl
1 In
8 Incrementer
2 Info
1 Information
1 Initializer
48 Interceptor
3 Invocation
2 Invoker
1 Lifecycle
1 List
17 Listener
10 Loader
1 Location
5 Locator
1 Log
1 Log904
6 Lookup
17 Manager
1 Manager102
4 Map
4 Mapper
13 Mapping
7 Matcher
1 Matchers
1 Matches
1 Mergeable
4 Message
1 Modified
2 Monitor
4 Multicaster
1 Naming
1 Notification
4 Object
7 Operation
11 Operations
1 Ordered
3 Override
1 Overrides
5 Parameter
1 Parameters
16 Parser
2 Persister
1 Point
14 Pointcut
1 Pointcuts
1 Pool
2 Portlet
1 Preparator
1 Problem
1 Procedure
21 Processor
1 Properties
6 Provider
11 Proxy
3 Publisher
3 Query
7 Reader
1 Record
3 Reference
1 References
1 Refreshable
2 Registrar
5 Registry
1 Replacer
2 Reporter
6 Request
2 Resolvable
63 Resolver
12 Resource
8 Result
1 Results
2 Runnable
4 Scope
2 Sender
7 Servlet
4 Set
5 Setter
49 Source
1 Sql
1 State
1 Stats
5 Status
1 Store
7 Strategy
1 Stream
4 Styler
34 Support
5 Synchronization
1 System
26 Tag
2 Task
12 Template
1 Template102
2 Theme
1 Translation
7 Translator
1 Trigger
11 Type
2 Update
67 Utils
2 Validator
6 Value
5 Values
2 Version
28 View
1 Visitor
1 Watch
1 Work
4 Wrapper
2 Writer

Some typical class names (alphabetical by suffix):

AbstractPropertyAccessor
HibernateAccessor
JdbcAccessor
AfterReturningAdviceAdapter
InstantiationAwareBeanPostProcessorAdapter
TransactionAwareSessionAdapter
AspectJExpressionPointcutAdvisor
AbstractConfigurableMBeanInfoAssembler
DelegatingTransactionAttribute
ApplicationEventPublisherAware
AbstractSingletonProxyFactoryBean
LocalConnectionFactoryBean
BeanDefinitionRegistryBuilder
PersistenceBrokerCallback
NullSafeComparator
ServletContextPropertyPlaceholderConfigurer
SimpleMessageListenerContainer
ConfigurableApplicationContext
AbstractWizardFormController
ServletWrappingController
PreparedStatementCreator
ScopedProxyBeanDefinitionDecorator
PrioritizedParameterNameDiscoverer
InternetAddressEditor
BadSqlGrammarException
BeanNotOfRequiredTypeException
CannotGetJdbcConnectionException
HttpSessionRequiredException
ModelAndViewDefiningException
TypeMismatchException
SimpleAsyncTaskExecutor
RowMapperResultSetExtractor
AbstractPoolingServerSessionFactory
HashMapCachingAdvisorChainFactory
WebContentGenerator
OracleLobHandler
AfterReturningAdviceInterceptor
AdvisedSupportListener
ServletContextResourceLoader
BeanFactoryLocator
HibernateTransactionManager
AbstractHandlerMapping
InterceptorAndDynamicMethodMatcher
MimeMailMessage
HibernateOperations
JdbcOperations
ResultSetSupportingSqlParameter
RemoteStatelessSessionBeanDefinitionParser
DefaultPropertiesPersister
ApplicationContextAwareProcessor
TransactionAwareDataSourceConnectionProvider
TransactionAwarePersistenceManagerFactoryProxy
ModelMBeanNotificationPublisher
SqlQuery
AbstractBeanDefinitionReader
ResourceEditorRegistrar
AbstractCachingViewResolver
AbstractResource
FileSystemResource
RemoteInvocationResult
SchedulingAwareRunnable
DispatcherServlet
ManagedSet
AbstractRefreshableTargetSource
CommonsPoolTargetSource
ObjectNamingStrategy
DefaultValueStyler
JdbcDaoSupport
JtaSessionSynchronization
HiddenInputTag
OptionTag
NamedParameterJdbcTemplate
TransactionTemplate
PersistenceExceptionTranslator
BlobStringType
ClobStringType
BeanFactoryUtils
DataAccessUtils
NumberUtils
AbstractSqlTypeValue
AbstractExcelView
AbstractPdfView
ActionRequestWrapper
OptionWriter

Reading Code - A General Approach

Over the years, when faced with a large body of code, I have approached it as I approach any technical reading task: an exercise in "concept extraction" - a De Bono simplicity technique. The essence of the technique is to try to reduce the sheer size of the problem to a manageable size - a variation of the "how do you eat an elephant" theme. In this post I will deal with reading very large amount of code. Reading a small amount of code (under 10,000 lines), line-by-line, still has different challenges, and is best done in an IDE, like Eclipse, using browsing, searching, and cross referencing facilities, together with some reverse engineering of classes and method executions. But before we get there we have to deal with the problem of size first.

First an anecdotal background to set the stage for the need for a code reading methodology. The largest code reading task I had was about 2 million lines of "legacy" Java code (about 5 years worth of development by about 200 developers) that I had to wrap my mind around when I joined a project at a large financial company. The documentation (although I was given three binders of it) was a couple of years behind the code, and I was told by the chief architect (who hired me) that if I read it, I will get the wrong idea about what the software does. I was part of the software architecture group (one chief architect, one system architect, and 15 subsystem architects). When I was hired, I was promised that I would have 2 hours a day with the system architect for a month, and any time I needed from any architect in the group). I was also promised time with each of 15 team leads and 15 project managers (each subsystem had a subsystem arch, a team lead, and a subsystem project manager as well as a build master). After a month on the job the total time I was able to get from all of these people combined was 3.5 hours. From the system architect, whose boss hired me, I got exactly half an hour - and that remained the average (half an hour per month) for my entire 19 month stint at that project. So the short version of this story is: you can't rely on "knowledge transfer" from others to get to know the system. My job, with an official title of "Subsystem Architect" was really "Performance Analyst". I was to be the software architects' representative with the performance testing and the engineering team (2 distinct teams one does the testing, the other performance modeling and capacity planning). The performance test leads, performance testers, and performance engineers, looked to me for "application knowledge". Since I was hired in from the outside, and have never worked on this project, I needed to ramp up very rapidly, and reading the source code was the only way to do it. Documentation was old. People were not available, and were not allowed to be available. I was not a priority to any of them. The urgency of the task was made clear when I was told that I was part of a 13 person "performance triage team" that had a "check-in" meeting every day at 10:30 am. Each and every day for three months! I was the "look to" guy for "application knowledge". I was greener than a broccoli and colder than an ice cube! You do the math on pressure per square inch.

Faced with 2 million lines of code, documentation you were told to avoid, because it was misleading, and UML that is 2-years behind (they had a full Rose model), how do you learn the system?

The obvious approach, reading the code line by line using a text editor or an IDE, will not be much help. It rapidly becomes overwhelming and lead to frustration and discouragement eventually leading to abandoning the effort. Reading line by line comes at later stage, after the scope of the task has been narrowed.

So here was my general approach, step by painful step:

Step 0. Preliminary: Use the application. Get familiar with what the application does from its user's point of view. Understand the operating concepts. If you can get time with key people, that would be great. In my case I got half an hour with the system architect, and the URL for the app. The architect did a one-page sketch on an 8 1/2 by 11 sheet stating that the system is basically very simple: we have to buy loans and contracts from banks - here is how it flows. The GUI had 110 use cases - but the basic ones are less than a dozen.

Step 1. Inventory the code. What do we have?

- Number of files
- Total lines of code
- Number of classes
- Average size of class (methods per class)
- Number of methods
- Average size of method (lines per method).

I used some tools that were freely available (Understand for Java) was one - for its trial period. I ended up just writing simple scripts to do the counting.

Step 2. Get an overview of the source code tree. What is there other than Java files? Config files, metadata files, XML, HTML,

Step 3. Understand the build process. What are the build-units (build chunks/subsystems/deployment packaging/...)

Step 4. Inventory the databases

- Number of databases
- Number of distinct schemas
- Tables per schema
- Average size of tables (columns per table)

I usually produce a "distilled schema" with this format: <tablename>(colum1, column2, ...) one line per table (only about six or seven columns - pk plus a few more). Essentially a table to me is <tablename>(primary key, info). You can put most databases on one or two pages. It is useful to show foreign keys as part of the info. A very large database (several hundred tables) would reduce to a a few pages.

Step 5. Inventory the main concepts. The database schema is the best source for that. If there is a domain model in UML that would be very nice. Distill the concepts down as much as possible. Most systems narrow down to about a dozen major concepts, while there may be hundreds of tables in the schema, and hundreds of classes in the domain model.

These basic preliminaries give you the big picture. The very big picture. I usually write scripts (mainly in AWK and Korn Shell - even on a PC with MKS toolkit - can't live without it). The scripts filter and summarize the code down to a digestible size.

A most useful script is one that extracts class names and sorts them by suffix. Most systems have class name patterns similar to the following:

XxxxService
XxxxBusinessController
XxxxFlowController
XxxxHandler
XxxxHelper
XxxxBusinessObject
XxxxLocator
XxxxAdapter
XxxxFactory
XxxxAction
XxxxForm
XxxFacade
XxxxBean
XxxxDAO
XxxxDTO
...
XxxxFoo
XxxxBar
XxxxThing
XxxxEntity
XxxxWhatever
....

I like to use AWK embedded in Korn Shell, but you can write a Java program, embedded in an Ant script, or use your favorite scripting language - Perl, Jython, whatever. If scripting gets too hard, I put the code in a database, and analyze it with SQL. On one project I had code for 1000 screens written in an HP 4GL proprietary language that I did not know. The language was pretty declarative and quite elegant. I wrote an AWK parser (simple one) and loaded the code into a relational database whose schema reflected the structure of the 4GL language. Then I wrote a GUI to browse through the code and extract requirements for the new system. The purpose of the project was to port the system fro the proprietary language to Forte - an OO 4GL.

Once you've sorted the class names by suffix, you can determine the number of distinct types of classes the system has. This list of suffixes is a great help in classifying the major concepts (and patterns) used to implement the system. Since there is a tremendous amount of repetition in the implementation (a major implementation pattern is usually repeated hundreds of time), this summarization alone will reveal most of the system secrets. In the financial system, there was a basic pattern of BizController, BizHelpers, BOs, DAOs, and DTOs, repeated 110 times, once for each use case. The architecture standards did not allow deviation from the basic pattern. I could have been saved tens of hours, if one of the architects had clued me in. But, as explained earlier, no one had time to meet with me. Actually, after I arrived at this pattern that most use cases used, and presented it to the architects, they were in denial! No way, our system is vastly more complex than that! Another side lesson you learn, when you start valuing simplicity, is that the "complexity priesthood" will not like what you say! They have a vested interest in complexity. They are the guardians of complexity. If things were suddenly to become simple, they could be out of a job!

The output of this script, a list of the distinct suffixes, and the supporting detail list of the all the classes, becomes "the index" to the system, and a guide to reading it.

Step 6. You can use reverse engineering tools to study the classes. I find the simplest UML the most helpful (just class names, no attributes or methods). For the main use cases, a sequence diagram of the major method can be helpful - if you have a good reverse engineering tool that can reduce the size (which can be overwhelmingly large - some tools will choke on it).

Step 7. Once you have put your arm around the big picture and you understand the "pattern of patterns", so to speak, you can start reading selectively. This is the time to fire up the IDE and start reading. I usually do that one use case at a time - guided by the system's operational concept, or its UI.

Step 8. My main reading goal still remains "concept extraction". What is this trying to do? Why is it doing it this way?

Step 9. One thing that can help is "exploratory testing". Read using the IDE and JUnit tests. Write test cases to explore what the code is doing. This may get complex if the code has external dependencies (accessing external services, writing to databases). You may have to use mock objects to fake any external dependency.

Step 10. Another approach that is useful is "mock refactoring". While reading the code, if it is hard to read, I go ahead and refactor it for readability. The most important refactoring is "extract method", "move method", and "introduce parameter object". Since the body of code is read-only for me, I won't re-check it back in the source code control, I slice it and dice it and refactor it (if it is poorly written).

I think a lot of these steps can be programmed as Eclipse plug-ins. If you know of any, please let me know.

Reading Technical Books

Technical books nowadays weigh in at 1000+ pages. As busy as you and I are, how would we approach reading such monsters? Here are my thoughts. I am still struggling with a lot of this, so it is not a "science", and I would love to hear other approaches.

Preliminary step: Put a dollar value on your technical reading time, and estimate the real cost of reading the book. Take your annual income, divide by a 1000 and multiply by a factor (either less than 1.0 or greater - personal choice). This gives you the cost of your reading hour. Say your annual income is $20,000 per year. Your hour is worth $20 to your employer. It may be worth more, or less, to you - while reading and learning. If you'd rather be golfing, or snow-boarding, then multiply that by "lost pleasure" factor, say 2.0, 3.0, or 5.0. The cost of a reading hour for you is then $40, $60, or $100. If you consider technical reading a pleasure activity, then it is not costly for you. Use a factor of 0.5, or 0.1, so the cost of a reading hour is $10.0, $2.0. Whatever you do, don't value your reading hour at $0.00. No one has valueless time. Anyway, you get the gist of this. So 10 hours spent on reading a technical book costs you, say, $400, if you settled on $40 as your hourly cost.The major take-away point here is that the cost of reading a book is not the price of the book. Since some technical books may require 40 or 80 hours, you are looking at a substantial investment in time - $1600 to $3200.

Step 1: Size the Book.

In a previous blog entry I outlined one book sizing approach - other than number of pages. Essentially concepts per page, or "concept density". If the concepts per page sizing is tiny, say less than 10 (when normalized to 1000, 0.01 absolute concepts per page), consider discarding the book.

Step 2: Classify the Book.

Tutorial, Reference, Survey, Hands-On, Not hands-on. Any other criterion that is meaningful to you.

Example 1: JDBC API Tutorial and Reference. Self-classifying in the title. About one fifth the book is tutorial, rest is pure, look-up, reference. I would classify this book as low-density tutorial. Ignoring the reference part, essentially obtainable online, so it does not represent a "reading load".

Example 2: Object-Oriented Software Construction. Tutorial. Not hands-on. High density, tutorial.

Step 3: Make a Concept List

Step 4: Use concept list to create a reading plan.

Step 5: Start Reading. Read selectively according to your reading plan.

But always remember: you don't have to read the book through to consider that you have read it.

A special difficulty is posed by hands-on books. If you just plunge-in and just "follow along" doing what the author says to do, you run the real danger of burning non-trivial amounts of time (weeks perhaps), and then getting frustrated and abandoning the reading that has now become a project on its own. To really make use of a hands-on book, I recommend reading it and not doing the hands-on part on a first read. After you've read it, then you can have a rough feel for how long the hands-on portion will take. I am sure this will be hard to do when you really want to dive in and start coding and testing in a hands-on development book. But try to resist, unless you have a fair idea of how long this dive will take - a day, a week, or four weeks. Or do just a little bit of the hands on, to get a feel for it. Then a quick read. Then a second read with following the hands-on.

General Comments on "Concepts".

- The concept of "concept" is a soft concept! A concept is any idea you, the reader of a big technical work, decide is a concept for whatever purpose that is useful to you. This is purely to give you a handle on conquering complexity resulting from sheer size, and an approach to "eating the elephant". Whatever works for you is valid. No one has to agree with your choice of what you consider, or discard, as concepts. No one has to see your list of concepts. It is by you, for you.

- What constitutes one concept to one person, may require a decomposition into many concepts for another. What is ignored as a concept (may be because it is old hat already) for one person, maybe a revelation in its novelty to another. We all require different size "bites", have different conceptualization schemes, and have different backgrounds, knowledge, and experience. Your choice depends on your background, your experience, your prior knowledge in the topics at hand, and your "current store" of concepts.

- Concepts come in chunks - or conceptual networks. You can do a concept map and draw simple connections between related concepts. Computer analysts, object modelers, and database designers are familiar with concept maps. Some elaborate forms of concept maps are called conceptual models, object models, data models, semantic networks, what have you. But there is no need to get too elaborate for our purposes here, simple circles and arrows would work very nicely. This concept is related to that, it refers to this, it is a kind of that, it requires that, it affects that, it depends on that, it leads to that, etc.

- You don't have to read a book to get its concept list and concept map. Start with the table of contents, then the index, then chapter summaries - if provided, or scan through diagrams. Some authors, Like Bertrand Meyer, actually tell you at the end of each chapter, "key concepts introduced in this chapter". This is what you're after exactly: key concepts introduced in this chapter. Any form of "power skimming" scheme can allow you to generate a concept list and a concept map, may be in a couple of hours. It really pays back, in "pre-reading" the book and "pre-digesting" it.

Sizing a Book

How do you size a book? The publishing industry wants us to think of a book size as the number of pages
in the book.

Number of pages has very little to do with the real size of the content of the book - the number of ideas the book contains - the number of distinct concepts.

As an example, a book on JDBC (Java Database Connectivity), should be about five pages long. Anything more is a lot of detail that can either be reasoned, or looked up in some online reference because JDBC has about five concepts (Driver, DataSource, Connection, Statement, and ResultSet).

I am exaggerating a bit here - JDBC may have seven concepts if you want to do things with metadata and transactions. But there is a 1280 page book on JDBC!

If, while skimming a book in a couple of hours, you can focus on finding the distinct concepts, then you can size it properly. A 1200 page JDBC book has far fewer concepts than The Elements of Style, which is about a 100 pages long. So, in that sense, The Elements of Style is a much "larger" book.

If we were to assign a size metric, say, concepts per page, normalized to 1000 (since most books will have a tiny number of concepts per page), we can compute the size of the JDBC book as 5.4 (7/1280 * 1000), while The Elements of Style would weigh in at 190 (19/100 * 1000).

This, of course, affects the reading time for each book. I need on average roughly half an hour per concept (it may take as much as an hour - different concepts are of different difficulty), so I would need roughly 5 hours, possibly 10, for the JDBC book. I would need 10 hours, possibly 20, for The Elements of Style. That's a pretty remarkable result, if you put the two books next to each other and physically weigh them. Bertrand Meyer's Object-Oriented Software Construction is 1296 pages. I counted 168 concepts (your count may vary). It would take me 84 to hours at least to read that one!

Most technical books, in this sizing approach, can come in as very thin, and their page count should not be intimidating. Even if you have a pile of ten such 1000+ pages books, you can look at them as about 50 or so pages of reading. You can apply the same logic to sizing a program, or a system. The number of concepts per 1000 lines of code. Many systems will come in very small, although they may contain thousands of lines of code.

Once you've located the distinct concepts in the book, and digested them, you've read it. There is never any necessity to read each and every page of a book. Many people don't read books, because we have been conditioned to consider that reading a book is reading each and every page. If that were true, considering how busy we all are, and how thick most technical books are, most of us would be limited to probably one book per year.

This of course does not apply to works of fiction, where you are reading for pleasure, not for information.