ࡱ > M [ bjbj== "\ W W 2< l p
B B B 8 C D g F bU bU ~ V V i[ {{ , $ 3 b ? G[ " i[ ? ? g
V V Q ( g g g ?
V V , g ? g g * J , V F 0S 6 B A R , H g B , g
Gregory Golberg
HYPERLINK "mailto:grisha@alum.mit.edu" grisha@alum.mit.edu
August, 2005 -- August, 2006
Dbdb a JDI-based Platform for Cross-language Debugging
Table of contents
TOC \o "1-5" \h \z HYPERLINK \l "_Toc143714897" Dbdb a JDI-based Platform for Cross-language Debugging PAGEREF _Toc143714897 \h 1
HYPERLINK \l "_Toc143714898" Introduction PAGEREF _Toc143714898 \h 4
HYPERLINK \l "_Toc143714899" About this document PAGEREF _Toc143714899 \h 4
HYPERLINK \l "_Toc143714900" More up-to-date information PAGEREF _Toc143714900 \h 5
HYPERLINK \l "_Toc143714901" Tools PAGEREF _Toc143714901 \h 5
HYPERLINK \l "_Toc143714902" Related work and the value of Dbdb PAGEREF _Toc143714902 \h 5
HYPERLINK \l "_Toc143714903" Why JDI? PAGEREF _Toc143714903 \h 7
HYPERLINK \l "_Toc143714904" What is JDI? PAGEREF _Toc143714904 \h 7
HYPERLINK \l "_Toc143714905" Proof of concept PAGEREF _Toc143714905 \h 8
HYPERLINK \l "_Toc143714906" Architectural overview PAGEREF _Toc143714906 \h 9
HYPERLINK \l "_Toc143714907" Infrastructure PAGEREF _Toc143714907 \h 10
HYPERLINK \l "_Toc143714908" Algorithm PAGEREF _Toc143714908 \h 12
HYPERLINK \l "_Toc143714909" Mediators PAGEREF _Toc143714909 \h 14
HYPERLINK \l "_Toc143714910" Using Dbdb as a framework PAGEREF _Toc143714910 \h 17
HYPERLINK \l "_Toc143714911" Registry reform PAGEREF _Toc143714911 \h 17
HYPERLINK \l "_Toc143714912" Mediator policy PAGEREF _Toc143714912 \h 18
HYPERLINK \l "_Toc143714913" Use cases PAGEREF _Toc143714913 \h 18
HYPERLINK \l "_Toc143714914" Framework or tool? PAGEREF _Toc143714914 \h 20
HYPERLINK \l "_Toc143714915" Post-mortem PAGEREF _Toc143714915 \h 20
HYPERLINK \l "_Toc143714916" Leaky abstractons PAGEREF _Toc143714916 \h 20
HYPERLINK \l "_Toc143714917" Speaking too soon: false starts and misdirections PAGEREF _Toc143714917 \h 20
HYPERLINK \l "_Toc143714918" Further work PAGEREF _Toc143714918 \h 21
HYPERLINK \l "_Toc143714919" Other language types PAGEREF _Toc143714919 \h 22
HYPERLINK \l "_Toc143714920" Parallel drill-down PAGEREF _Toc143714920 \h 22
HYPERLINK \l "_Toc143714921" Other features PAGEREF _Toc143714921 \h 23
HYPERLINK \l "_Toc143714922" Conclusion PAGEREF _Toc143714922 \h 24
HYPERLINK \l "_Toc143714923" Appendix A. Code listings PAGEREF _Toc143714923 \h 25
HYPERLINK \l "_Toc143714924" Listing 1. org.hrum.dbdb.OracleExample1 PAGEREF _Toc143714924 \h 25
HYPERLINK \l "_Toc143714925" Listing 2. PL/SQL stored procedures PAGEREF _Toc143714925 \h 26
HYPERLINK \l "_Toc143714926" Appendix B. Screenshots PAGEREF _Toc143714926 \h 27
HYPERLINK \l "_Toc143714927" Appendix C. Criticism of BEAs Patent Application PAGEREF _Toc143714927 \h 28
HYPERLINK \l "_Toc143714928" Appendix D. Other use cases for single-stack debugging PAGEREF _Toc143714928 \h 29
HYPERLINK \l "_Toc143714929" Appendix E. Why not JDI? PAGEREF _Toc143714929 \h 30
HYPERLINK \l "_Toc143714930" JDWP (Java Debug Wire Protocol) PAGEREF _Toc143714930 \h 30
HYPERLINK \l "_Toc143714931" Eclipse Debug Framework PAGEREF _Toc143714931 \h 31
HYPERLINK \l "_Toc143714932" Pro EDF PAGEREF _Toc143714932 \h 31
HYPERLINK \l "_Toc143714933" Pro JPDA PAGEREF _Toc143714933 \h 32
HYPERLINK \l "_Toc143714934" Why not, indeed? PAGEREF _Toc143714934 \h 32
HYPERLINK \l "_Toc143714935" Appendix F. References PAGEREF _Toc143714935 \h 33
Introduction
Source-level debuggers, and GUI for them, have been around for a long time and are indispensable tools for many programmers. As most real-world programs involve more than one language, modern IDEs present a consistent graphical front-end to different debuggers, for instance, the Eclipse IDE (for Java, C, C++, Python, Ruby, etc.), Oracles JDeveloper (for Java and PL/SQL debuggers), the ubiquitous Emacs (for everything but the kitchen sink), etc. However, while ability to debug multiple languages in a single IDE is no doubt useful, integration between the different debuggers (of importance to projects consisting of modules spanning several languages) still presents a problem. A notable enhancement is presented by tools such as SCORE, a debugger supporting Ada and C and allowing to seamlessly transition between languages ADDIN EN.CITE 101012SCORE: Multi-Language Debuggerhttp://www.info.uni-karlsruhe.de/~andf/documents/scoremld.pdf[1].
A still further improvement would be to allow for single-stack debugging of mixed-language programs. In other words, as program in language A calls a program in language B, the programmer sees a single call stack from A to B, rather than two separate ones. In many modern-day enterprise applications, there are countless instances of code in, say, Java, calling stored procedures in PL/SQL. As one who encounters this situation daily, I believe that a seamless transition of debugging from one language to another and the ability to see the call in a single stack would be of immense value to the multi-language developer. (Other useful applications of this technique are discussed below, in REF _Ref142590400 \h \* MERGEFORMAT Related work and the value of Dbdb and REF _Ref142590412 \h \* MERGEFORMAT Other language types chapters.)
To this end, I propose an implementation of a framework for single-stack debugging of multi-language applications based on the JDI (Java Debug Interface), an API layer of JPDA ADDIN EN.CITE 161612Java Platform Debugger Architecturehttp://java.sun.com/j2se/1.5.0/docs/guide/jpda/[2] (Java Platform Debugging Architecture).
About this document
In this paper I will present and discuss:
An overview of related work in this area, serving as proof that the proposed technique is not merely a whim, but offers a tangible benefit for real-world development
The benefits of the proposed solution over the existing related work
An overview of the JPDA in general, JDI in particular and the reasons for choosing it as a foundation of the proposed framework, along with a discussion of alternative choices.
The working proof of concept, including design, documentation, working code and demo. Here, I will also discuss:
the generalization of the proof of concept into a framework
directions for further extending the framework, and
use cases for plugging into the framework
Post-mortem of the project
Conclusion and thoughts on further work in this area
More up-to-date information
This project has been released as Open Source, as more than a few colleagues expressed interest in its ultimate value, and in working on it to achieve such. Thus, a reader is referred to the following up-to-date online maintained sources:
Homepage at HYPERLINK "http://db-db.sourceforge.net" http://db-db.sourceforge.net
Project [b]log at HYPERLINK "http://fish37.livejournal.com/tags/dbdb" http://fish37.livejournal.com/tag/dbdb
Project at Sourceforge.net, at HYPERLINK "http://sourceforge.net/projects/db-db/" http://sourceforge.net/projects/db-db/
DefectBug tracking at HYPERLINK "http://sourceforge.net/tracker/?group_id=150446&atid=777815" http://sourceforge.net/tracker/?group_id=150446&atid=777815
Demo at HYPERLINK "http://www.viewletcentral.com/vc/viewlet.html?id=87625530" http://www.viewletcentral.com/vc/viewlet.html?id=87625530
Tools
This project was implemented and tested with the Eclipse 3.2 IDE ( HYPERLINK "http://www.eclipse.org" www.eclipse.org), Java Standard Edition 6 Mustang ( HYPERLINK "https://mustang.dev.java.net/" https://mustang.dev.java.net/), and Oracle 10g.
As for the creation of this document, free trial of EndNote ( HYPERLINK "http://www.endnote.com/" www.endnote.com) was very useful for managing references in this document. HYPERLINK "http://www.phruby.com/stencildownload.html#Visio2003" http://www.phruby.com/stencildownload.html#Visio2003.
Related work and the value of Dbdb
The value of seamless, single-stack cross-language debugging is recognized in the industry. For example:
Stylus Studio features an XSL debugger features include the ability to seamlessly switch contexts and step into Java XSLT extension functions using Stylus Studio's integrated Java IDE, then returning back to calling the XSLT stylesheet. ADDIN EN.CITE 191910Stylus Studio's XSL Debuggerhttp://www.stylusstudio.com/xsl_debugger.html[3] (See also REF _Ref142589795 \h \* MERGEFORMAT Other language types chapter below).
According to Abdul Al-Azzawe, DB2 Development Tools Architect, IBM is investigating cross-language debugging support in the same call stack, such as mixing of Java and SQL nested stored procedure calls ADDIN EN.CITE Al-Azzawe20042243Abdul Al-AzzaweAbdul Al-Azzawe on development enhancements in DB2 Universal Database V8.2IBM developerWorksIBM developerWorks2004http://www-128.ibm.com/developerworks/db2/library/techarticle/dm-0409alazzawe/09/30/2004[4]. I am not aware of any implementation available to the public to date.
According to David Alpern of Oracle, their combined-VM algorithmsare more general [] and from the early stages of this work it wasclear to us that it would be desirable to also support combined-VMdebugging more generally for cross-tier calls of various sorts. However, it is unclear whether, if at all, this functionality (if it exists) will be available, as he notes that he is not at this time allowed to say when cross-tier debugging support might actually appear in our released products. ADDIN EN.CITE Alpern20053326David AlpernPersonal communication (e-mail)2005[5]
Engineers at BEA Systems also thought about this problem, and submitted a patent application, for a method that provides, in part, that if more that one language appears on a stack, a developer can see the frames for each language, as well as be able to inspect variables for each language. ADDIN EN.CITE Pugh20035512William A. PughJoshua Moll EckelsPatent Application: System for multi-language debugging, Provisional application No. 60/450,014.2003USABEA Systemshttp://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&u=%2Fnetahtml%2FPTO%2Fsearch-adv.html&r=1&p=1&f=G&l=50&d=PG01&S1=20040230955.PGNR.&OS=dn/20040230955&RS=DN/20040230955Fliesler Meyer[6] This particular offering is examined in more detail in REF _Ref142663246 \h \* MERGEFORMAT Appendix C. Criticism of BEAs Patent Application below.
Other possible uses of single-stack cross-language debugging includes stepping into C implementations of native Java methods, for example. In describing a technique debugging these kinds of mixed-language applications, Matthew White of IBM laments, do you effectively debug the Java/C hybrid programming since there is no debugger available that can check on this software chimera? ADDIN EN.CITE White20016643Matthew WhiteDebugging integrated Java and C/C++ code: Two approaches using JNIIBM developerWorksIBM developerWorks200111/01/2001http://www-128.ibm.com/developerworks/java/library/j-jnidebug/index.html[7]. Almost answering his plea are Eclipse developers, listing investigat[ing] debugging from Java into natives and back as potential debugger work items and areas of investigation for 2.1 and future releases of Eclipse ADDIN EN.CITE 2020122.0.1 Candidate Fixes: Possible Work Itemshttp://dev.eclipse.org/viewcvs/index.cgi/jdt-debug-home/DebugDirections2.1.htm?rev=1.11[8].
Additionally, Wrapped Application Debugger (WAD) ADDIN EN.CITE Beazley20018810David M. Beazley An Embedded Error Recovery and Debugging Mechanism for Scripting Language Extensions USENIX1471602001http://db.usenix.org/events/usenix01/full_papers/beazley/beazley_html/index.htmlCao20057743Jing CaoDavid M. BeazleyEmbedded Debugging of C/C++ Plugins and Extension ModulesTechnical report TR-2005-07, Department of Computer Science, University of ChicagoTechnical report TR-2005-07, Department of Computer Science, University of Chicago2005http://www.cs.uchicago.edu/files/tr_authentic/TR-2005-07.pdf[9, 10] attempts to address integration of C/C++ code with scripting language (Perl, Python, Tcl) extensions by embedding a debugger into the application. The authors complaint that little work has been done on the area of debugging in mixed-language environment can perhaps be assuaged, at least in part, by considerations presented here.
A single-stack view may also be of interest in debugging distributed applications using remote calls (RPC, RMI, EJB, etc.) Lovas et al. discuss a framework (using JPDA) usage for a step-into mechanism for remote method invocations (RMI) that unifies the debugging mechanism for local and remote calls ADDIN EN.CITE Lovas20029910Robert LovasVaidy SunderamDebugging of metacomputing applicationsInternational Parallel and Distributed Processing Symposium2002Ft.Lauterdale, Flhttp://www.mathcs.emory.edu/harness/pub/general/lovas4.pdf[11].
Further, a casual perusal of various discussion groups reveals the desire for the single-stack debugging popping up often enough; some of these is summarized in REF _Ref142667263 \h \* MERGEFORMAT Appendix D. Other use cases for single-stack debugging
And, finally, upon taking a quick look at HYPERLINK "http://db-db.sourceforge.net" http://db-db.sourceforge.net, Darin Wright of Eclipse Debug team had this to say: it certainly looks interesting. I know that people are interested in cross language debugging in general. [] I wonder if it could be applied in a more general way to address cross-language debugging, or if it is specific to stored procedures/SQL and Java? ADDIN EN.CITE Wright2006212126Darin WrightPersonal communication (e-mail)200607/07/2006[12]
Why JDI?
First, of course, in order to have a multi-language debugger (never mind combined call stacks yet), one needs a common framework that provides access to the needed debuggees information. I propose using the JDI layer of JPDA (Java Platform Debugger Architecture) as such a framework. As I will show, JDI is really quite language-independent (J for Java notwithstanding).
What is JDI?
JPDA consists of three layers of APIs; here, I propose using one of them, JDI (Java Debug Interface). For the discussion of the choice of this layer vs. other JPDA layers, or other similar approaches, see REF _Ref142667068 \h \* MERGEFORMAT Appendix E.
JDI is a set of Java interfaces, the implementations of which are to be provided by a debuggee that is, the target programs execution environment. At the root of it is the VirtualMachineinterface which represents the debuggee, that is, the target execution environment. While nominally the debuggee is meant to be a Java Virtual Machine, it does not have to be one. For example, it could be an interpreter, an intermediary program driving yet another debugger (say, gdb), or a native executable specially compiled. As long as the appropriate interfaces are implemented, any JPDA-compliant debugger can control any debuggee. By manipulating these implementations, the debugger is able to query and control the debuggees execution: suspend and resume threads, set breakpoints, evaluate variables and methods, etc. As we will see, these interfaces are generic enough to represent the execution of a program in any imperative programming language.
Two of the methods of VirtualMachine interface, eventRequestManager() and eventQueue(), return implementations of EventRequestManager and EventQueue, respectively. These two objects are the primary channels of communication between the debugger and debuggee. EventRequestManager is used by the debugger to request to be notified when a certain debuggable event occurs. (Such a request can also contain a directive to suspend the thread in which the event occurred). The debugger then polls EventQueue for the events it is interested in. Examples of such events are MethodEntryEvent, ThreadStartEvent, StepEvent, ExceptionEvent, etc.
Each event contains references to other artifacts of the debuggee. For example, as can be expected, a MethodEntryEvent contains a reference to the method entered, and to a thread in which this entry occurred. The thread is represented by ThreadReference interface, which further contains a reference to the stack of StackFrames in this thread. In turn, a StackFrame, which represents the state of one method invocation on a thread's call stack, has references to the variables visible in this frames scope.
Notice the lack of anything Java-specific. JDI, in fact, is so well designed that (especially with the adoption of JSR-45 ADDIN EN.CITE Bracha2003171712Gilad BrachaArthur RymanJSR (Java Specification Request) 45: Debugging Support for Other Languages2003http://www.jcp.org/en/jsr/detail?id=45[14]) the J for Java in JPDA is an unnecessary qualifier. Its set of abstractions is similar to that of Eclipses language-neutral debugging framework (centering on threads, stack frames, breakpoints and stepping into/over/out).
Finally, JDI is also used very creatively for such purposes as: dynamic analysis of running applications ADDIN EN.CITE Gueheneuc2002121243Yann-Gael GueheneucRemi DouenceNarendra JussienNo Java without Caffeine: A Tool for Dynamic Analysis of Java ProgramsEcole des Mines de NantesEcole des Mines de Nantes2002http://www.emn.fr/x-info/jussien/publications/gueheneuc-RR0207.pdf[15], self-healing applications ADDIN EN.CITE Savarese2002131343Daniel F. SavareseApplication, Heal ThyselfJavaProJavaPro2002http://www.fawcette.com/javapro/2002_09/magazine/columns/proshop/default_pf.aspx (login required)[16], load-time transformation of compiled Java programs for the purposes of Aspect-Oriented Programming ADDIN EN.CITE Kniesel15155Gunter KnieselPascal CostanzaMichael AustermannR.FilmanT.ElrodS.ClarkeJMangler - A Powerful Back-end for Aspect-oriented ProgrammingAspect-oriented Software Development2004Prentice Hallhttp://www.informatik.uni-bonn.de/~gk/papers/jmanglerChapterPreprint.pdf[17], dynamic creation of sequence and dependency diagrams from running programs ADDIN EN.CITE Loton2001141443Tony LotonUsing The Java Platform Debugger ArchitectureJava Developer's JournalJava Developer's Journal2001http://java.sys-con.com/read/36221.htm[18], etc.
A rough block diagram of JPDA-debugger integration is shown in REF _Ref143707024 \h Figure 1. The JVMDI and JDWP components of JPDA are discussed in REF _Ref142930781 \h \* MERGEFORMAT Appendix E. Why not JDI?.
Figure SEQ Figure \* ARABIC 1. JPDA - debugger block diagram.
Proof of concept
For a proof of concept, I will consider a simple Java program that calls a simple PL/SQL stored procedure. The Java program is org.hrum.dbdb.example.OracleExample1. We are interested in the fragment shown in REF _Ref139129199 \h \* MERGEFORMAT Listing 1. org.hrum.dbdb.OracleExample1 in Appendix A. The stored PL/SQL function FUNC, called from line 25 (and introduced in lines 20-21), and the function it calls in turn (FUNC2) are presented in REF _Ref143630846 \h Listing 2. PL/SQL stored procedures.
Architectural overview
Very roughly, a debugger capable of multiple language support (such as Eclipse) can be described by a block diagram in REF _Ref143710514 \h Figure 2. (Oracle prior to 10 is specified to provide yet another way of interfacing with the actual debugger; 10 and after support JDWP ADDIN EN.CITE Antognini1143Christian AntogniniDebugging PL/SQL and Java Stored Programs with JPDAhttp://www.trivadis.ch/Images/JPDA_tcm17-7136.pdf[19]).
Figure SEQ Figure \* ARABIC 2. Multi-language debugger
The proposed solution would look as shown in REF _Ref143714252 \h Figure 3. The Dbdb JDI implementation will interface to the debugger in the same way as the Java debugging module did. Everything else is hidden from the IDE. All communications between the IDE and the multiple debuggee targets is mediated by the Dbdb layer, which provides a single stack per thread executed, even though different frames of the stack may represent different languages.
Figure SEQ Figure \* ARABIC 3. Combined call-stack multi-language debugger
A sample sequence diagram of a debugging session with this approach is shown in REF _Ref143715656 \h Figure 4. (Here, Program 1 calls Program 2).
Figure SEQ Figure \* ARABIC 4. Sequence diagram of combined call-stack debugging.
The actual implementation is more involved than that, such as:
Since Program 1 and Dbdb JDI Implementation run asynchronously, the sequence of setting a breakpoint right after the method in Program 1 that calls Program 2 is reached is done by other means. For example, if Program 1 is a JPDA-enabled target, this is done by creating a MethodEntryRequest for the appropriate method that will suspend the thread as soon as the method is entered.
The modification of arguments step may not be necessary if a debugger can be launched separately and can then be attached to the debuggee process.
Infrastructure
The following are the basic building blocks for a JPDA-based combined call stack debugger.
DbdbVirtualMachine
Implementing VirtualMachine, this was originally intended to hold the virtual machines encountered throughout the debugging and switch among them. This is not the case (see below). The primarily function of this class is to present itself to the debugger and return DbdbEventQueue and DbdbRequestManagerInvocationHandler. Everything else this class does is there for a reason but is not really important.
DbdbRequestManagerInvocationHandler
Serves as an implementation of RequestManager for the virtual machines managed by DbdbVirtualMachine. The client (debuggers) requests are routed through this class, but components of Dbdb itself call the appropriate RequestManagers of managed virtual machines directly. More on this below, under REF _Ref139387865 \h \* MERGEFORMAT Mediators.
DbdbVirtualMachine.Registry
This is merely a collection of Maps, that holds mappings such as:
ThreadReference with which a DelegatingThreadReference was constructed to the DelegatingThreadReference that contains it
ObjectReference for a java.sql.Statement to an ObjectReference for a java.sql.Connection which created this Statement
A from2To Map, which associates a ThreadReference in one VM to the one in the VM that will be called from. For example, it will associate a ThreadReference that calls Statement.execute() in Java VM to the ThreadReference of the actual stored procedure that this Statement.execute() invokes in the Oracle VM.
NOTE: This one is specifically mentioned here, because it will be referred to heavily in the REF _Ref139390854 \h \* MERGEFORMAT Algorithm chapter below.
Etc. (See also REF _Ref139555809 \h Registry reform below).
EventListener hierarchy
Classes implementing this interface are called by the DbdbEventQueue to handle MethodEntryEvent and MethodExitEvent appropriately in their process() method. They are expected to be self-registering (registered upon instantiation). More defails are given below.
DbdbEventQueue
Implementation of EventQueue, that is responsible for:
polling EventQueues of all Virtual Machines being debugged
calling all EventListeners
deciding which events to return to the caller and which to consume (based on the return value from EventListener.process() method)
doing more things than it, perhaps, should upon encountering a StepEvent. This piece of logic should be removed from this class in order for Dbdb to be a more flexible framework. As it stands now, and as will be seen in the Algorithm chapter below, this class is responsible for recognizing
More on this below, under REF _Ref139387865 \h \* MERGEFORMAT Mediators.
DelegatingThreadReference
An implementation of ThreadReference that is responsible for exposing a unified call stack to the user. Internally, it holds a stack of ThreadReferences from managed virtual machines. The Dbdb framework (normally, various EventListeners) will push to and pop this stack as needed.
FakeStepEvent, DelegatingStackFrame, DbdbEventSet
Further infrastructure for supporting the DelegatingThreadReference.
org.hrum.dbdb.plugin package
This package contains the Eclipse plug-in for the Dbdb functionality. It includes code for dynamically looking up code for the stored procedures and displaying it in an editor.
Algorithm
The algorithm employed is as follows (we will use the OracleExample1 program for illustration):
When a debuggee is launched, the DbdbVirtualMachine is initialized with:
VirtualMachine object gotten from the appropriate Connector
Currently, the following EventListeners (others can be added, of course):
__java_sql_DriverManagerListener
__oracle_jdbc_driver_PhysicalConnectionListener
__java_lang_RuntimeListener
These are explained below.
On receiving MethodExitEvent from DriverManager.getConnection() method (line 15), __java_sql_DriverManagerListener:
Obtains the reference to the java.sql.Connection object as the MethodExitEvents returnValue().
Starts a thread with a ListeningConnector (com.sun.jdi.SocketListen, or, for Eclipse, org.eclipse.jdi.internal.connect.SocketListeningConnectorImpl, listening on a port.
Starts a thread that will execute DBMS_DEBUG_JDWP.CONNECT_TCP ADDIN EN.CITE Antognini1143Christian AntogniniDebugging PL/SQL and Java Stored Programs with JPDAhttp://www.trivadis.ch/Images/JPDA_tcm17-7136.pdf[19] on the connection obtained above. This will establish a debugging session within Oracle.
Waits until the connection is made, and obtains the Oracles virtual machine from the ListeningConnector.
Instantiates a of OracleDbmsMethodEntryEventListener, which will listen for MethodEntryEvents from the Oracles virtual machine. This listener, during construction, associates (in Registry.from2To, see REF _Ref139390930 \w \p \h 3 above) the Java thread it the connection was created in with a DelegatingThreadReference.WAITING_FOR_THREAD_REFERENCE constant.
On receiving a MethodExitEvent from Connection.prepareCall() (line 20-21), __oracle_jdbc_driver_PhysicalConnectionListener:
Loads the source code for the stored procedure being prepared from Oracle (a little earlier than necessary, but why not)
Associates reference to the returned java.sql.Statement from the MethodExitEvents returnValue() to the reference to java.sql.Connection which created the statement.
On receiving a MethodEntryEvent from an Oracle stored procedure, OracleDbmsMethodEntryEventListener:
Finds the ThreadReference for the Java thread created in its constructor (see REF _Ref139353612 \r \p \h 2.e above), creates a DelegatingThreadReference for it, and pushes onto it the ThreadReference from the Oracle stored procedure.
This step also associates the found ThreadReference with the DelegatingThreadReference in the Registry (instead of earlier WAITING_FOR_THREAD_REFERENCE created in REF _Ref139353612 \r \p \h 2.e above)
Instantiates an OracleDbmsMethodExitEventListener
Returns, for the debugger, a FakeStepEvent, with this DelegatingThreadReference.
On receiving a MethodExitEvent from an Oracle stored procedure, OracleDbmsMethodExitEventListener (see REF _Ref139390388 \w \p \h 4.b above):
Executes a popThreadReference() on the DelegatingThreadReference instance it is constructed with
Resumes the thread that originally called the popped thread.
Mediators
All the while, Dbdbs implementations of EventRequestManager. and EventQueue mediate the interaction between the debugger and the debuggee as follows:
DbdbRequestManagerInvocationHandler, merely, forwards all requests to EventRequestManagers of all managed VMs. The only special case is a call to createStepRequest(). If this is received, the Registry.from2To map (see REF _Ref139390930 \w \p \h 3 above)is checked. If the ThreadReference on which the step is to be created maps to a to ThreadReference object which is not a WAITING_FOR_THREAD_REFERENCE constant, the request is forwarded, of course, to the VM of the to ThreadReference.
DbdbEventQueue, every time the debugger wishes to retrieve an EventSet from the VM it thinks it queries, does the following:
Polls EventQueues of all managed VMs. All obtained EventSets are added to a local q2 variable.
If q2 is empty, just return null.
Create a new instance of DbdbEventSet, called toReturn. This will be the EventSet that will be returned to the debugger; its VM is DbdbVirtualMachine.
Removes the EventSet from the local q2 variable, and, for each Event evt, executes the algorithm shown below, in REF _Ref139650278 \h Figure 4. DbdbEventQueue algorithm, part 1 .
Finally, if toReturn set is not empty, it is returned. If it is:
If resumed is false, a resume is called on it.
A null is returned.
Figure SEQ Figure \* ARABIC 5. DbdbEventQueue algorithm, part 1
Figure SEQ Figure \* ARABIC 6. DbdbEventQueue algorithm, part 2.
The resulting combined call stack is shown in REF _Ref139388631 \h \* MERGEFORMAT Appendix B. Screenshots .
Using Dbdb as a framework
While this may not be a full-fledged framework at this point, let us consider two use cases of a developer wishing to extend this approach.
Registry reform
First and foremost, a better solution should be found for REF _Ref139388823 \h \* MERGEFORMAT DbdbVirtualMachine.Registry, for this to be truly extensible. But this should, perhaps, wait, until the Dbdb applications has been extended further.
Some thought may be given of implementing a simple relational-database-like data structure in a programming language. In Java, for instance, Map is nice, but what if you could have an easy SymmetricMap, or, going further, something resembling an RDBMS table structure (without the complexity of triggers et al.)
Mediator policy
Secondly, then, a policy of using Dbdb implementations of com.sun.jdi.* interfaces (such as VirtualMachine, Event, EventManager, EventRequest, etc.) vs. managed VMs implementations needs to be clarified. In other words, when is a Dbdb implementation returned to the debugger, and when is it an implementation for the managed VM that the client is currently interested in? Currently it is not explicitly defined. It is my opinion that a few more extensions of Dbdb are needed (such as are outlined in use cases below) in order to formulate this policy better.
Use cases
Now, to the more general things
Actions required of a developer using Dbdb in the use cases listed below could be generalized; in other words, made simpler or pluggable. The comments on how to do that are in corresponding footnote for the item.
Use case 1: Repeat the proof-of-concept on another JDBC-compliant RDBMS
Find out the appropriate implementation of java.sql.Statement for this RDBMS, and create an EventListener for it, using __oracle_jdbc_driver_PhysicalConnectionListener as a template. In particular:
It is advised that this EventListener extend the MethodXEventListener, which would require for it to be named in the following pattern (as should be obvious):
The name starts with __ (two underscores).
The rest of the name is the fully qualified name of the class implementing java.sql.Statement for this JDBC vendor, with periods replaced by underscores, and a suffix Listener added at the end.
Register this EventListener with the DbdbVirtualMachine.
Modify __java_sql_DriverManagerListener to:
Actually parse the arguments to getConnection() method(s) to see whether the JDBC URL provided is the one that is handled by the target RDBMS.
If so, get the appropriate com.sun.jdi.VirtualMachine implementation for the debuggee. (If, unlike Oracle, no direct JPDA-compliant VirtualMachine implementation is available), see REF _Ref139471369 \h \* MERGEFORMAT Use case 2. Implement cross-language debugging of other languages below.
Implement the equivalents of OracleDbmsMethodEntryEventListener and OracleDbmsMethodExitEventListener, using the above classes as guidelines.
Use case 2. Implement cross-language debugging of other languages supporting JPDA
For the sake of example, let us consider a made-up need to debug another Java program called from our Java program. All that needs to be done is an implementation of DbdbProcessDebugger has to be created (the beginnings can be seen in JavaProcessDebugger class). The fully qualified name of this class has to be registered in dbdb_debuggers.properties file which must be on the classpath.
This is how it works.
Use case 3. Implement cross-language debugging of other languages
For the sake of example, let us consider a made-up need to debug a Perl script called from Java. In this case, in addition to performing the steps from the above use cases, a developer would need to provide the adapters from the target debugging framework ADDIN EN.CITE Foy2001222243Brian D. FoyUsing the Perl DebuggerDr.Dobb's JournalDr.Dobb's Journal2001http://www.ddj.com/184404744[20] to implementations of com.sun.jdi.VirtualMachine and all related artifacts.
Framework or tool?
As can be seen above, while this not a defined framework/API, it is certainly poised to evolve into one.
Post-mortem
A postmortem is a useful exercise. Not that I believe that this project is dead, but this concept can and should be applied even to a release. Thus, for this proof-of-concept, here is a post-mortem. It explains some pitfalls, false leads, and reasons for this project taking more than it should.
Leaky abstractons
The following Leaky abstractions ADDIN EN.CITE Spolsky2002232343Joel SpolskyThe Law of Leaky Abstractions2002http://www.joelonsoftware.com/articles/LeakyAbstractions.html[21] certainly took away time and effort:
For all their talk about good OO design and interfaces, Sun shows that they are sinners as well. It is nice to learn that Sun's implementation when manipulating ostensibly insterfaces, in fact, expects them to be implementation classes inside. For example, com.sun.jdi.MirrorImpl.validateMirrorOrNull() inside casts things to com.sun.jdi.MirrorImpl rather than com.sun.jdi.Mirror. Ditto for ThreadReference vs ThreadReferenceImpl.
This is not a big deal, but a caveat nonetheless for those attempting to integrate Dbdb with exiting debuggers. What is one to do with a null EventSet? What about an empty one? Javadt does not like a null EventSet -- it just does not check for nulls (which is ok, for a throwaway reference implementation). But an empty one is not good for Eclipse, because it indiscriminately calls a resume() on it, which is not what we want. Back to returning null then. But a thought: how many of such little things would render this "framework" not really a framework (still, certainly, usable, but just a nagging feeling)? Or should this all be configurable?
Speaking too soon: false starts and misdirections
Because of my general fear of having to spend time on learning Eclipses plug-in architecture, I proposed putting off integration with this very popular IDE in favor of using simpler tools. But indeed Quis emendabit ipsos emendatra. Eclipse should have been used from the very beginning. This would have saved quite a lot of time and I would never learn about the leaky abstraction 2, above
On a related note, see the chapter REF _Ref142585703 \h \* MERGEFORMAT Why JDI? above for the discussion of choosing JPDA over Eclipse Debug Framework. Had I started again, I would probably had used the latter
The idea of using JDWP was judged to be an overkill, and ultimately abandoned (but see also REF _Ref142930781 \h \* MERGEFORMAT Appendix E. Why not JDI?). The discovery of Oracles support of JPDA-based debugging would have sped things up. But it happened after I spent inordinate amount of time on task that ultimately is not needed researching ways of using existing JDWP implementations. JSR-45 is not needed as well.
The idea of loading debugged classes with an alternate class loader, which would substitute real classes (in this case, JDBC drivers and their subsequent artifacts) for their equivalents, and attendant ideas (such as using bytecode modification) did not find its use, despite much time spent on this research. This was due to:
Mouthing off about this in proposal without really knowing that much about JPDA
It is hard to debug (but see REF _Ref139292670 \h \* MERGEFORMAT Parallel drill-down chapter below).
However, this may yet find its use. I just have not thought about it again
I have labored for some time believing that returnValue() feature exists in JPDA. When I realized that it does not, I was forced to produce an elaborate workaround, which was quite complicated and not extremely robust. It is only after the workaround has been created that I found that this feature indeed exists in JPDA starting with Java 6 so add more time to clean up the ugly workaround, to use more elegant solution with returnValue().
Further work
The project is released to open-source, and will continue to evolve in its current incarnation as an Eclipse plug-in primarily for stored procedure debugging. In this chapter I will outline some other interesting possibilities for this project.
Other language types
It would be interesting to develop a use for this framework for debugging not just procedural/imperative languages for which it is created, but also for:
Logical/constraint languages (such as Prolog, which has been adapted for Eclipse ADDIN EN.CITE Kroening2005242443Mary KroeningEclipse: Adapting and Updating an IDEDr.Dobb's JournalDr.Dobb's Journal2005http://www.ddj.com/184407751[22]).
Perhaps as a subset of this functionality, or of an application in its own right, could be adaptation of this technique for debugging parsers (such as those based on grammars ANTLR, yacc, etc.)
Other declarative languages, such as XSLT.
Cross-cutting concerns/Aspects (I have not even begun to appreciate this, but I thought Id throw it into the mix).
As with Prolog implementation, the three areas above may involve some creative redefinition of semantics of certain debugging operations, such as stepping. These problems, as applied to declarative, and an Eclipse-based solution, are presented in detail by Hui Wu et al. in Grammar-Driven Generation of Domain-Specific Language Testing Tools ADDIN EN.CITE Wu2005323210Hui WuJeff CrayMarjan MernikGrammar-Driven Generation of Domain-Specific Language Testing ToolsOOPSLA2005San Diegohttp://www.cis.uab.edu/gray/Pubs/ddf.pdf[23].
Stepping into, C/C++ implementation of Java native methods, or, more generally, into assembly/machine code/byte code (but this is, perhaps, more of a topic for the REF _Ref139292670 \h \* MERGEFORMAT Parallel drill-down approach, outlined below).
Parallel drill-down
Often, the call stack actually executed is different from the abstract call stack though of by a developer (the latter being the one depicted as a sequence diagram). In this example, it is useful for a developer to think of a call to java.sql.Statement.execute() as logically leading directly into the stored procedure.
For example, from a standpoint of an application developer, it would be good do away with the Oracle implementation, which we currently include in a combined stack shown in REF _Ref139388631 \h \* MERGEFORMAT Appendix B. Screenshots . In other words, in this particular screenshot, your average application developer is only interested in first (the Java frame containing a Statement.execute() call) and last (in this screenshot, the first and only, but in general, the first PL/SQL frame) stack frames of the shown call stack. On the other hand, for a developer of Oracles JDBC driver, both may be useful.
For a lack of a better term, Id like to coin one: parallel drill-down (OK, so its not a cool term). This would mean creating a framework for debugging where at a point where logical and implementation scenarios diverge, the developer is presented with multiple parallel call stacks to step through. Any path can be selected, and the developer can switch between paths at will; a step in one path must be synchronized with other possible paths. For all my criticism of the BEA patent application (see REF _Ref143631569 \h \* MERGEFORMAT Appendix C. Criticism of BEAs Patent Application), the authors correctly identified this particular problem when they wrote: For example, it is not uncommon for a developer to see stack information not directly related to the software being debugged when encountering a stack frame for one language, when using a debugger intended for another language. As another example, when using a debugger intended for the Java language, a Java stack will not include the XScript [] stack, and can sometimes show the set of Java classes that implement the XScript engine (these are part of the environment, but not the software the developer is working on). ADDIN EN.CITE Pugh20035512William A. PughJoshua Moll EckelsPatent Application: System for multi-language debugging, Provisional application No. 60/450,014.2003USABEA Systemshttp://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&u=%2Fnetahtml%2FPTO%2Fsearch-adv.html&r=1&p=1&f=G&l=50&d=PG01&S1=20040230955.PGNR.&OS=dn/20040230955&RS=DN/20040230955Fliesler Meyer[6]
Obviously, there is no limit to splitting these, however, common scenarios can be a well-defined and be a part of the framework, while also providing the developer with a way to define her own. Here, SMAP mechanism of JSR-45 can probably be leveraged.
Another example is a call to by reflection or related things. For instance, debugging things using java.lang.reflect.Proxy mechanism (such as Dbdbs own) is notoriously painful. We usually want logical debugging the developer may not want to see internal JDK infrastructure, but sometimes we may want implementation one. Since reflection is quite common when using generic framework, this may be a nice-to-have feature.
As an example of the usefulness of this feature, consider the screenshot in REF _Ref139388631 \h \* MERGEFORMAT Appendix B. Screenshots From the point of view of the application developer, the stack frames between the application code calling Statement.execute() and the stored procedure code are extraneous plumbing.
If this feature is implemented, providers of generic frameworks can provide the split scenarios as part of the product, allowing the user of the framework an easier way to separate the debugging of her logic vs. the compound logic of her components and the underlying framework.
This approach would be even more useful to programs that are worse than those using reflection those using byte code manipulation.
Other features
Other features that are not implemented in proof of concept, but would be useful, include
drop-to-frame functionality, allowing popping a selected stack frame (and all frames above it) from the execution stack and then stepping back into the frame. ADDIN EN.CITE 353512Eclipse 3.1, documentation of org.eclipse.debug.core.model.IDropToFrame interface (Javadoc)http://help.eclipse.org/help31/index.jsp?topic=/org.eclipse.platform.doc.isv/reference/api/org/eclipse/debug/core/model/IDropToFrame.html[24].
Conclusion
Dixi.
Appendix A. Code listings
Listing 1. org.hrum.dbdb.OracleExample1
01: package org.hrum.dbdb.example;
02:
03: import java.sql.CallableStatement;
04: import java.sql.Connection;
05: import java.sql.DriverManager;
06: import java.sql.Types;
07:
08: public class OracleExample1 {
10:
11: public static void main(String[] args) {
12: try {
13: Class.forName("oracle.jdbc.driver.OracleDriver");
14: System.out.println("DriverManager.getConnection()");
15: Connection con = DriverManager.getConnection(
16: "jdbc:oracle:thin:@localhost:1521:dbdb",
17: "sys",
18: "password");
19: System.out.println("con.prepareCall()");
20: CallableStatement st =
21: con.prepareCall("begin ?:=func(?); end;");
22: st.setInt(2, 1);
23: st.registerOutParameter(1, Types.INTEGER);
24: System.out.println("Statement.execute()");
25: st.execute();
26: System.out.println(st.getObject(1));
27: } catch (Exception e) {
28: e.printStackTrace(System.out);
29: System.exit(-1);
30: }
31: }
32: }
Listing 2. PL/SQL stored procedures
The PL/SQL stored procedures are created with the following code:
create or replace function func2(num number) return number
is
begin
return num * 3;
end;
/
create or replace function func(ctr number) return number
is
ret number:=0;
begin
for i in 1..ctr loop
ret := ret + func2(i);
ret := ret + 3;
ret := ret - 3;
end loop;
return ret;
end;
/
alter function func compile debug;
alter function func2 compile debug;
Appendix B. Screenshots
Appendix C. Criticism of BEAs Patent Application
Id like to expound more on this particular proposition, as it is a claim for a patent, especially in light of claims that Oracle has a similar patent filing ADDIN EN.CITE Alpern20053326David AlpernPersonal communication (e-mail)2005[5]. No implementation has been made available to my knowledge; and the approach itself, as described in the application, does not seem to be novel and non-obvious in November of 2004. At this time both JPDA and Eclipse Debug Framework (discussed in REF _Ref142585703 \h \* MERGEFORMAT Why JDI? chapter below) were already available, and given that, a remark that creating debugging tools that can be applied to software applied to more than one programming language, and running in the same environment, has proved to be extremely difficult seems particularly disingenuous. But at least they saw a need, and briefly identified (but not addressed) some problems I also discuss below, in the REF _Ref139292670 \h \* MERGEFORMAT Parallel drill-down chapter below.
Further, consider this pronouncement One multi-language debugger, JSR 45, can only be used to debug languages that are easily transformed into Java and then compiled. Such a statement makes one wonder whether this was merely lost in translation to the patent attorney, or the claimants misunderstand or, for the purposes of patent claim, misrepresent, JSR 45. First, it is not a debugger, but a specification. Second, JSR 45s stated goal is to establish [a] mechanism [] by which programs executed under the Java virtual machine but written in languages other than the Java programming language, can be debugged with references to the original source ADDIN EN.CITE Bracha2003171712Gilad BrachaArthur RymanJSR (Java Specification Request) 45: Debugging Support for Other Languages2003http://www.jcp.org/en/jsr/detail?id=45[14] (Emphasis mine). This renders their next point a non-sequitur: This and most other multi-language debuggers won't work with languages such as XScript that where [sic] the language will be run by an interpreter or the language can not be mapped directly to Java because, for example, the language has a different data structure. Perhaps what they are trying to say is that this is the case when a language cannot be easily mapped to a procedural/imperative model (like XScript, perhaps), which indeed seems to be the case with JPDA. However, they provide no proof that their model does...
Appendix D. Other use cases for single-stack debugging
I'm wondering: do you have ideas on how to support cross-language debugging for RDT, ie. a (J)Ruby methods that call Java methods (and the other way around), ie. the StackTrace would contain both Ruby and Java stack frames. The question is: if the JRuby process is launched with the Debugger, you'll have a JDT DebugModel, but you'll also want a Ruby DebugModel. I'm not sure if there's an easy way to combing StackTraces from both DebugModels (maybe some kind of Delegating DebugModel that collects StackTraces from JDT and Ruby DebugModels).
HYPERLINK "http://www.mail-archive.com/rubyeclipse-development@lists.sourceforge.net/msg00042.html" http://www.mail-archive.com/rubyeclipse-development@lists.sourceforge.net/msg00042.html
Is there an easy way to set up integrated debugging between a Java application and PL/SQL?
A reply to a tutorial on using DBMS_DEBUG, Oracles debugging package at HYPERLINK "http://www.orablogs.com/shay/archives/001571.html" http://www.orablogs.com/shay/archives/001571.html
In my private fantasy land, I'd be able to run emacs and somehow invoke pdb with pdbtrack to do source-level debugging of my python code, then automagically step into gdb when the python calls out to C++ code I've written via boost::python.
HYPERLINK "http://mail.python.org/pipermail/c++-sig/2003-June/004315.html" http://mail.python.org/pipermail/c++-sig/2003-June/004315.html
Appendix E. Why not JDI?
The choice of JDI was a bit opportunistic. It did, indeed, strike me as a fairly generic framework; however, its choice was also influenced by my greater familiarity with it (versus REF _Ref142928747 \h \* MERGEFORMAT Eclipse Debug Framework) and by relative ease of implementation of the proof-of-concept for a JPDA-compliant debugger (versus JDWP, see below). This chapter attempts to examine alternatives to JDI for the implementation
JDWP (Java Debug Wire Protocol)
The two API layers left unexamined in here are JDWP (Java Debug Wire Protocol), which which defines the format of information and requests transferred between the debugging process and the debugger front end ADDIN EN.CITE 161612Java Platform Debugger Architecturehttp://java.sun.com/j2se/1.5.0/docs/guide/jpda/[2] and JVMTI (JVM Tools Interface), which a programming interface [that] provides both a way to inspect the state and to control the execution of applications running in the Java virtual machine ADDIN EN.CITE 282812JVM Tool Interface http://java.sun.com/j2se/1.5.0/docs/guide/jvmti/jvmti.html[25]. The latter is outside the scope of this paper. But JDWP deserves further examination here.
JDWP provides a packet-based, stateless protocol for communication between a debugger and a debuggee. It can be said that it defines a serialization model for the JDI objects discussed in chapter REF _Ref142929524 \h \* MERGEFORMAT What is JDI?, above. Another way of saying it is that the Suns JDI API is merely a reference implementation, in Java, of JDWP. In this way, it is completely agnostic as to the implementation of a debugger and can be used as a universal debugging protocol. As the JPDA FAQ points out, [t]heoretically JPDA could have only one interface, the Java Debug Wire Protocol (JDWP) ADDIN EN.CITE 262612Java Platform Debugger Architecture FAQhttp://java.sun.com/products/jpda/faq.html[26]. Indeed, there already is a Common LISP ADDIN EN.CITE Lichteblau272712David LichteblauCL-JDI (a Common Lisp implementation of Java Debug Protocol)http://www.lichteblau.com/cloak/cl-jdi/README.html[27] and a Ruby ADDIN EN.CITE Kilmer2003292947Richard KilmerRuby and the Java Debug Wire Protocol: What were we thinking?RubyConf 20032003http://www.zenspider.com/dl/rubyconf2003/RubyAndJDWP.pdfFowler303012Chad FowlerRichard KilmerRuby implementation of the Java Debug Wire Protocolhttp://rubyforge.org/projects/rubyjdwp/[28, 29] implementation of JDWP.
At this time, despite the two above-mentioned (and encouraging) cases, pretty much all JDWP-aware debuggers are those that are fully JPDA-aware; that is, these are Java debuggers, which already use JDI API. If it is recognized that JDWP can be used not just as a Java debugging protocol, but as a generic one, and other debuggers are aware of it, the single-stack implementation could perhaps be pushed down to the layer that is responsible for reading/writing JDWP packets and creating the data structures, rather than on top of the data structures themselves. In fact, as we saw earlier (and made use of in proof of concept), Oracle, as of version 9i, provides for JDWP-based remote debugging of its stored procedures not only Java, but PL/SQL ADDIN EN.CITE Antognini1143Christian AntogniniDebugging PL/SQL and Java Stored Programs with JPDAhttp://www.trivadis.ch/Images/JPDA_tcm17-7136.pdf[19]. This is a real-world example of JPDA usage for non-Java languages.
It is true that [w]riting directly to JDWP however is painstaking work, information sent across the wire must be read and written precisely. But implementations for doing that already exist in Java (GNU Classpath ADDIN EN.CITE 313112GNU Classpathhttp://www.gnu.org/software/classpath/[30]), C (Suns own implementation of JDWP), and, as we have seen above, in Common LISP and Ruby. As it is a well-designed protocol, perhaps the next iteration of this project should be rewritten with JDWP in mind.
Both JDWP and JDI approaches can be used, out of the box, for remote debugging, which is very useful.
And, one a final thought. As I mentioned in chapter REF _Ref142929524 \h \* MERGEFORMAT What is JDI?, when debugging a native executable, it may be simple enough to create Java mediator programs that would drive the third-party debugger (e.g., gdb) and translate their interactions with this debugger into the objects implementing JDI API. Driving the third-party debugger through, perhaps, a command-line interface (CLI) seems straightforward. However, in case of the gdb debugger, it was found that CLI has proven to be highly unreliable and Eclipse has since switched to use machine-oriented text interface ADDIN EN.CITE Leszek333343Pawel LeszekC/C++ development with the Eclipse Platform: How to use the C/C++ Development Toolkit (CDT)IBM developerWorksIBM developerWorkshttp://www-128.ibm.com/developerworks/opensource/library/os-ecc/[13] which is quickly becoming a de facto standard for integrating debuggers into a variety of environments ADDIN EN.CITE Roberts343412Nick RobertsBob RossiEli ZaretskiiDebugger Machine Interfacehttp://www.freestandards.org/en/DMI[31]. But should using CLI with another third-party debugger present a problem, and there is no MI-like solution, perhaps it could be more efficient to devise a way to enhance a compiler so that the executable itself is JDWP-compliant.
Eclipse Debug Framework
Obviously, well-designed IDEs that provide multi-language debugging have also implemented language-independent debugging frameworks. Perhaps one of the most well-adopted and best-designed of these is Eclipse Debug Framework ADDIN EN.CITE Wright2004181843Darin WrightBjorn Freeman-BensonHow to write an Eclipse debugger2004http://www.eclipse.org/articles/Article-Debugger/how-to.html[32] (hereafter, EDF). Because it is language-neutral, one may think it a naturally better candidate for this project. Here, I will examine this proposition. Full disclosure: large portion of the work was already completed using JDI by the time I have more thoroughly familiarized myself with this alternative, which may make me seem biased against it
Pro EDF
It is intended to be language-independent. Its design, centered around the notion of threads, stack frames, breakpoints and stepping seems similar to JPDA and so seemingly not well-suited for languages that are not procedural/imperative languages (see the REF _Ref142592244 \h \* MERGEFORMAT Other language types chapter below). However, even those can be accommodated: for example, a working Prolog debugger ADDIN EN.CITE Kroening2005242443Mary KroeningEclipse: Adapting and Updating an IDEDr.Dobb's JournalDr.Dobb's Journal2005http://www.ddj.com/184407751Kroening252512Mary KroeningAmzi! Prolog: Source Code Debuggerhttp://www.amzi.com/manuals/amzi7/pro/pug_debugger_ide.htm[22, 33] utilizing EDF creatively redefines the semantics of stepping into and out ADDIN EN.CITE Kroening252512Mary KroeningAmzi! Prolog: Source Code Debuggerhttp://www.amzi.com/manuals/amzi7/pro/pug_debugger_ide.htm[33].
For Eclipse-based debuggers, implementing single-stack debugging in this layer (in a way probably very similar to the one described in this paper) is easier, because no adaptation is needed for a new language to support JPDA.
Pro JPDA
EDF is indeed designed for [m]odeling a set of artifacts and actions common to many debuggers, known as the debug model. For example, some common debug artifacts are threads, stack frames, variables and breakpoints; and some common actions are suspending, stepping, resuming and terminating. ADDIN EN.CITE Wright2004181843Darin WrightBjorn Freeman-BensonHow to write an Eclipse debugger2004http://www.eclipse.org/articles/Article-Debugger/how-to.html[32] However, as we saw earlier, a functionally equivalent set artifacts and actions is available via JPDA. The Pragmatics of Java Debugging article also hints at the cross-language potential of this architecture ADDIN EN.CITE Winchester2001111143Joe WinchesterArthur RymanThe Pragmatics Of Java DebuggingSys-Con IndiaSys-Con India2001http://in.sys-con.com/read/36223.htm[34].
For JPDA-compliant non-Eclipse-based debuggers (and there are a number of those, the obvious advantage is that EDF does not need to be re-implemented. Further, because of the definition of JDWP, and existing implementations of it in a number of languages (Java, C, Ruby and LISP), it can be easier integrated with debuggers not written in Java, and allow for remote debugging as an added bonus.
Why not, indeed?
As we have seen above, the choice of JDI was largely expedience and opportunism. This implementation certainly does the job, and JDI can be used in similar applications to great success. But I am not going to rationalize my choices to no end. I believe that the best implementation is JDWP, as it is closer to a universal debugging protocol.
Appendix F. References
ADDIN EN.REFLIST 1. SCORE: Multi-Language Debugger. HYPERLINK "http://www.info.uni-karlsruhe.de/~andf/documents/scoremld.pdf" http://www.info.uni-karlsruhe.de/~andf/documents/scoremld.pdf
2. Java Platform Debugger Architecture. HYPERLINK "http://java.sun.com/j2se/1.5.0/docs/guide/jpda/" http://java.sun.com/j2se/1.5.0/docs/guide/jpda/
3. Stylus Studio's XSL Debugger HYPERLINK "http://www.stylusstudio.com/xsl_debugger.html" http://www.stylusstudio.com/xsl_debugger.html
4. Al-Azzawe, A. (2004) Abdul Al-Azzawe on development enhancements in DB2 Universal Database V8.2. IBM developerWorks. HYPERLINK "http://www-128.ibm.com/developerworks/db2/library/techarticle/dm-0409alazzawe/" http://www-128.ibm.com/developerworks/db2/library/techarticle/dm-0409alazzawe/
5. Alpern, D., Personal communication (e-mail). 2005.
6. Pugh, W.A. and J.M. Eckels. Patent Application: System for multi-language debugging, Provisional application No. 60/450,014. 2003 HYPERLINK "http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&u=%2Fnetahtml%2FPTO%2Fsearch-adv.html&r=1&p=1&f=G&l=50&d=PG01&S1=20040230955.PGNR.&OS=dn/20040230955&RS=DN/20040230955" http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&u=%2Fnetahtml%2FPTO%2Fsearch-adv.html&r=1&p=1&f=G&l=50&d=PG01&S1=20040230955.PGNR.&OS=dn/20040230955&RS=DN/20040230955
7. White, M. (2001) Debugging integrated Java and C/C++ code: Two approaches using JNI. IBM developerWorks. HYPERLINK "http://www-128.ibm.com/developerworks/java/library/j-jnidebug/index.html" http://www-128.ibm.com/developerworks/java/library/j-jnidebug/index.html
8. 2.0.1 Candidate Fixes: Possible Work Items. HYPERLINK "http://dev.eclipse.org/viewcvs/index.cgi/jdt-debug-home/DebugDirections2.1.htm?rev=1.11" http://dev.eclipse.org/viewcvs/index.cgi/jdt-debug-home/DebugDirections2.1.htm?rev=1.11
9. Beazley, D.M. An Embedded Error Recovery and Debugging Mechanism for Scripting Language Extensions in USENIX. 2001 HYPERLINK "http://db.usenix.org/events/usenix01/full_papers/beazley/beazley_html/index.html" http://db.usenix.org/events/usenix01/full_papers/beazley/beazley_html/index.html
10. Cao, J. and D.M. Beazley (2005) Embedded Debugging of C/C++ Plugins and Extension Modules. Technical report TR-2005-07, Department of Computer Science, University of Chicago. HYPERLINK "http://www.cs.uchicago.edu/files/tr_authentic/TR-2005-07.pdf" http://www.cs.uchicago.edu/files/tr_authentic/TR-2005-07.pdf
11. Lovas, R. and V. Sunderam. Debugging of metacomputing applications. in International Parallel and Distributed Processing Symposium. 2002. Ft.Lauterdale, Fl HYPERLINK "http://www.mathcs.emory.edu/harness/pub/general/lovas4.pdf" http://www.mathcs.emory.edu/harness/pub/general/lovas4.pdf
12. Wright, D., Personal communication (e-mail). 2006.
13. Leszek, P. C/C++ development with the Eclipse Platform: How to use the C/C++ Development Toolkit (CDT). IBM developerWorks. HYPERLINK "http://www-128.ibm.com/developerworks/opensource/library/os-ecc/" http://www-128.ibm.com/developerworks/opensource/library/os-ecc/
14. Bracha, G. and A. Ryman. JSR (Java Specification Request) 45: Debugging Support for Other Languages. 2003 HYPERLINK "http://www.jcp.org/en/jsr/detail?id=45" http://www.jcp.org/en/jsr/detail?id=45
15. Gueheneuc, Y.-G., R. Douence, and N. Jussien (2002) No Java without Caffeine: A Tool for Dynamic Analysis of Java Programs. Ecole des Mines de Nantes. HYPERLINK "http://www.emn.fr/x-info/jussien/publications/gueheneuc-RR0207.pdf" http://www.emn.fr/x-info/jussien/publications/gueheneuc-RR0207.pdf
16. Savarese, D.F. (2002) Application, Heal Thyself. JavaPro. HYPERLINK "http://www.fawcette.com/javapro/2002_09/magazine/columns/proshop/default_pf.aspx" http://www.fawcette.com/javapro/2002_09/magazine/columns/proshop/default_pf.aspx (login required)
17. Kniesel, G., P. Costanza, and M. Austermann, JMangler - A Powerful Back-end for Aspect-oriented Programming, in Aspect-oriented Software Development, R.Filman, T.Elrod, and S.Clarke, Editors. 2004, Prentice Hall HYPERLINK "http://www.informatik.uni-bonn.de/~gk/papers/jmanglerChapterPreprint.pdf" http://www.informatik.uni-bonn.de/~gk/papers/jmanglerChapterPreprint.pdf
18. Loton, T. (2001) Using The Java Platform Debugger Architecture. Java Developer's Journal. HYPERLINK "http://java.sys-con.com/read/36221.htm" http://java.sys-con.com/read/36221.htm
19. Antognini, C. Debugging PL/SQL and Java Stored Programs with JPDA. HYPERLINK "http://www.trivadis.ch/Images/JPDA_tcm17-7136.pdf" http://www.trivadis.ch/Images/JPDA_tcm17-7136.pdf
20. Foy, B.D. (2001) Using the Perl Debugger. Dr.Dobb's Journal. HYPERLINK "http://www.ddj.com/184404744" http://www.ddj.com/184404744
21. Spolsky, J. (2002) The Law of Leaky Abstractions. HYPERLINK "http://www.joelonsoftware.com/articles/LeakyAbstractions.html" http://www.joelonsoftware.com/articles/LeakyAbstractions.html
22. Kroening, M. (2005) Eclipse: Adapting and Updating an IDE. Dr.Dobb's Journal. HYPERLINK "http://www.ddj.com/184407751" http://www.ddj.com/184407751
23. Wu, H., J. Cray, and M. Mernik. Grammar-Driven Generation of Domain-Specific Language Testing Tools. in OOPSLA. 2005. San Diego HYPERLINK "http://www.cis.uab.edu/gray/Pubs/ddf.pdf" http://www.cis.uab.edu/gray/Pubs/ddf.pdf
24. Eclipse 3.1, documentation of org.eclipse.debug.core.model.IDropToFrame interface (Javadoc). HYPERLINK "http://help.eclipse.org/help31/index.jsp?topic=/org.eclipse.platform.doc.isv/reference/api/org/eclipse/debug/core/model/IDropToFrame.html" http://help.eclipse.org/help31/index.jsp?topic=/org.eclipse.platform.doc.isv/reference/api/org/eclipse/debug/core/model/IDropToFrame.html
25. JVM Tool Interface HYPERLINK "http://java.sun.com/j2se/1.5.0/docs/guide/jvmti/jvmti.html" http://java.sun.com/j2se/1.5.0/docs/guide/jvmti/jvmti.html
26. Java Platform Debugger Architecture FAQ. HYPERLINK "http://java.sun.com/products/jpda/faq.html" http://java.sun.com/products/jpda/faq.html
27. Lichteblau, D. CL-JDI (a Common Lisp implementation of Java Debug Protocol). HYPERLINK "http://www.lichteblau.com/cloak/cl-jdi/README.html" http://www.lichteblau.com/cloak/cl-jdi/README.html
28. Kilmer, R., Ruby and the Java Debug Wire Protocol: What were we thinking?, in RubyConf 2003. 2003.
29. Fowler, C. and R. Kilmer. Ruby implementation of the Java Debug Wire Protocol. HYPERLINK "http://rubyforge.org/projects/rubyjdwp/" http://rubyforge.org/projects/rubyjdwp/
30. GNU Classpath. HYPERLINK "http://www.gnu.org/software/classpath/" http://www.gnu.org/software/classpath/
31. Roberts, N., B. Rossi, and E. Zaretskii. Debugger Machine Interface. HYPERLINK "http://www.freestandards.org/en/DMI" http://www.freestandards.org/en/DMI
32. Wright, D. and B. Freeman-Benson (2004) How to write an Eclipse debugger. HYPERLINK "http://www.eclipse.org/articles/Article-Debugger/how-to.html" http://www.eclipse.org/articles/Article-Debugger/how-to.html
33. Kroening, M. Amzi! Prolog: Source Code Debugger. HYPERLINK "http://www.amzi.com/manuals/amzi7/pro/pug_debugger_ide.htm" http://www.amzi.com/manuals/amzi7/pro/pug_debugger_ide.htm
34. Winchester, J. and A. Ryman (2001) The Pragmatics Of Java Debugging. Sys-Con India. HYPERLINK "http://in.sys-con.com/read/36223.htm" http://in.sys-con.com/read/36223.htm
While Javadoc is available, this document can complement that.
It is language-independent (or, rather, language-agnostic) in terms of the debuggees it can handle. Of course, being a Java API, it is most suited for use with debuggers written in Java (such as Eclipse). For simplicity of implementation, this will suffice, as I was interested in creating a proof of concept rapidly. It is with this in mind that this chapter should be understood. A truly independent implementation, agnostic both to the debugger and the debuggee, is discussed in REF _Ref142667068 \h \* MERGEFORMAT Appendix E.
All JDI interfaces are located in com.sun.jdi package or its subpackages. For more information, see HYPERLINK "http://download.java.net/jdk6/docs/jdk/api/jpda/jdi/com/sun/jdi/package-summary.html" http://download.java.net/jdk6/docs/jdk/api/jpda/jdi/com/sun/jdi/package-summary.html.
This is indeed how Eclipse Debug Framework does it. Their tutorial features a rudimentary implementation (in Perl) of a debugger for a made-up assembly language which exposes a TCP interface; the existing Eclipse C debugger is driving GDB behind the scenes ADDIN EN.CITE Leszek333343Pawel LeszekC/C++ development with the Eclipse Platform: How to use the C/C++ Development Toolkit (CDT)IBM developerWorksIBM developerWorkshttp://www-128.ibm.com/developerworks/opensource/library/os-ecc/13. Leszek, P. C/C++ development with the Eclipse Platform: How to use the C/C++ Development Toolkit (CDT). IBM developerWorks. http://www-128.ibm.com/developerworks/opensource/library/os-ecc/. As we show here, and in REF _Ref142667068 \h \* MERGEFORMAT Appendix E. that the JDI approach is equivalent to it. This is also discussed further in REF _Ref142938011 \h \* MERGEFORMAT Use cases.
One could make a case that data types are indeed those that Java uses. However, these can be used with other languages, especially since an Object data type can really simulate anything else say, Cs struct. (I will gloss over the intricacies of C structures such as pointer/array model, or unions, as this is outside the scope).
Further discussed in REF _Ref142930781 \h \* MERGEFORMAT Appendix E. Why not JDI?
All names of Dbdb classes mentioned here are in org.hrum.dbdb package, unless a fully qualified name is explicitly specified
See issue #1514829 at HYPERLINK "http://sourceforge.net/tracker/index.php?func=detail&aid=1514829&group_id=150446&atid=777815" http://sourceforge.net/tracker/index.php?func=detail&aid=1514829&group_id=150446&atid=777815.
This is a bit crude assuming this thread was originally suspended by the STEP_INTO request of a debugger. But if it wasnt, no harm no foul.
Who cares if GoF and their flock differ with us on minute nomenclature?
See also HYPERLINK "http://fish37.livejournal.com/2820.html" http://fish37.livejournal.com/2820.html.
See item REF _Ref139560898 \r \p \h 2 in REF _Ref139648775 \h \* MERGEFORMAT Leaky abstractons below.
Building a framework: this should be automatically picked up by DbdbVirtualMachine from a properties file, for example, rather than requiring to modify DbdbVirtualMachine class.
Building a framework: There is currently only one taking 3 arguments, URL, username and password. But all the overloaded methods should be taken care of, its just a matter of typing.
Building a framework: Of course, a mechanism to do this based on some sort of property files, rather than modifying __java_sql_DriverManagerListener every time, is a step that should be done prior to that. The more generic __java_sql_DriverManagerListener should thus be extended to dynamically (again, based on property files or similar mechanism) use strategy design pattern for JDBC URL recognition, getting a VirtualMachine object from the debuggee connection, etc.
Building a framework: Perhaps this one is not even needed, with the right reworking of the framework.
Since a sample dbdb_debuggers.properties file is included with Dbdb, the developer-modified one must be earlier on the classpath than Dbdb.
Lots more on this on the projects blog: HYPERLINK "http://fish37.livejournal.com/tag/dbdb" http://fish37.livejournal.com/tag/dbdb.
Its nice to see this (very Shakespearean) comment in com.sun.tools.example.debug.bdi.JDIEventSource:
//### Gross foul hackery!
Or something. This is described in HYPERLINK "http://fish37.livejournal.com/2230.html" http://fish37.livejournal.com/2230.html, HYPERLINK "http://fish37.livejournal.com/2699.html" http://fish37.livejournal.com/2699.html and HYPERLINK "http://fish37.livejournal.com/2820.html" http://fish37.livejournal.com/2820.html.
More detail on this at HYPERLINK "http://fish37.livejournal.com/6285.html" http://fish37.livejournal.com/6285.html.
More detail on this at HYPERLINK "http://fish37.livejournal.com/3984.html" http://fish37.livejournal.com/3984.html.
Which, in turn, required another little workaround, described at HYPERLINK "http://fish37.livejournal.com/1619.html" http://fish37.livejournal.com/1619.html.
A to-do list, masquerading as a bug list, is at HYPERLINK "http://sourceforge.net/tracker/?group_id=150446&atid=777815" http://sourceforge.net/tracker/?group_id=150446&atid=777815.
The Object-Oriented part is incidental to this debugging architecture. Because of models such as StackFrame, and things such as MethodEntryEvent, JPDA is based on a procedural/imperative model.
Pun intended.
Dont forget to commit. Duh!
See Defect #1497650 ( HYPERLINK "http://sourceforge.net/tracker/index.php?func=detail&aid=1497650&group_id=150446&atid=777815" http://sourceforge.net/tracker/index.php?func=detail&aid=1497650&group_id=150446&atid=777815).
It can, though, be used for a special case of mixed-mode debugging; per the JPDA FAQ:
HYPERLINK "http://java.sun.com/products/jpda/" \l "IA4" Can JPDA be used to write a mixed mode (Java and C/C++) debugger?
Yes, however this is a case where you would probably need to use JVMDI (see HYPERLINK "http://java.sun.com/products/jpda/" \l "QA2" Which interface layer should I use?). We know of one product, not released but working quite well, that uses a combination of JVMDI and native debugging functionality to provide mixed mode debugging.
Including in this API, of course, the native C code library used to actually read and write JDWP bits
Considering only open-source variants, of course.
Figure SEQ Figure \* ARABIC 7. Dbdb as Eclipse plug-in.
: ; < O P m w 2 3 4 N pd\P\ j UmH nH umH nH u 0J OJ QJ mH nH u&j >*B*UmH nH ph u mH nH u0J mH nH uj 0J UmH nH u CJ OJ QJ j CJ OJ QJ UCJ OJ QJ CJ OJ QJ ^J aJ 0J 6CJ OJ QJ ] !j 6CJ OJ QJ U]6CJ OJ QJ ] j 6CJ OJ QJ U]CJ OJ QJ Q n o p q r s t u v w T W O J
!
!
2@ F[ J[ [ N O P Q R S T U V r s t u ѿљދѿwљiѿ jD UmH nH u &j >*B*UmH nH ph u jJ UmH nH u 0J OJ QJ mH nH u&j >*B*UmH nH ph u mH nH u0J mH nH u:PJ mH nH u j 0J UmH nH u mH nH u j UmH nH ujP UmH nH u) 5 6 7 Q R S T U V W X Y u v w x } ~ ¶¨¶֝¶u¶֝a¶&j >*B*UmH nH ph u j8 UmH nH u &j >*B*UmH nH ph u mH nH u6PJ ]mH nH uj> UmH nH u j UmH nH umH nH u 0J OJ QJ mH nH uj 0J UmH nH u &j >*B*UmH nH ph u 0J mH nH u & " # $ % - . / I J K L M N O P Q m n o p | } ~ ѿљѿwљi j& UmH nH u &j >*B*UmH nH ph u j, UmH nH u 0J OJ QJ mH nH u&j >*B*UmH nH ph u mH nH u0J mH nH u:PJ mH nH u j 0J UmH nH u j UmH nH uj2 UmH nH u mH nH u& ( ) * D E F G H I J K L h i 繱y繱k j
UmH nH u &j >*B*UmH nH ph u :PJ mH nH u j UmH nH u j UmH nH umH nH u 0J OJ QJ mH nH u&j >*B*UmH nH ph u mH nH u0J mH nH uj 0J UmH nH u 6PJ ]mH nH u %i j k y z {
˿˱˿ߦߞ˿v˿ߦߞb &j >*B*UmH nH ph u j UmH nH u &j >*B*UmH nH ph u mH nH u0J mH nH u6PJ ]mH nH uj UmH nH u j UmH nH umH nH u 0J OJ QJ mH nH uj 0J UmH nH u &j
>*B*UmH nH ph u"
/ 0 1 3 4 5 6 7 8 T U V W p q r Żųōtų`ų &j >*B*UmH nH ph u 6PJ ]mH nH uj UmH nH u 0J OJ QJ mH nH u&j
>*B*UmH nH ph u mH nH u0J mH nH uPJ aJ mH nH uj 0J UmH nH u j
UmH nH u j UmH nH umH nH u 0J 6]mH nH u $ 6 9
+ E F
\ j ( s
!
!
!
!
2
3
4
6
7
8
9
:
;
W
X
Y
Z
c
d
e
ѿѿޗѿѿuѿ j UmH nH u &js >*B*UmH nH ph u j UmH nH u &jy >*B*UmH nH ph u mH nH u0J mH nH uPJ aJ mH nH uj 0J UmH nH u mH nH u j UmH nH uj UmH nH u*
$ % & ( ) * + , - I J K L ϵ߫ץב߅wmץY &ja >*B*UmH nH ph u :PJ mH nH u j UmH nH u 0J OJ QJ mH nH u&jg >*B*UmH nH ph u mH nH uPJ aJ mH nH uj UmH nH u j UmH nH umH nH u 0J mH nH uj 0J UmH nH u &jm >*B*UmH nH ph u"L ] ^ _ y z { } ~ " # $ > ? @ B C ĹıĹıug j UmH nH u &jU >*B*UmH nH ph u j UmH nH u &j[ >*B*UmH nH ph u mH nH u0J mH nH u6PJ ]mH nH uj 0J UmH nH u j UmH nH u j UmH nH umH nH u 0J OJ QJ mH nH u (C D E F G c d e f z { |
zl j UmH nH u &jI >*B*UmH nH ph u 6PJ ]mH nH uj UmH nH u j UmH nH umH nH u 0J OJ QJ mH nH u&jO >*B*UmH nH ph u mH nH u0J mH nH u:PJ mH nH u j 0J UmH nH u&
#
$
%
?
@
A
C
D
E
F
G
H
d
e
f
g
q
r
s
˿˱˿ߦߞ˿v˿lߞX &j7 >*B*UmH nH ph u :PJ mH nH u j UmH nH u &j= >*B*UmH nH ph u mH nH u0J mH nH u6PJ ]mH nH uj UmH nH u j UmH nH umH nH u 0J OJ QJ mH nH uj 0J UmH nH u &jC >*B*UmH nH ph u"
9 : ; U V W Y Z [ \ ] ^ z { | } ĺIJIJIJkIJ &j+ >*B*UmH nH ph u 6PJ ]mH nH uj UmH nH u &j1 >*B*UmH nH ph u mH nH u0J mH nH u:PJ mH nH u j 0J UmH nH u j UmH nH u j UmH nH umH nH u 0J OJ QJ mH nH u $ ! = > ? @ q r s Ѿј|rѾ^ј &j >*B*UmH nH ph u :PJ mH nH u j UmH nH u 0J CJ OJ QJ mH nH u0J OJ QJ mH nH u&j% >*B*UmH nH ph u mH nH u0J mH nH u6PJ ]mH nH uj 0J UmH nH u mH nH u j UmH nH uj UmH nH u!
, - . / G H I c d e g h i j k l ѿљދѿwљiѿ j UmH nH u &j >*B*UmH nH ph u j UmH nH u 0J OJ QJ mH nH u&j >*B*UmH nH ph u mH nH u0J mH nH u:PJ mH nH u j 0J UmH nH u mH nH u j UmH nH uj UmH nH u) ! " # % & ' ( ) * F G H I P Q R ¶¨¶֝¶u¶֝a¶&j# >*B*UmH nH ph u j" UmH nH u &j" >*B*UmH nH ph u mH nH u6PJ ]mH nH uj! UmH nH u j UmH nH umH nH u 0J OJ QJ mH nH uj 0J UmH nH u &j
! >*B*UmH nH ph u 0J mH nH u &R l m n p q r s t u
ѿљѿwљi jr% UmH nH u &j$ >*B*UmH nH ph u jx$ UmH nH u 0J OJ QJ mH nH u&j# >*B*UmH nH ph u mH nH u0J mH nH uPJ aJ mH nH uj 0J UmH nH u j UmH nH uj~# UmH nH u mH nH u& 1 2 3 4 J K L f g h j k l m n o | . / 2 3 繱獂 } v q q q memZme j&