Fixes a repeated stop on the first line of a range-over-func body when
stepping through a program. The entry point breakpoint is changed to a
special breakpoint that, when hit:
- clears the current temporary breakpoints
- sets breakpoints like 'next' would, and
- resumes execution.
Set a breakpoint on the return address of the current function, if it's
a range-over-func body, and clear the stepping breakpoints for the
current function (except the entry one) when its hit.
Without this what can happen is the following:
1. the range-over-func body finishes and returns to the iterator
2. the iterator calls back into the range-over-func body
3. a stepping breakpoint that's inside the prologue gets hit
Updates #3733
Supports viewing local variables and evaluating expressions correctly
when range-over-func is used.
The same limitations that the previous commit on this line had still
apply (no inlining, wrong way to identify the range parent in some
cases).
Updates #3733
* rebasing on master to implement --followcalls
* in progress changes to enable --followcalls
* rebase to master: modified function to add children to funcs array
* modify main traversal loop
* added tests to check different scenarios
* added tests to check different scenarios
* added tests to check different scenarios
* add test to check for overlapping regular expression
* modified type of strings array as a return only
* changed depth to a simple integer instead of a global map
* avoid calling traverse on recursive calls
* Added tests for various call graphs to test trace followfuncs
* Added tests for various call graphs to test trace followfuncs
* Added tests for various call graphs to test trace followfuncs
* made auxillary changes for build to go through for new option follow-calls
* Add support to print depth of the function calls as well
* Added two sample output files for checking
* Bypass morestack_noctxt in output for verification testing
* Corrected newline error by adding newlines only if the line does not match morestack_noctxt
* Added more tests
* Cleanup
* Updated documentation
* fixed error message in fmt.Errorf
* Fixed result of Errorf not used error
* Addressing review comments to fix depth reporting and other issues
* dont invoke stacktrace if tracefollowcalls is enabled, compute depth from main regex root symbol than main.main
* Addressing a part of review comments
* Added changes to allow deferred functions to be picked up for tracing
* Fix issue to avoid printing stack for a simple trace option
* Moving most tests to integration2_test.go and keeping only one in dlv_test.go
* Moving most tests to integration2_test.go and keeping only one in dlv_test.go
* Adding panic-defer test case
* Moved rest of the tests to integration2_test.go
* addressing review comments: folding Functions and FunctionsDeep, reducing branches by using depth prefix, wrap using %w and other comments
* Optimize traversal and parts of printing trace point function and modify trace output layout
and adjust tests accordingly
* Resolved error occurring due to staticcheck
* Implemented traversal algorithm using breadth first search
* Addressing review comments on the breadth first search implementation and
other comments
* Inline filterRuntimeFuncs and remove duplicate initialization
Change 'step' command so that when stepping into a 'go statement' the
debugger will stop on the newly created goroutine, instead of just
stepping over the go statement.
Show the location expression that will be used to set a suspended
breakpoint in the breakpoints list.
Also change 'target' called without arguments to print a better error
message and 'target follow-exec' without the last argument to print the
state of follow-exec.
* logflags,proc: flag to log stacktrace execution
Add a log flag to write logs about what the stacktracer does.
* proc: read context from sigtrampgo, fixes TestCgoStacktrace2 on 1.21
Changes stacktrace code to read the signal context from the arguments
of sigtrampgo.
Also changes the automatic fatalthrow breakpoint for go 1.21.
In combination these two changes fix TestCgoStacktrace2 on Go 1.21 on
various platforms.
We used to parse the .gopclntab section but removed support in favor of
simply using DWARF debug information, due to lack of C symbols among
other reasons. This makes it impossible to debug stripped binaries,
which some distrubutions ship by default.
Add back in basic support for .gopclntab which survives if the binary
is stripped, allowing for rudimentary debugging such as basic
program navigation, tracing, etc...
When using Step on a function that has a dynamic CALL instruction we
set a Step breakpoint on the call.
When it is hit we determine the destination of the CALL by looking at
registers, set a breakpoint there and continue.
If the Step breakpoint is hit simultaneously with a normal breakpoint
our Step logic will take precedence and the normal breakpoint hit will
be hidden from the user.
Move the Step logic to a breaklet callback so that it does not
interfere with the decision to stop.
The compiler produces ABI compatibility wrappers for some functions.
We have changed the support for breakpoints to allow a single logical
breakpoint to correspond to multiple physical breakpoints, take
advantage of that to set breakpoints on both the ABI wrapper and the
real function.
Fixes#3296
Adds field to breakpoint struct to track how a breakpoint was
originally set, moves the logic for disabling and enabling a breakpoint
to proc.
This will allow creating suspended breakpoints that are automatically
enabled when a plugin is loaded. When follow exec mode is implemented
it will also be possible to automatically enable breakpoints (whether
or not they were suspended) on new child processes, as they are
spawned.
It also improves breakpoint restore after a restart, before this after
a restart breakpoints would be re-enabled using their file:line
position, for breakpoints set using a function name or a location
expression this could be the wrong location after a recompile.
Updates #1653
Updates #2551
Go 1.20 switched to uint64 to represent goroutine IDs, we can't
actually follow suit because we have allowed clients to use -1 to refer
to the currently selected goroutine, however we should at least switch
to int64 and also update the rtype check to accept the 1.20 type.
1. return an error when SetUProbe fails, while creating ebpf tracepoints
2. if no tracepoint can be set for 'dlv trace' exit early
3. never leave the traced program running
4. fix typo in description of --ebpf flag
Fixes#3006
Introduces a new TargetGroup abstraction that can be used to manage
multiple related targets.
No actual management of child processes is implemented here, this is
just a refactoring to make it possible to do that in the future.
Updates #2551
Adds a LogicalBreakpoint type to represent logical breakpoints
explicitly. Until now logical breakpoints were constructed implicitly
by grouping physical breakpoints together by their LogicalID.
Having logical breakpoints represented explicitly allows for a simpler
implementation of disabled breakpoints, as well as allowing a simple
implementation of delayed breakpoints (#1653, #2551) and in general of
breakpoints spanning multiple processes if we implement debugging
process trees (#2551).
Updates #1653
Updates #2551
Moves breakpoindIDCounter out of BreakpointsMap and into
service/debugger.Debuggger to simplify proc.Target's API and aid with
implementing fork follow mode, where we'll have to debug multiple
processes simultaneously.
This commit improves the handling of hardcoded breakpoints in Delve.
A hardcoded breakpoint is a breakpoint instruction hardcoded in the
text of the program, for example through runtime.Breakpoint.
1. hardcoded breakpoints are now indicated by setting the breakpoint
field on any thread stopped by a hardcoded breakpoint
2. if multiple hardcoded breakpoints are hit during a single stop all
will be notified to the user.
3. a debugger breakpoint with an unmet condition can't hide a hardcoded
breakpoint anymore.
* service/debugger: fix bug internal err with Restart on recorded target
If Restart is called after a Continue and Rewind on a recorded target
that has already terminated it will return an internal error.
* proc/gdbserial: allow rewind to work after process exit with rr
It is sometimes useful to set breakpoints and rewind a terminated
process when using rr, for example if interested in the last execution
of some function.
RR will not allow a backward continue after the process exit packet has
been sent, however rr will also generate a synthetic SIGKILL right
before process exit.
Treat this packet as a process exit and change some things so both
continuing backwards and setting breakpoints can be done, on recorded
targets, after process exit has been reported.
* proc,locspec: support setting breakpoints by func name on generic funcs
Changes proc.Function to parse function names correctly when they
contain instantiation lists and locspec to match generic functions.
* vendor: update golang.org/x/tools
The old version of golang.org/x/tools is incompatible with the new
iexport format.
When the function we are calling is an autogenerated stub (because, for
example, we are calling it through a function pointer) the declaration
line of variables is meaningless and could cause us to discard valid
return arguments.
Go 1.18 removed the jmpdefer call from deferreturn, now deferreturn is
a normal function call that can appear on the stack, rules for
detecting a deferreturn call must be changed and new code must be added
to skip it while stepping out.
Internal breakpoints do not need IDs and assigning them from a counter
separate from the user ID counter can be a cause of confusion.
If a user breakpoint is overlayed on top of a pre-existing internal
breakpoint the temporary ID will be surfaced as if it was a user ID,
possibly conflicting with another user ID.
If a temporary breakpoint is overlayed on top of a pre-existing user
breakpoint and the user breakpoint is first deleted and then
re-created, the user ID will be resurrected along with the breakpoint,
instead of allocating a fresh one.
This change removes internal breakpoint IDs entirely, only user
breakpoints receive an ID.
Log points are special kinds of breakpoints that do not 'break' but instead log a message and then continue. This change implements basic log points that simply log the provided message, without any interpolation.
In order to resume execution after hitting a breakpoint, I added a new lock resumeMu and tracked the running state within the DAP server. resumeMu must be held in order to issue a debugger request that would start execution. This means it can be used to make sure that another goroutine does not resume execution while you are holding the lock.
Most of the synchronization logic is taken from PR #2530
Updates golang/vscode-go#123
This patch enables the eBPF tracer backend to parse the ID of the
Goroutine which hit the uprobe. This implementation is specific to AMD64
and will have to be generalized further in order to be used on other
architectures.
* terminal,service: add way to see internal breakpoints
Now that Delve has internal breakpoints that survive for long periods
of time it will be useful to have an option to display them.
* proc,terminal,service: support stack watchpoints
Adds support for watchpoints on stack allocated variables.
When a stack variable is watched, in addition to the normal watchpoint
some support breakpoints are created:
- one breakpoint inside runtime.copystack, used to adjust the address
of the watchpoint when the stack is resized
- one or more breakpoints used to detect when the stack variable goes
out of scope, those are similar to the breakpoints set by StepOut.
Implements #279
* proc: move breakpoint condition evaluation out of backends
Moves breakpoint condition evaluation from the point where breakpoints
are set, inside ContinueOnce, to (*Target).Continue.
This accomplishes three things:
1. the breakpoint evaluation method needs not be exported anymore
2. breakpoint condition evaluation can be done with a full scope,
containing a Target object, something that wasn't possible before
because ContinueOnce doesn't have access to the Target object.
3. moves breakpoint condition evaluation out of the critical section
where some of the threads of the target process might be still
running.
* proc/native: handle process death during stop() on Windows
It is possible that the thread dies while we are inside the stop()
function. This results in an Access is denied error being returned by
SuspendThread being called on threads that no longer exist.
Delay the reporting the error from SuspendThread until the end of
stop() and only report it if the thread still exists at that point.
Fixes flakyness with TestIssue1101 that was exacerbated by moving
breakpoint condition evaluation outside of the backends.
Changes Breakpoint to allow multiple overlapping internal breakpoints
on the same instruction address.
This is done by changing the Breakpoint structure to contain a list of
"breaklets", each breaklet has a BreakpointKind and a condition
expression, independent of the other.
A breakpoint is considered active if any of its breaklets are active.
A breakpoint is removed when all its breaklets are removed.
We also change the terminology "internal breakpoint" to "stepping
breakpoint":
HasInternalBreakpoints -> HasSteppingBreakpoints
IsInternal -> IsStepping
etc...
The motivation for this change is implementing watchpoints on stack
variables.
Watching a stack variable requires also setting a special breakpoint to
find out when the variable goes out of scope. These breakpoints can not
be UserBreakpoints because only one user breakpoint is allowed on the
same instruction and they can not be internal breakpoints because they
should not be cleared when a next operation is completed (they should
be cleared when the variable watch is cleared).
Updates #279
* proc: support new Go 1.17 panic/defer mechanism
Go 1.17 will create wrappers for deferred calls that take arguments.
Change defer reading code so that wrappers are automatically unwrapped.
Also the deferred function is called directly by runtime.gopanic, without going through runtime.callN which means that sometimes when a panic happens the stack is either:
0. deferred function call
1. deferred call wrapper
2. runtime.gopanic
or:
0. deferred function call
1. runtime.gopanic
instead of always being:
0. deferred function call
1. runtime.callN
2. runtime.gopanic
the isPanicCall check is changed accordingly.
* test: miscellaneous minor test fixes for Go 1.17
* proc: resolve inlined calls when stepping out of runtime.breakpoint
Calls to runtime.Breakpoint are inlined in Go 1.17 when inlining is
enabled, resolve inlined calls in stepInstructionOut.
* proc: add support for debugCallV2 with regabi
This change adds support for the new debug call protocol which had to
change for the new register ABI introduced in Go 1.17.
Summary of changes:
- Abstracts over the debug call version depending on the Go version
found in the binary.
- Uses R12 instead of RAX as the debug protocol register when the binary
is from Go 1.17 or later.
- Creates a variable directly from the DWARF entry for function
arguments to support passing arguments however the ABI expects.
- Computes a very conservative stack frame size for the call when
injecting a call into a Go process whose version is >=1.17.
Co-authored-by: Michael Anthony Knyszek <mknyszek@google.com>
Co-authored-by: Alessandro Arzilli <alessandro.arzilli@gmail.com>
* TeamCity: enable tests on go-tip
* goversion: version compatibility bump
* TeamCity: fix go-tip builds on macOS/arm64
Co-authored-by: Michael Anthony Knyszek <mknyszek@google.com>
We told clients that further loading of variables can be done by
specifying a type cast using the address of a variable that we
returned.
This does not work for registerized variables (or, in general,
variables that have a complex location expression) because we don't
give them unique addresses and we throw away the compositeMemory object
we made to read them.
This commit changes proc so that:
1. variables with location expression divided in pieces do get a unique
memory address
2. the compositeMemory object is saved somewhere
3. when an integer is cast back into a pointer type we look through our
saved compositeMemory objects to see if there is one that covers the
specified address and use it.
The unique memory addresses we generate have the MSB set to 1, as
specified by the Intel 86x64 manual addresses in this form are reserved
for kernel memory (which we can not read anyway) so we are guaranteed
to never generate a fake memory address that overlaps a real memory
address of the application.
The unfortunate side effect of this is that it will break clients that
do not deserialize the address to a 64bit integer. This practice is
contrary to how we defined our types and contrary to the specification
of the JSON format, as of json.org, however it is also fairly common,
due to javascript itself having only 53bit integers.
We could come up with a new mechanism but then even more old clients
would have to be changed.
* pkg/proc: implement support for hit count breakpoints
* update comment
* udpate hitcount comment
* update HitCond description
* add test for hit condition error
* respond to review
* service/dap: add support for hit count breakpoints
* use amendbps to preserve hit counts
* update test health doc
* fix failing test
* simplify hit conditions
* REmove RequestString, use name instead
* update backend_test_health.md
* document hit count cond
* fix tests
Adds the low-level support for watchpoints (aka data breakpoints) to
the native linux/amd64 backend.
Does not add user interface or functioning support for watchpoints
on stack variables.
Updates #279
* Adds toggle command
Also adds two rpc2 tests for testing the new functionality
* Removes Debuggers' ToggleBreakpoint method
rpc2's ToggleBreakpoint now calls AmendBreakpoint
Refactors the ClearBreakpoint to avoid a lock.
On linux we can not read memory if the thread we use to do it is
occupied doing certain system calls. The exact conditions when this
happens have never been clear.
This problem was worked around by using the Blocked method which
recognized the most common circumstances where this would happen.
However this is a hack: Blocked returning true doesn't mean that the
problem will manifest and Blocked returning false doesn't necessarily
mean the problem will not manifest. A side effect of this is issue
#2151 where sometimes we can't read the memory of a thread and find its
associated goroutine.
This commit fixes this problem by always reading memory using a thread
we know to be good for this, specifically the one returned by
ContinueOnce. In particular the changes are as follows:
1. Remove (ProcessInternal).CurrentThread and
(ProcessInternal).SetCurrentThread, the "current thread" becomes a
field of Target, CurrentThread becomes a (*Target) method and
(*Target).SwitchThread basically just sets a field Target.
2. The backends keep track of their own internal idea of what the
current thread is, to use it to read memory, this is the thread they
return from ContinueOnce as trapthread
3. The current thread in the backend and the current thread in Target
only ever get synchronized in two places: when the backend creates a
Target object the currentThread field of Target is initialized with the
backend's current thread and when (*Target).Restart gets called (when a
recording is rewound the currentThread used by Target might not exist
anymore).
4. We remove the MemoryReadWriter interface embedded in Thread and
instead add a Memory method to Process that returns a MemoryReadWriter.
The backends will return something here that will read memory using
the current thread saved by the backend.
5. The Thread.Blocked method is removed
One possible problem with this change is processes that have threads
with different memory maps. As far as I can determine this could happen
on old versions of linux but this option was removed in linux 2.5.
Fixes#2151
An internal breakpoint condition shouldn't ever error:
* use a ThreadContext to evaluate conditions if a goroutine isn't
available
* evaluate runtime.curg to a fake g variable containing only
`goid == 0` when there is no current goroutine
Fixes#2113
* proc: move defer breakpoint code into a function
Moves the code that sets a breakpoint on the first deferred function,
used by both next and StepOut, to its function.
* proc: implement reverse step/next/stepout
When the direction of execution is reversed (on a recording) Step, Next and
StepOut will behave similarly to their forward version. However there are
some subtle interactions between their behavior, prologue skipping, deferred
calls and normal calls. Specifically:
- when stepping backwards we need to set a breakpoint on the first
instruction after each CALL instruction, once this breakpoint is reached we
need to execute a single StepInstruction operation to reverse step into the
CALL.
- to insure that the prologue is skipped reverse next needs to check if it
is on the first instruction after the prologue, and if it is behave like
reverse stepout.
- there is no reason to set breakpoints on deferred calls when reverse
nexting or reverse stepping out, they will never be hit.
- reverse step out should generally place its breakpoint on the CALL
instruction that created the current stack frame (which will be the CALL
instruction immediately preceding the instruction at the return address).
- reverse step out needs to treat panic calls and deferreturn calls
specially.
* service,terminal: implement reverse step, next, stepout
Modifies FindFileLocation, FindFunctionLocation and LineToPC as well as
service/debugger to support inlining and introduces the concept of
logical breakpoints.
For inlined functions FindFileLocation, FindFunctionLocation and
LineToPC will now return one PC address for each inlining and one PC
for the concrete implementation of the function (if present).
A proc.Breakpoint will continue to represent a physical breakpoint, at
a single memory location.
Breakpoints returned by service/debugger, however, will represent
logical breakpoints and may be associated with multiple memory
locations and, therefore, multiple proc.Breakpoints.
The necessary logic is introduced in service/debugger so that a change
to a logical breakpoint will be mirrored to all its physical
breakpoints and physical breakpoints are aggregated into a single
logical breakpoint when returned.
If proc.Step encounters a CALL instruction that points to an address
that isn't associated with any function it should still follow the
CALL.
The circumstances creating this problem do not normally occur, it was
encountered in the process of fixing a bug created by Go1.12.
