There are persistent issues with watchpoints on Windows, it is not
clear whether it's a problem with the VM running the tests or if there
is a real bug in our implementation of hardware breakpoints on Windows.
Until the cause can be determined watchpoints on Windows will be
disabled.
Updates #2768
We want to provide more flexibility for users to make changes to their configurations while the debug session is running. This could be accomplished by creating a custom request, but that were require a new UI as well, and every client of dlv dap to provide its own UI for this. By using the evaluate context, users can use the already existing debug console to change their configurations.
This change includes a refactor of the terminal code in order to share the code with the dap package.
This change provides a very similar to UI as the terminal package, but there are different configuration options that are DAP specific. We plan to use this same mechanism to expose a few other commands including "sources" to help users debug an ineffective substitutePath configuration.
* service/dap: remove deadlock in TestLaunchDebugRequest
Fixes#2746
* terminal: fix TestScopePrefix flakiness
When there are more than 10 frames api.PrintStack will prefix the
output with spaces to right justify the frame number, which confuses
TestScopePrefix.
* _scripts: pass -buildvcs for Go 1.18 and later on TeamCity
Go 1.18 will try to stamp builds with the VCS version, this doesn't
work on TeamCity because the checkout isn't a valid repository (but
looks like it).
Pass -buildvcs=false to disable this feature.
* proc: switch to goroutine stack if first frame's func can not be found
If the first frame on the system stack can not be resolved to a
function switch directly to the goroutine stack.
Normally calls can't be performed on recorded processes, becuase the
future instructions executed by the target are predetermined. The rr
debugger however has a mechanism that allows this by taking the current
state of the recording and allowing it to diverge from the recording,
temporarily.
This commit adds support for starting and ending such diversions around
function calls.
Note: this requires rr version 5.5 of later to work, see:
https://github.com/rr-debugger/rr/pull/2748
LoadConfig warnings should obey the logflags configuration and should
also be delayed until after the "listening at" message, which should
always be the first thing output.
Co-authored-by: Suzy Mueller <suzmue@golang.org>
* proc/native: always stop after RequestManualStop on Windows
On Windows RequestManualStop will generate an exception on a special
DbgUiRemoteBreakin thread, sometimes this thread will die before we
finish stopping the process. We need to account for that and still stop
even if the thread is gone and no other thread hit a breakpoint.
Fixes flakiness of TestIssue419.
* proc/native: fix watchpoints with new threads on Windows
When a new thread is created we must reapply all watchpoints to it,
like we do on linux.
* tests: be lenient on goroutinestackprog tests on Windows
We can not guarantee that we find all goroutines stopped in a good
place and sometimes the stacktrace fails on Windows.
Adds watchpoint support to gdbserver backend for rr debugger and
debugserver on macOS/amd64 and macOS/arm64.
Also changes stack watchpoints to support reverse execution.
Change debug_info type reader and proc to convert parametric types into
their real types by reading the corresponding dictionary entry and
using the same method used for interfaces to retrieve the DIE from a
runtime._type address.
'2586e9b1'.
With generics a single function can have multiple concrete
instantiations, the old version of FindFileLocation supported at most
one concrete instantiation per function and any number of inlined
calls, this supports any number of inlined calls and concrete
functions.
When the function we are calling is an autogenerated stub (because, for
example, we are calling it through a function pointer) the declaration
line of variables is meaningless and could cause us to discard valid
return arguments.
Go 1.18 removed the jmpdefer call from deferreturn, now deferreturn is
a normal function call that can appear on the stack, rules for
detecting a deferreturn call must be changed and new code must be added
to skip it while stepping out.
Internal breakpoints do not need IDs and assigning them from a counter
separate from the user ID counter can be a cause of confusion.
If a user breakpoint is overlayed on top of a pre-existing internal
breakpoint the temporary ID will be surfaced as if it was a user ID,
possibly conflicting with another user ID.
If a temporary breakpoint is overlayed on top of a pre-existing user
breakpoint and the user breakpoint is first deleted and then
re-created, the user ID will be resurrected along with the breakpoint,
instead of allocating a fresh one.
This change removes internal breakpoint IDs entirely, only user
breakpoints receive an ID.
Right now, if (*compositeMemory).WriteMemory needs to write a value to
a register that's smaller than the full size of the register (say, a
uint32 being passed as an argument), then (*AMD64Registers).SetReg can
later fail a sanity check that ensures the passed DwarfRegister is a
full size register.
Fix this by reading the old value of the register and overwriting just
the relevant parts with the new register. For the purposes of an
argument, it would probably be fine to just pad with zeroes, but merging
with the existing value is what gdb does.
Fixes#2698
Log points are special kinds of breakpoints that do not 'break' but instead log a message and then continue. This change implements basic log points that simply log the provided message, without any interpolation.
In order to resume execution after hitting a breakpoint, I added a new lock resumeMu and tracked the running state within the DAP server. resumeMu must be held in order to issue a debugger request that would start execution. This means it can be used to make sure that another goroutine does not resume execution while you are holding the lock.
Most of the synchronization logic is taken from PR #2530
Updates golang/vscode-go#123
This changes allow us to parse Go+BoringCrypto which formatted in <GoVersion>b<BoringCryptoVersion> so that we can surpress `Version of Go is too old for this version of Delve` error.
Fixes#2711
* pkg/proc: update check for system goroutine
The finalizer goroutine can be either a system goroutine or a user goroutine. It is considered a user goroutine only when it calls back to user code. This change attempts to get closer to the implementation in the src/runtime/traceback.go by checking the value of fingRunning.
We could alternatively adopt the approximation done by src/cmd/trace/trace.go which only special cases "runtime.main", and always considers the finalizer to be a system goroutine.
The string variant is more powerful and can be passed
directly to "echo -e" / "printf" and friends.
Remove the mention of the integer variant from the default config. This
makes the option appear more consistent with other color options.
The user shouldn't care about the historic differences.
Commit 3d6bbbe9 made "config -list" print strings in quotes to avoid
the strings being interpreted as terminal escape codes. This commit
does the same for "source-list-line-color", which is a raw interface.
This fixes "config -list" with a config like
source-list-line-color: "\x1b[34m"
The "int" variant is already printed correctly, so just use the
default case for that.
If we cannot find debug information locally on the system, try and
invoke debuginfod-find in order to ask the linux distribution we're on
to find the debug information for us.
This patch enables the eBPF tracer backend to parse the ID of the
Goroutine which hit the uprobe. This implementation is specific to AMD64
and will have to be generalized further in order to be used on other
architectures.
Adds a script that check automatically that the the assumptions that
pkg/proc makes about runtime are met by the actual runtime, using a
combination of parsing and magic comments.
Also emits a file describing all the struct fields, constants and
variables of the runtime that we use in pkg/proc.
Commit 8e91d3b0b added a number of configuration options to control the
colors of sytnax highlighting, unfortunately 'config -list' will print
all of those to stdout without quoting them, resulting in the color of
the last one being applied to all subsequent text.
Change 'config -list' to print strings in quotes so that we don't
accidentally send escape sequences to the terminal.
In order for DAP to support halting the program (either manually or on a breakpoint) performing some action and then resuming execution, there needs to be a way to stop the program without clearing the internal breakpoints. This is necessary for log points and stopping the program to set breakpoints.
The debugging UI makes it seem like a user should be able to set or clear a breakpoint at any time. Adding this ability to complete synchronous requests while the program is running is thus important to create a seamless user experience.
This change just adds a configuration to determine whether the target should clear the stepping breakpoints, and changes the server to use this new mode. Using the new mode means that the DAP server must determine when it expect the next to be canceled and do this manually.
* terminal,service: add way to see internal breakpoints
Now that Delve has internal breakpoints that survive for long periods
of time it will be useful to have an option to display them.
* proc,terminal,service: support stack watchpoints
Adds support for watchpoints on stack allocated variables.
When a stack variable is watched, in addition to the normal watchpoint
some support breakpoints are created:
- one breakpoint inside runtime.copystack, used to adjust the address
of the watchpoint when the stack is resized
- one or more breakpoints used to detect when the stack variable goes
out of scope, those are similar to the breakpoints set by StepOut.
Implements #279
* proc: move breakpoint condition evaluation out of backends
Moves breakpoint condition evaluation from the point where breakpoints
are set, inside ContinueOnce, to (*Target).Continue.
This accomplishes three things:
1. the breakpoint evaluation method needs not be exported anymore
2. breakpoint condition evaluation can be done with a full scope,
containing a Target object, something that wasn't possible before
because ContinueOnce doesn't have access to the Target object.
3. moves breakpoint condition evaluation out of the critical section
where some of the threads of the target process might be still
running.
* proc/native: handle process death during stop() on Windows
It is possible that the thread dies while we are inside the stop()
function. This results in an Access is denied error being returned by
SuspendThread being called on threads that no longer exist.
Delay the reporting the error from SuspendThread until the end of
stop() and only report it if the thread still exists at that point.
Fixes flakyness with TestIssue1101 that was exacerbated by moving
breakpoint condition evaluation outside of the backends.
This patch implements fuzzy searching for tab completions in the
terminal client. Under the hood it is using a trie data structure
(https://en.wikipedia.org/wiki/Trie) to perform very fast prefix / fuzzy
searches.
* proc: Go 1.18 removes the _defer.siz field
As of Go 1.17, the _defer.siz field is always 0 because _defer no
longer stores defer call arguments at all. golang.org/cl/326062
removes it entirely for Go 1.18. Simply treat it as 0 if the field is
missing from the _defer type.
* proc: Go 1.18 changes _defer.fn from *funcval to func()
golang.org/cl/325918 changed the type of the _defer.fn field from
*funcval to func() for Go 1.18. This CL was later reverted because it
caused failures in Delve, but we would like to un-revert it. Handle
this change by inspecting the type of this field before decoding it.
Ignore existing breakpoints when using the continue command to continue
to a specific location such as `continue main.main`. The point of this
command is to continue to a specific location, so if there is already a
breakpoint set there there should be no error returned, just continue
until we hit the breakpoint already set in that location.
Before this change when you typed `help` at the Delve prompt you would
only see the following:
```
examinemem (alias: x) Examine memory:
```
Now with this patch the output is more descriptive:
```
examinemem (alias: x) Examine raw memory at the given address.
```
* pkg/proc: Prefer throw instead of fatalthrow
Currently there is a breakpoint set at runtime.fatalthrow to catch any
situation where the runtime crashes (e.g. deadlock).
When we do this, we go up a frame in order to parse the crash reason.
The problem is that there is no guarentee the "s" variable we attempt to
parse will still be considered "live".
Since runtime.fatalthrow is never called directly, set a breakpoint on
runtime.throw instead and prevent having
to search up a stack frame in order to
get the throw reason.
Fixes#2602
* service/dap: Fix TestFatalThrowBreakpoint
* Reenable TestFatalThrow DAP test
* service/dap: Don't skip test on < 1.17
* service/dap: Update test constraint for 1.16
* pkg/proc: Reinstate runtime.fatalthrow as switchstack exception
* cmd/dlv: dlv version --verbose
That prints out runtime/debug.BuildInfo read from the dlv binary.
Users can retrieve the same info using `go version -m <path_to_dlv>`
but I think it is convenient to have.
If dlv was built from cloned delve repo:
```
$ ./dlv version -v
Delve Debugger
Version: 1.7.0
Build: $Id: e353a65161e6ed74952b96bbb62ebfc56090832b $
Build Details: go1.16.5
mod github.com/go-delve/delve (devel)
dep github.com/cosiner/argv v0.1.0 h1:BVDiEL32lwHukgJKP87btEPenzrrHUjajs/8yzaqcXg=
...
```
If dlv was built with `go install github.com/go-delve/delve@latest`
with go1.16+, or
`GO111MODULE=on go get github.com/go-delve/delve@latest`
from a clean main module:
```
$ ./dlv version -v
Delve Debugger
Version: 1.7.0
Build: $Id: e353a65161e6ed74952b96bbb62ebfc56090832b $
Build Details: go1.16.5
mod github.com/go-delve/delve v1.7.0
dep github.com/cosiner/argv v0.1.0 h1:BVDiEL32lwHukgJKP87btEPenzrrHUjajs/8yzaqcXg=
...
```
* remove an accidentally added bogus test
* terminal: add prompt when breakpoint is hit during next/step/stepout
Adds a prompt asking the user what to do when a breakpoint is hit on a
different goroutine while next/step/stepout is being executed.
Gives the user the opportunity to cancel next/step/stepout, continue
once skipping the breakpoint or automatically skipping all other
concurrent breakpoints until next/step/stepout is finished.
Fixes#2317
StaticBase is the difference between the entry point declared in the
image file and the entry point as loaded in memory, since this
difference could be a negative number we have to use a signed
comparison when searching for a mapping.
Causes intermittent test failures on windows when resolving interface
types for position independent executables.
Fixes#2620
Conversion form a moduledata object into an image object was
implemented by looking for a function covering the start address of the
text section of the moduledata object, and then converting that into
its corresponding image.
Unfortunately this seems to not always work. In particular it does not
work on linux/386 with go1.17 (but it might also fail on other
combinations): the start address of the text section is, for whatever
reason, not part of any function.
As a fallback simply scan all images we know of and return the closest
one that has start address less than or equal to the start address of
the text section we are looking for.
Fixes TestPluginVariables on go1.17/linux/386.
Fixes#2611
Co-authored-by: a <a@kra>
* terminal: improve 'on' command
Adds the ability to edit the list of commands executed after stopping
on a breakpoint, as well as converting a breakpoint into a tracepoint
and vice versa.
Prior to this it was possible to add commands to a breakpoint but
removing commands or changing a breakpoint into a tracepoint, or vice
versa, could only be done by removing and recreating the breakpoint.
Changes Breakpoint to allow multiple overlapping internal breakpoints
on the same instruction address.
This is done by changing the Breakpoint structure to contain a list of
"breaklets", each breaklet has a BreakpointKind and a condition
expression, independent of the other.
A breakpoint is considered active if any of its breaklets are active.
A breakpoint is removed when all its breaklets are removed.
We also change the terminology "internal breakpoint" to "stepping
breakpoint":
HasInternalBreakpoints -> HasSteppingBreakpoints
IsInternal -> IsStepping
etc...
The motivation for this change is implementing watchpoints on stack
variables.
Watching a stack variable requires also setting a special breakpoint to
find out when the variable goes out of scope. These breakpoints can not
be UserBreakpoints because only one user breakpoint is allowed on the
same instruction and they can not be internal breakpoints because they
should not be cleared when a next operation is completed (they should
be cleared when the variable watch is cleared).
Updates #279
This PR aims to add support for rr replay and core actions from the DAP layer. This basically encloses the following:
New launch modes: replay and core
The following modes are added:
replay: Replays an rr trace, allowing backwards flows (reverse continue and stepback). Requires a traceDirPath property on launch.json pointing to a valid rr trace directory.
Equivalent to dlv replay <tracedir> command.
core: Replays a core dump file, showing its callstack and the file matching the callsite. Requires a coreFilePath property on launch.json pointing to a valid coredump file.
Equivalent to dlv core <exe> <corefile> command.
Dependencies
To achieve this the following additional changes were made:
Implement the onStepBackRequest and onReverseContinueRequest methods on service/dap
Adapt onLaunchRequest with the requried validations and logic for these new modes
Use CapabilitiesEvent responses to enable the StepBack controls on the supported scenarios (see dicussion here)
Add the corresponding launch.json support on vs code:
Support for replay and core modes golang/vscode-go#1268
* service/dap: add panic and throw text to stopped event
We can add more information to the stopped events on errors using
the `Text` field in the stopped event. We already use this to display
the runtime errors. Adding this information to the stopped reason will
also help to show the user additional info when a stopped event is not
associated with a particular goroutine.
* proc: support new Go 1.17 panic/defer mechanism
Go 1.17 will create wrappers for deferred calls that take arguments.
Change defer reading code so that wrappers are automatically unwrapped.
Also the deferred function is called directly by runtime.gopanic, without going through runtime.callN which means that sometimes when a panic happens the stack is either:
0. deferred function call
1. deferred call wrapper
2. runtime.gopanic
or:
0. deferred function call
1. runtime.gopanic
instead of always being:
0. deferred function call
1. runtime.callN
2. runtime.gopanic
the isPanicCall check is changed accordingly.
* test: miscellaneous minor test fixes for Go 1.17
* proc: resolve inlined calls when stepping out of runtime.breakpoint
Calls to runtime.Breakpoint are inlined in Go 1.17 when inlining is
enabled, resolve inlined calls in stepInstructionOut.
* proc: add support for debugCallV2 with regabi
This change adds support for the new debug call protocol which had to
change for the new register ABI introduced in Go 1.17.
Summary of changes:
- Abstracts over the debug call version depending on the Go version
found in the binary.
- Uses R12 instead of RAX as the debug protocol register when the binary
is from Go 1.17 or later.
- Creates a variable directly from the DWARF entry for function
arguments to support passing arguments however the ABI expects.
- Computes a very conservative stack frame size for the call when
injecting a call into a Go process whose version is >=1.17.
Co-authored-by: Michael Anthony Knyszek <mknyszek@google.com>
Co-authored-by: Alessandro Arzilli <alessandro.arzilli@gmail.com>
* TeamCity: enable tests on go-tip
* goversion: version compatibility bump
* TeamCity: fix go-tip builds on macOS/arm64
Co-authored-by: Michael Anthony Knyszek <mknyszek@google.com>
ConvertEvalScope() attempts to find the scope for the specified
goroutine id and frame index. If the goroutine that is found is nil,
then it falls back to the threads stack trace to find the scope.
This fix makes sure that the frame id is taken into account for
thread strack traces as well.
We told clients that further loading of variables can be done by
specifying a type cast using the address of a variable that we
returned.
This does not work for registerized variables (or, in general,
variables that have a complex location expression) because we don't
give them unique addresses and we throw away the compositeMemory object
we made to read them.
This commit changes proc so that:
1. variables with location expression divided in pieces do get a unique
memory address
2. the compositeMemory object is saved somewhere
3. when an integer is cast back into a pointer type we look through our
saved compositeMemory objects to see if there is one that covers the
specified address and use it.
The unique memory addresses we generate have the MSB set to 1, as
specified by the Intel 86x64 manual addresses in this form are reserved
for kernel memory (which we can not read anyway) so we are guaranteed
to never generate a fake memory address that overlaps a real memory
address of the application.
The unfortunate side effect of this is that it will break clients that
do not deserialize the address to a 64bit integer. This practice is
contrary to how we defined our types and contrary to the specification
of the JSON format, as of json.org, however it is also fairly common,
due to javascript itself having only 53bit integers.
We could come up with a new mechanism but then even more old clients
would have to be changed.
Adds filtering and grouping to the goroutines command.
The current implementation of the goroutines command is modeled after
the threads command of gdb. It works well for programs that have up to
a couple dozen goroutines but becomes unusable quickly after that.
This commit adds the ability to filter and group goroutines by several
different properties, allowing a better debugging experience on
programs that have hundreds or thousands of goroutines.
While Go still mostly uses DWARF v4, newer versions of GCC will emit
DWARF v5 by default. This patch improves support for DWARF v5 by parsing
the .debug_line_str section and using that during file:line lookups.
This patch only includes support for files, not directories.
Co-authored-by: Derek Parker <deparker@redhat.com>
Ensure that any command executed after the process we are trying to
debug prints a correct and consistent exit status.
Previously the exit code was being lost after the first time we printed
that a process has exited. Additionally, certain commands would print
the PID of the process and other would not. This change makes everything
more correct and consistent.
Commit 30cdedae6910f5e9af6739845bacfd5b8778e745 introduced a dependency
from service/dap to pkg/terminal to call a stack printing function,
it's weird to have code that implements the DAP protocol depend on the
code for the JSON-RPC client.
Move PrintStack to a different package that can be called by both.
Apply a presentation hint to the internal runtime stack frames, so that these can be deemphasized in the UI. This should allow
users to more easily inspect their own code, and will keep the option to view those frames if they choose to.
If the client supports paging, we allow them to fetch array and slice items in chunks that we assume will be of a reasonable size. For example, VS Code requests indexed variables in chunks of 100.
Fixesgolang/vscode-go#1518
If the base address isn't set then indexing and slicing will not work.
Large floating point registers already had the base set but small
general purpose registers did not.
* pkg/proc: implement support for hit count breakpoints
* update comment
* udpate hitcount comment
* update HitCond description
* add test for hit condition error
* respond to review
* service/dap: add support for hit count breakpoints
* use amendbps to preserve hit counts
* update test health doc
* fix failing test
* simplify hit conditions
* REmove RequestString, use name instead
* update backend_test_health.md
* document hit count cond
* fix tests
Adds DWARF register number and support for AVX-512 registers.
Changes proc/gdbserial so that the 'g' and 'G' commands are never used
with debugserver since they seem to corrupt the thread state when used
on AVX-512 capable hardware.
Also changes TestClientServer_FpRegisters to be simpler and more
resilient to changes to the Go runtime.
Fixes#2479
The code populating runtimeTypeToDIE was incorrectly adding StaticBase
to the offset. We never noticed because on statically compiled
executables StaticBase is always zero and on PIE and plugins the
fallback code took care of the problem anyway.
A change in Go 1.17 broke the fallback code, making the issue apparent.
This commit fixes the setup of runtimeTypeToDIE and disables the
fallback code for Go 1.17 and later.
This change also fixes a rare failure in TestPluginVariables when PIE
is enabled.
Truncates the result of binary operations on integers to the size of
the resulting type.
Also rewrites convertInt to not require allocations.
Fixes#2454
The maintainer of debugserver says he wants to fix the problem with the
'g' command but doesn't know when it will happen. Treat the error 'E74'
for the 'g' command on debugserver as if the server had returned an
unsupported error so that, for this specific problem, the error doesn't
resurface in the future.
A RequestManualStop received while the target program is stopped can
induce a crash when the target is restarted.
This is caused by the phantom breakpoint detection that was introduced
in PR #2179 / commit e69d536.
Instead of always interpreting an unexplained SIGTRAP as a phantom
breakpoint memorize all possible unreported breakpoint hits and only
act on it when the thread hasn't moved from one.
Also clarifies the behavior of the halt command when it is received
while the target is stopped or in the process of stopping.
Handle the signaled status for the thread leader like we handle the
exited status, by returning ErrProcessExited and recording the killer
signal in it.
Prior to this commit we would find out about the death of the thread
later in the loop, the condition would still be reported as
ErrProcessExited, but without recording the signal number anywhere.
Also fixes a bug in TestAttachStopOnEntry where the test would
inadvertently cause a SIGPIPE to be sent to the target process, making
it terminate early.
* service/dap: implement exception info
* remove adding additional thread
* Fix tests
* add exceptionInfo tests
* update comments
* map paths to client paths
* remove launch.json
* remove change to ConvertEvalScope
* correct name of supportsExceptionInfoRequest
* Add TODO for deleting output event
* Print Stack header to buffer
* Try to move resolving exception info to onExceptionInfoRequest
* save the error and return if it is the current thread
* rename thread to g
* findgoroutine returns goroutine
* clean up findgoroutine
* log errors
* remove output event
* fix grammar
* service/dap: add type information to dap variables
* add comment explaining map type choice
* rename to setClientCapabilities
* respond to review
* update TypeString definition
Adds the low-level support for watchpoints (aka data breakpoints) to
the native linux/amd64 backend.
Does not add user interface or functioning support for watchpoints
on stack variables.
Updates #279
Changes the expression evaluation code so that register names, when not
shadowed by local or global variables, will evaluate to the current
value of the corresponding CPU register.
This allows a greater flexibility with displaying CPU registers than is
possible with using the ListRegisters API call. Also it allows
debuggers users to view register values even if the frontend they are
using does not implement a register view.
* examinememory: evaluate addr as expression
This makes it easy to read memory locations at an offset of a known
address, e.g.:
x 0xc000046800 + 32
* use feedback from @aarzilli
- expression mode is now enabled via -x flag
- support "-x var", "-x &var" in addition to "-x <addr expr>"
- some refactoring
* add test cases
* deal with double spaces
* update docs
* add new failing test
* fix docs
* simplify implementation, update test & docs
* Fix docs
