Skip to content

Conversation

@dominikschubert
Copy link
Member

@dominikschubertdominikschubert commented Mar 17, 2023

Adds the xray daemon as a dependency and starts the daemon asynchronously.

The daemon is stopped and recorded segments are flushed when init shuts down (see signal handler / shutdown funcs in sandbox).

It's not yet ideal, but it provides parity with the old xray implementation. For proper active & passive tracing with additional generated (sub-)segments we'll need to add a few more things

  • Generate & publish traces in init. There's actually already some things we can re-use here
  • Properly use provided credentials for service calls
  • Generate the root trace segment in the lambda service.

Other changes:

  • Added a basic LOCALSTACK_INIT_LOG_LEVEL internal variable for us to be able to configure the log level in the init binary without re-compiling. I don't think this will be the final setup, since I still want to rework the general log setup here, but it's kinda useful in the meantime.

@dominikschubertdominikschubert changed the title Start x-ray daemon when the request is sampledAdd X-Ray daemonMar 19, 2023
Copy link
Member

@joe4devjoe4dev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

just minor comments but nothing blocking.

xrayConfig.NoVerifySSL=util.Bool(true) // obvious
xrayConfig.LocalMode=util.Bool(true) // skip EC2 metadata check
xrayConfig.Region=GetEnvOrDie("AWS_REGION")
xrayConfig.Logging.LogLevel="dev"// TODO
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When do we adjust that to avoid log spamming?
The daemon itself should be stable, this might just be interesting for debugging.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It periodically logs Skipped telemetry data as no segments found or the number of traces sent.

@joe4dev
Copy link
Member

The PR description is not up-to-date anymore after the change we introduced for updating the X-Ray TraceId with every request.

@joe4dev
Copy link
Member

Could it be that the runtime starts before the X-Ray daemon due to async runDaemon(d) given the warning: Subsegment lambda discarded due to Lambda worker still initializing 🤔 (probably not blocking now but good to follow up later)

2023-03-20 23:02:07 2023-03-20T22:02:07Z [Debug] Batch size: 10 2023-03-20 23:02:07 2023-03-20T22:02:07Z [Info] Starting proxy http server on 127.0.0.1:2000 2023-03-20 23:02:07 time="2023-03-20T22:02:07Z" level=debug msg="Hot reloading disabled." func=main.RunHotReloadingListener file="/Users/joe/Projects/LocalStack/lambda-runtime-init/cmd/localstack/awsutil.go:150" 2023-03-20 23:02:07 time="2023-03-20T22:02:07Z" level=debug msg="Runtime API Server listening on 127.0.0.1:9001" func="go.amzn.com/lambda/rapi.(*Server).Listen" file="/Users/joe/Projects/LocalStack/lambda-runtime-init/lambda/rapi/server.go:93" 2023-03-20 23:02:07 time="2023-03-20T22:02:07Z" level=info msg="Configure environment for Init Caching." func="go.amzn.com/lambda/rapid.(*rapidContext).acceptStartRequestForInitCaching" file="/Users/joe/Projects/LocalStack/lambda-runtime-init/lambda/rapid/start.go:391" 2023-03-20 23:02:07 time="2023-03-20T22:02:07Z" level=info msg="extensionsDisabledByLayer(/opt/disable-extensions-jwigqn8j) -> stat /opt/disable-extensions-jwigqn8j: no such file or directory" func=go.amzn.com/lambda/rapid.extensionsDisabledByLayer file="/Users/joe/Projects/LocalStack/lambda-runtime-init/lambda/rapid/start.go:363" 2023-03-20 23:02:07 time="2023-03-20T22:02:07Z" level=debug msg="Received RUNNING" func="go.amzn.com/lambda/rapidcore.(*Server).Init" file="/Users/joe/Projects/LocalStack/lambda-runtime-init/lambda/rapidcore/server.go:533" 2023-03-20 23:02:07 time="2023-03-20T22:02:07Z" level=warning msg="Cannot list external agents" func=go.amzn.com/lambda/agents.ListExternalAgentPaths file="/Users/joe/Projects/LocalStack/lambda-runtime-init/lambda/agents/agent.go:71" error="open /opt/extensions: no such file or directory" 2023-03-20 23:02:07 time="2023-03-20T22:02:07Z" level=debug msg="Preregister runtime" func=go.amzn.com/lambda/rapid.doInit file="/Users/joe/Projects/LocalStack/lambda-runtime-init/lambda/rapid/start.go:189" 2023-03-20 23:02:07 time="2023-03-20T22:02:07Z" level=debug msg="Start runtime" func=go.amzn.com/lambda/rapid.doInit file="/Users/joe/Projects/LocalStack/lambda-runtime-init/lambda/rapid/start.go:222" 2023-03-20 23:02:10 [INFO] 2023-03-20T22:02:10.899Z successfully patched module requests 2023-03-20 23:02:10 [INFO] 2023-03-20T22:02:10.925Z successfully patched module botocore 2023-03-20 23:02:10 [INFO] 2023-03-20T22:02:10.948Z successfully patched module sqlite3 2023-03-20 23:02:11 [INFO] 2023-03-20T22:02:11.032Z Found credentials in environment variables. 2023-03-20 23:02:11 [WARNING] 2023-03-20T22:02:11.290Z Subsegment lambda discarded due to Lambda worker still initializing 2023-03-20 23:02:12 [WARNING] 2023-03-20T22:02:12.044Z No subsegment to end. 

@dominikschubert
Copy link
MemberAuthor

Subsegment lambda discarded due to Lambda worker still initializing

Seems to be related to this issue here: aws/aws-xray-sdk-python#185

@dominikschubertdominikschubert merged commit bede0a0 into localstackMar 21, 2023
@dominikschubertdominikschubert deleted the feat_xray-integration branch March 21, 2023 07:51
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@dominikschubert@joe4dev