Skip to content

Add bundle telemetry for variables, validation gap, and deploy graph#4852

Draft
shreyas-goenka wants to merge 7 commits intomainfrom
more-telemetyr
Draft

Add bundle telemetry for variables, validation gap, and deploy graph#4852
shreyas-goenka wants to merge 7 commits intomainfrom
more-telemetyr

Conversation

@shreyas-goenka
Copy link
Copy Markdown
Contributor

@shreyas-goenka shreyas-goenka commented Mar 26, 2026

Summary

  • Detect nested variable references (${var.foo_${var.bar}}) to understand how common var-in-var patterns are
  • Track variable patterns: defaults count, lookup type distribution, override sources (env var / file / CLI flag), resolution round depth
  • Track validation-pass-deploy-fail gap: validation_passed bool metric + deferred telemetry logging so deploy failures are always captured
  • Track direct engine deployment plan metrics: action type distribution, per-type resource counts, dependency graph max depth and max width (informs parallelism decisions)
  • Remove unused telemetry fields: SetFields, TerraformStateSizeBytes, FromWebTerminal

Test plan

  • Unit tests for nested var ref detection
  • Unit tests for variable override source tracking
  • Unit tests for resolution round depth metrics
  • Unit tests for deployment plan telemetry (13 test cases covering graph shapes)
  • Acceptance test outputs updated for validation_passed metric
  • go build ./... passes
  • make lint clean

This pull request was AI-assisted by Isaac.

The codegen tool (`go run .`) now automatically downloads the SHA256SUMS
file from the GitHub release and embeds the checksums into the generated
root.go. When bumping the provider version, developers only need to
update version.go — checksums are resolved automatically.

Co-authored-by: Isaac
These tests download large archives from the internet. Gate them behind
`testing.Short()` so they're skipped during normal CI (`make test`) and
only run in nightly/long test sessions.

Co-authored-by: Isaac
…check

Instead of a separate test that downloads large archives on every test
run, verify the checksum inline during codegen: FetchProviderChecksums
now downloads the linux_amd64 zip and verifies it matches the parsed
SHA256SUMS entry. This runs once during `go run .` (provider version
bump) rather than on every `make test`.

Co-authored-by: Isaac
Co-authored-by: Isaac
@eng-dev-ecosystem-bot
Copy link
Copy Markdown
Collaborator

eng-dev-ecosystem-bot commented Mar 26, 2026

Commit: 3c456c1

Run: 23605248732

Env 🔄​flaky 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
💚​ aws linux 7 10 270 804 8:10
💚​ aws windows 7 10 272 802 7:10
💚​ aws-ucws linux 7 10 366 720 20:42
🔄​ aws-ucws windows 3 6 10 366 718 20:45
💚​ azure linux 1 12 273 802 8:57
💚​ azure windows 1 12 275 800 6:41
💚​ azure-ucws linux 1 12 371 716 21:03
🔄​ azure-ucws windows 4 12 370 714 22:37
💚​ gcp linux 1 12 269 805 7:34
💚​ gcp windows 1 12 271 803 6:17
21 interesting tests: 10 SKIP, 6 RECOVERED, 5 flaky
Test Name aws linux aws windows aws-ucws linux aws-ucws windows azure linux azure windows azure-ucws linux azure-ucws windows gcp linux gcp windows
🔄​ TestAccept 💚​R 💚​R 💚​R 🔄​f 💚​R 💚​R 💚​R 🔄​f 💚​R 💚​R
🙈​ TestAccept/bundle/resources/permissions 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🔄​ TestAccept/bundle/resources/permissions/jobs/delete_one/cloud 🙈​s 🙈​s ✅​p 🔄​f 🙈​s 🙈​s ✅​p 🔄​f 🙈​s 🙈​s
🔄​ TestAccept/bundle/resources/permissions/jobs/delete_one/cloud/DATABRICKS_BUNDLE_ENGINE=direct ✅​p 🔄​f ✅​p ✅​p
🔄​ TestAccept/bundle/resources/permissions/jobs/delete_one/cloud/DATABRICKS_BUNDLE_ENGINE=terraform ✅​p ✅​p ✅​p 🔄​f
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions 💚​R 💚​R 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions 💚​R 💚​R 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/resources/postgres_branches/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/recreate 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/update_protected 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/without_branch_id 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_endpoints/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_endpoints/recreate 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_projects/update_display_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/synced_database_tables/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🔄​ TestSyncIncrementalFileSync ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f ✅​p ✅​p
Top 50 slowest tests (at least 2 minutes):
duration env testname
4:28 aws-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
4:16 aws-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
4:12 aws-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
4:07 azure-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
4:00 azure linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:59 azure windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:49 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:46 azure-ucws windows TestAccept/bundle/resources/model_serving_endpoints/basic/DATABRICKS_BUNDLE_ENGINE=direct
3:44 aws-ucws linux TestAccept/bundle/resources/volumes/recreate/DATABRICKS_BUNDLE_ENGINE=direct
3:43 azure-ucws windows TestAccept/bundle/resources/permissions/jobs/delete_one/cloud/DATABRICKS_BUNDLE_ENGINE=direct
3:33 azure-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:33 aws-ucws windows TestAccept/bundle/resources/volumes/recreate/DATABRICKS_BUNDLE_ENGINE=terraform
3:31 azure-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:27 aws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:25 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:18 azure linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:17 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:16 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:15 azure-ucws linux TestFilerWorkspaceFilesExtensionsRead
3:14 aws-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:07 azure-ucws linux TestAccept/bundle/deployment/bind/job/generate-and-bind/DATABRICKS_BUNDLE_ENGINE=terraform
3:07 aws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:05 aws-ucws windows TestAccept/bundle/deploy/files/no-snapshot-sync/DATABRICKS_BUNDLE_ENGINE=terraform
3:03 azure windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:00 azure-ucws windows TestSparkJarTaskDeployAndRunOnWorkspace/Databricks_Runtime_15.4_LTS
2:58 azure-ucws linux TestAccept/bundle/deploy/files/no-snapshot-sync/DATABRICKS_BUNDLE_ENGINE=terraform
2:49 aws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:49 aws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:45 azure-ucws linux TestAccept/bundle/deploy/files/no-snapshot-sync/DATABRICKS_BUNDLE_ENGINE=direct
2:41 azure-ucws windows TestAccept/bundle/deployment/bind/job/generate-and-bind/DATABRICKS_BUNDLE_ENGINE=direct
2:40 aws-ucws linux TestAccept/bundle/resources/model_serving_endpoints/basic/DATABRICKS_BUNDLE_ENGINE=direct
2:38 azure-ucws windows TestAccept/bundle/resources/volumes/recreate/DATABRICKS_BUNDLE_ENGINE=direct
2:37 aws-ucws linux TestAccept/bundle/resources/volumes/recreate/DATABRICKS_BUNDLE_ENGINE=terraform
2:37 aws-ucws windows TestAccept/bundle/resources/permissions/jobs/delete_one/cloud/DATABRICKS_BUNDLE_ENGINE=terraform
2:36 azure-ucws windows TestAccept/bundle/resources/permissions/jobs/delete_one/cloud/DATABRICKS_BUNDLE_ENGINE=terraform
2:35 azure-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:34 aws-ucws windows TestAccept/bundle/resources/alerts/with_file/DATABRICKS_BUNDLE_ENGINE=terraform
2:33 azure-ucws linux TestExportDir
2:28 aws-ucws windows TestImportDir
2:27 aws-ucws windows TestAccept/bundle/destroy/jobs-and-pipeline/DATABRICKS_BUNDLE_ENGINE=terraform
2:25 aws-ucws windows TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform
2:25 azure-ucws windows TestAccept/bundle/deploy/files/no-snapshot-sync/DATABRICKS_BUNDLE_ENGINE=terraform
2:23 aws-ucws windows TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=terraform
2:22 azure-ucws linux TestAccept/bundle/resources/volumes/recreate/DATABRICKS_BUNDLE_ENGINE=direct
2:20 aws-ucws linux TestAccept/bundle/resources/model_serving_endpoints/basic/DATABRICKS_BUNDLE_ENGINE=terraform
2:19 azure-ucws linux TestAccept/bundle/resources/permissions/jobs/delete_one/cloud/DATABRICKS_BUNDLE_ENGINE=direct
2:19 aws-ucws windows TestAccept/bundle/resources/model_serving_endpoints/basic/DATABRICKS_BUNDLE_ENGINE=direct
2:17 aws-ucws windows TestFilerReadDir/workspace_files
2:16 azure-ucws windows TestAccept/bundle/deployment/bind/job/generate-and-bind/DATABRICKS_BUNDLE_ENGINE=terraform
2:16 aws-ucws linux TestAccept/bundle/deploy/files/no-snapshot-sync/DATABRICKS_BUNDLE_ENGINE=terraform

…pendency graph

Add several new telemetry signals to improve understanding of DABs usage patterns:

- Detect nested variable references (var-in-var patterns like ${var.foo_${var.bar}})
- Track variable defaults count, lookup type distribution, override sources (env var, file, CLI flag), and resolution depth
- Track validation-pass-deploy-fail gap via "validation_passed" metric with deferred telemetry logging
- Track direct engine deployment plan metrics: action types, resource counts, dependency graph depth/width
- Remove unused telemetry fields: SetFields, TerraformStateSizeBytes, FromWebTerminal

Co-authored-by: Isaac
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants