perf: use batch brew install with native parallelism#19
perf: use batch brew install with native parallelism#19jerryjrxie wants to merge 8 commits intoopenbootdotdev:mainfrom
Conversation
|
👋 Thanks for opening this pull request! 🎉 This is your first PR to OpenBoot — welcome! Before merging:
@fullstackjam will review this soon. Thanks for contributing! 🚀 |
acd4139 to
8bfa497
Compare
There was a problem hiding this comment.
Pull request overview
This PR simplifies Homebrew package installation by removing the custom parallel worker pool and switching to a single batch brew install ... invocation (letting Homebrew manage native parallel downloads). It also carries forward the stacked fix to ensure remote configs include casks (GUI apps) in selected packages.
Changes:
- Replace per-formula concurrent
brew install <pkg>calls with one batchbrew install <pkg1> <pkg2> ...for CLI packages, and update dry-run output accordingly. - Remove the custom
runParallelInstallWithProgressworker-pool implementation. - Ensure casks from remote configs are included in
SelectedPkgs(installer + package customization step).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
internal/brew/brew.go |
Switch CLI installs to batch brew install and remove parallel worker pool; adjust dry-run printing. |
internal/installer/installer.go |
Include remote-config casks in SelectedPkgs during custom installs. |
internal/installer/step_packages.go |
Include remote-config casks in SelectedPkgs during package customization. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
internal/brew/brew.go
Outdated
|
|
||
| args := append([]string{"install"}, newCli...) | ||
| cmd := brewInstallCmd(args...) | ||
| output, err := cmd.CombinedOutput() | ||
| outputStr := string(output) | ||
|
|
||
| for _, pkg := range newCli { | ||
| progress.IncrementWithStatus(true) | ||
| if err == nil || !strings.Contains(outputStr, pkg) { | ||
| installedFormulae = append(installedFormulae, pkg) | ||
| } else { | ||
| errMsg := parseBrewError(outputStr) | ||
| if errMsg == "" { | ||
| errMsg = "install failed" | ||
| } | ||
| allFailed = append(allFailed, failedJob{ | ||
| installJob: installJob{name: pkg, isCask: false}, | ||
| errMsg: errMsg, | ||
| }) | ||
| } |
There was a problem hiding this comment.
The per-package success detection for the batch brew install is not reliable and will misclassify installs. In the error case (err != nil), strings.Contains(outputStr, pkg) is neither a failure indicator nor a success indicator: successful packages will often appear in output and be marked failed, and packages that never ran may be marked installed if they don't appear in the output. This can also cause incorrect state tracking (packages marked installed when they are not) and incorrect progress status (currently always increments as success).
A more robust approach is to run the batch install, then re-check installed formulae (e.g., call GetInstalledPackages() again) and classify each pkg based on whether it is now installed; only those still missing should be added to allFailed / retried, and IncrementWithStatus should use the real success value.
| args := append([]string{"install"}, newCli...) | |
| cmd := brewInstallCmd(args...) | |
| output, err := cmd.CombinedOutput() | |
| outputStr := string(output) | |
| for _, pkg := range newCli { | |
| progress.IncrementWithStatus(true) | |
| if err == nil || !strings.Contains(outputStr, pkg) { | |
| installedFormulae = append(installedFormulae, pkg) | |
| } else { | |
| errMsg := parseBrewError(outputStr) | |
| if errMsg == "" { | |
| errMsg = "install failed" | |
| } | |
| allFailed = append(allFailed, failedJob{ | |
| installJob: installJob{name: pkg, isCask: false}, | |
| errMsg: errMsg, | |
| }) | |
| } | |
| args := append([]string{"install"}, newCli...) | |
| cmd := brewInstallCmd(args...) | |
| output, err := cmd.CombinedOutput() | |
| outputStr := string(output) | |
| installedAfter, installedErr := GetInstalledPackages() | |
| if installedErr != nil { | |
| // Fallback to output-based heuristic if we cannot query installed packages. | |
| for _, pkg := range newCli { | |
| // Preserve previous behavior: always increment as success in this path. | |
| progress.IncrementWithStatus(true) | |
| if err == nil || !strings.Contains(outputStr, pkg) { | |
| installedFormulae = append(installedFormulae, pkg) | |
| } else { | |
| errMsg := parseBrewError(outputStr) | |
| if errMsg == "" { | |
| errMsg = "install failed" | |
| } | |
| allFailed = append(allFailed, failedJob{ | |
| installJob: installJob{name: pkg, isCask: false}, | |
| errMsg: errMsg, | |
| }) | |
| } | |
| } | |
| } else { | |
| installedSet := make(map[string]struct{}, len(installedAfter)) | |
| for _, name := range installedAfter { | |
| installedSet[name] = struct{}{} | |
| } | |
| for _, pkg := range newCli { | |
| _, installed := installedSet[pkg] | |
| progress.IncrementWithStatus(installed) | |
| if installed { | |
| installedFormulae = append(installedFormulae, pkg) | |
| } else { | |
| errMsg := parseBrewError(outputStr) | |
| if errMsg == "" { | |
| errMsg = "install failed" | |
| } | |
| allFailed = append(allFailed, failedJob{ | |
| installJob: installJob{name: pkg, isCask: false}, | |
| errMsg: errMsg, | |
| }) | |
| } | |
| } |
| for _, cask := range st.RemoteConfig.Casks { | ||
| st.SelectedPkgs[cask.Name] = true | ||
| } | ||
|
|
There was a problem hiding this comment.
runCustomInstall now includes casks in SelectedPkgs, but the earlier status message still reports only len(st.RemoteConfig.Packages) as the number of packages to install. This will undercount when remote configs include casks (and/or npm). Consider updating that ui.Info(...) to use the already-computed totals (formulae+casks+npm) or at least include cask count so the message matches what will actually be installed.
| totalBrewPkgs := len(st.RemoteConfig.Packages) + len(st.RemoteConfig.Casks) | |
| totalNpmPkgs := len(st.RemoteConfig.Npm) | |
| if totalBrewPkgs > 0 { | |
| if totalNpmPkgs > 0 { | |
| ui.Info(fmt.Sprintf("Installing %d Homebrew packages (including casks) and %d npm packages...", totalBrewPkgs, totalNpmPkgs)) | |
| } else { | |
| ui.Info(fmt.Sprintf("Installing %d Homebrew packages (including casks)...", totalBrewPkgs)) | |
| } | |
| } else if totalNpmPkgs > 0 { | |
| ui.Info(fmt.Sprintf("Installing %d npm packages...", totalNpmPkgs)) | |
| } |
| if len(newCli) > 0 { | ||
| failed := runParallelInstallWithProgress(newCli, progress) | ||
| failedSet := make(map[string]bool, len(failed)) | ||
| for _, f := range failed { | ||
| failedSet[f.name] = true | ||
| } | ||
| for _, p := range newCli { | ||
| if !failedSet[p] { | ||
| installedFormulae = append(installedFormulae, p) | ||
| progress.SetCurrent(fmt.Sprintf("Installing %d CLI packages...", len(newCli))) | ||
| ui.Info(fmt.Sprintf("Installing %d CLI packages with native brew parallelism...", len(newCli))) | ||
|
|
||
| args := append([]string{"install"}, newCli...) | ||
| cmd := brewInstallCmd(args...) | ||
| output, err := cmd.CombinedOutput() | ||
| outputStr := string(output) | ||
|
|
||
| for _, pkg := range newCli { | ||
| progress.IncrementWithStatus(true) | ||
| if err == nil || !strings.Contains(outputStr, pkg) { | ||
| installedFormulae = append(installedFormulae, pkg) | ||
| } else { | ||
| errMsg := parseBrewError(outputStr) | ||
| if errMsg == "" { | ||
| errMsg = "install failed" | ||
| } | ||
| allFailed = append(allFailed, failedJob{ | ||
| installJob: installJob{name: pkg, isCask: false}, | ||
| errMsg: errMsg, | ||
| }) | ||
| } | ||
| } |
There was a problem hiding this comment.
This change introduces new non-dry-run behavior in InstallWithProgress (batch install + per-package classification) but current tests only cover the dry-run path. Adding unit-level coverage around the new classification path would help prevent regressions (e.g., by refactoring to inject the brew command runner and an "installed packages" provider so failures/partial successes can be simulated deterministically).
00141ed to
562ebe3
Compare
The previous implementation only checked --global git config, but users may have their git identity configured in: - Local repository config (.git/config) - System config (/etc/gitconfig) - Other scopes This change first tries --global, then falls back to checking all scopes if --global returns empty. This ensures we detect git configuration regardless of where it's set. Closes git config detection issue
203a0b2 to
28a4700
Compare
Adds TestGetGitConfig_FallsBackToAnyScope to verify that GetGitConfig checks all git config scopes (global, local, system) when looking for user.name and user.email, not just --global. Related to git config detection issue
The diffDotfiles function was always checking the local ~/.dotfiles git state, even when comparing snapshots with empty dotfiles URLs. This caused test failures when the user's actual dotfiles repo had uncommitted changes. Fix: Only check dotfiles repo state if at least one URL is configured. If both system and reference have empty dotfiles URLs, skip the git state check entirely. This makes the tests deterministic and not dependent on the user's local dotfiles repo state.
When installing from a remote config (e.g., 'openboot install jx'), GUI apps (casks) were being silently skipped. The code only added CLI packages to the SelectedPkgs map but never added the Casks. This fix adds the missing loop to include casks in both: - internal/installer/installer.go (runCustomInstall) - internal/installer/step_packages.go (stepPackageCustomization) Closes openbootdotdev#17
Adds TestRunCustomInstall_IncludesCasksInSelectedPkgs to verify that GUI apps (casks) from remote configs are properly added to SelectedPkgs. This prevents regression of the bug where casks were silently skipped during remote config installation. Related to openbootdotdev#17
Simplifies the install process by piping brew's native output directly to stdout/stderr instead of capturing it and building custom progress. Changes: - CLI packages: brew install output shown directly - GUI apps: brew install --cask output shown directly with TTY for passwords - Removed complex output parsing and custom progress display - Progress bar still tracks completion count but doesn't intercept brew output - 91 lines removed, much simpler and more maintainable Users now see Homebrew's native progress indicators which are more accurate and familiar.
- TestInstallWithProgress_BatchMode: verifies batch install behavior - TestResolveFormulaName: tests formula alias resolution These tests ensure the batch install approach works correctly and formula aliases are properly resolved before tracking.
5eb2afd to
f5271b2
Compare
- Check GetInstalledPackages() after batch install to determine per-package success - Update status messages to include cask count in package totals
f99c6a5 to
b176f99
Compare
Summary
Simplifies package installation by piping Homebrew's native output directly to the user instead of building custom progress indicators.
Problem
The custom progress implementation was complex, inaccurate, and duplicated what Homebrew already does well. We were capturing output, parsing it, and building our own progress bars.
Solution
Just let brew show its output:
brew installstdout/stderr directly to the terminalChanges
91 lines removed → much simpler
Benefits
Testing
go vetpassesStacked on PR #18
Includes the GUI apps fix from #18.