OCPBUGS-80939: Add robots.txt policy to console

[logonoff]

Also simplify locales copy/paste into a glob (which also prevents the locales OWNERS files from being bundled)

Summary by CodeRabbit

• Chores
  • Added crawler policy to restrict search engine indexing of all site paths.

[openshift-ci-robot]

@logonoff: This pull request references Jira Issue OCPBUGS-80939, which is invalid:

• expected the bug to target the "4.22.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Also simplify locales copy/paste into a glob (which also prevents the locales OWNERS files from being bundled)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@logonoff: This pull request references Jira Issue OCPBUGS-80939, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

Also simplify locales copy/paste into a glob (which also prevents the locales OWNERS files from being bundled)

Summary by CodeRabbit

• Chores
• Added crawler policy to restrict search engine indexing of all site paths.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

📝 Walkthrough

Walkthrough

This change introduces web crawler directives by adding a robots.txt file that disallows all crawler access to the application. The frontend build configuration was updated to copy the robots.txt file during the webpack build process, while the locales copying pattern was refactored to use a glob-based approach for broader coverage. The backend Go server now includes an explicit HTTP route handler that serves the robots.txt file from the public directory. The webpack plugin import was also converted from CommonJS require to ES module syntax.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@frontend/public/robots.txt`:
- Around line 1-2: The robots.txt stanza has a malformed User-agent directive
("User-Agent *" missing the colon) so crawlers may ignore the stanza; update the
directive to the correct syntax by changing the first line to "User-agent: *"
(ensure exact capitalization/spelling used in your repository's convention) so
it pairs with the existing "Disallow: /" line and correctly blocks all crawlers.

In `@pkg/server/server.go`:
- Around line 338-340: The route for robots.txt is currently registered via
handleFunc which prefixes it with s.BaseURL.Path and therefore only serves
${basePath}/robots.txt; instead register an origin-root route for "/robots.txt"
(in addition to the existing base-path one) so crawlers can fetch "/robots.txt".
Update the registration so that the root path "/robots.txt" is served directly
(e.g., use http.HandleFunc or the router instance that is not wrapped with
s.BaseURL.Path) and call http.ServeFile(w, r, path.Join(s.PublicDir,
"robots.txt")) in that handler; reference the existing handleFunc,
s.BaseURL.Path, s.PublicDir and robots.txt when making the change.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: abbcfbe3-69a6-4fd5-a7b0-f767741cea44

📥 Commits

Reviewing files that changed from the base of the PR and between 0623c9b and b6a8b0f.

📒 Files selected for processing (3)

• frontend/public/robots.txt
• frontend/webpack.config.ts
• pkg/server/server.go

📜 Review details

🧰 Additional context used

📓 Path-based instructions (1)

**

⚙️ CodeRabbit configuration file

-Focus on major issues impacting performance, readability, maintainability and security. Avoid nitpicks and avoid verbosity.

Files:

• pkg/server/server.go
• frontend/public/robots.txt
• frontend/webpack.config.ts

🔇 Additional comments (1)

frontend/webpack.config.ts (1)

293-300: Good cleanup of the locale copy rules.

This keeps the /static/locales/... layout intact and filters out non-JSON files, so OWNERS files stop leaking into the bundle.

[jhadvig]

THank you @logonoff
adding one comment otherwise looks good ! :)

[jhadvig]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jhadvig, logonoff

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• frontend/OWNERS [jhadvig,logonoff]
• pkg/OWNERS [jhadvig]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[logonoff]

/assign @yapei

[yapei]

no functional changes and did a very simple verification in local testing, believe it should be working fine after PR got merged

/verified by @yapei

[openshift-ci-robot]

@yapei: This PR has been marked as verified by @yapei.

Details

In response to this:

no functional changes and did a very simple verification in local testing, believe it should be working fine

/verified by @yapei

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[logonoff]

/test all

[logonoff]

/test all

[openshift-ci]

@logonoff: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-gcp-console	930da3d	link	true	/test e2e-gcp-console

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD 703563f and 2 for PR HEAD 930da3d in total