Enhancement: Allow Disabling Indexing

[barreeeiroo]

Proposed change

Allows passing the disableIndexing parameter to settings in order to disable indexing in robots.txt and in the meta tag.

Closes #5952

Type of change

• New service widget
• Bug fix (non-breaking change which fixes an issue)
• New feature or enhancement (non-breaking change which adds functionality)
• Documentation only
• Other (please explain)

Checklist:

• If applicable, I have added corresponding documentation changes.
• If applicable, I have reviewed the feature / enhancement and / or service widget guidelines.
• I have checked that all code style checks pass using pre-commit hooks and linting checks.
• If applicable, I have tested my code for new features & regressions on both mobile & desktop devices, using the latest version of major browsers.

AI Disclaimer

This change was built with the assistance of Generative AI.

[shamoon]

Thanks. I will say the spirit of the feature request requirement is to allow for some input to suggest there is community interest in a feature, though I do expect there would be for this.

Also did you use AI for the PR?

[barreeeiroo]

Hi @shamoon,

Yeah, I read the guideliness requiring all PRs to be "backed" by substantial interest in the community. However, I reckon that I literally opened the feature request just a few hours ago, and no interest has appeared (hopefully "yet").
But, given this was a generic change, which is mostly supported by many self-hostable services, I assumed it would be useful to have here as well.

Regarding the addition of extra maintenance, once I saw this project was using Next.js 15, I think it might require changes in the future... I've experienced the breaking updates which were introduced in earlier versions, so not sure if anything that I've used here (rewrite or api/ path) might break in the future...
I'm also not 100% sure if this implementation is the cleanest one (having a dynamic api/ generating the content, and rewriting the root URL). Ideally it should be directly in the root path so api/robots.txt does not exist. I'll check later if there are other possible implementations though.

And yes, I've used Claude Code to kinda help with understanding the repo (as it was the first time modifying it). I was expecting it to be more complex, but turned out to be a simple implementation.
Anyhow, added a disclaimer in the PR description now (I didn't find it in the template nor in the guidelines though).

[shamoon]

It just helps to know if AI was used so we can have a slightly better understanding of a PR authors abilities (both to generate the PR and / or deal with anything that arises).

Yea I’m also not sure if this is the best way to do it, suppose I’ll have to look around / think.

I agree I expect others will want this, honestly we have just found that even the most seemingly small/benign changes can have unintended consequences, hence asking for some demonstration of community interest before we agree to “take on” the changes. Of course many small changes don’t cause issues.

[barreeeiroo]

So, I'm checking their docs for v15 using pages router, and it seems like this is the only way to achieve it... The only "official" way to achieve something similar without the rewrite is using the app router:

• https://nextjs.org/docs/15/app/api-reference/file-conventions/metadata/robots

[shamoon]

I guess I dont know for sure but perhaps 'official' is the way to go, seems like it would be simple to adapt that to being dynamic based on the setting?

[barreeeiroo]

But the issue is that the official cannot be implemented here, as homepage is using the pages router. I tried creating the file following the guide for the app router, but it just crashes in homepage if so.

• Tried creating both robots.js and robots.txt.js in pages/, but both expect a React component (not "raw text") as output.
  • See this: https://www.reddit.com/r/nextjs/comments/qrfwnh/comment/hkbe95e/
• Also tried returning the "robots object" they say we can use in app router, but it complains saying it's not a React component.

[shamoon]

Weird. I guess I’d have to play with it, it’s not something I know off the top of my head. but yea maybe not

[barreeeiroo]

How feasible is it to migrate to the app router? Jk, this was one of the breaking changes I was referring to: they introduced the app router a while ago, but somehow the pages router is still there but seems like it has less features...

[shamoon]

Ah I see, yea I’m not familiar. Yea let’s not do that

[shamoon]

I think this is a bit cleaner. Take a look and LMK if you have any thoughts. I will note there are still 0 votes on your FR... of course its bare been a day

[barreeeiroo]

Ah, didn't know you can manually write a response from getServerSideProps. Yes, definitely cleaner, thanks!

The only thing I would change, but it's more of a nitpick, is that I would prefer to explicitly define as a string the possible content of robots.txt, rather than an array of strings dynamically generated. I think it's cleaner and easier to understand (basically the res.send I had in the original change).
But up to your preference, it's more of a small change for readability. I can change it if you also agree.

Regarding the number of upvotes, yeah... I was just concerned because for some reason Google found my site, and having to manually request page removals is not fun.
Also, I just came across this now: #2351 (I don't know how I missed it yesterday). Does this work for non-Docker installs though?

[shamoon]

Also, I just came across this now: #2351 (I don't know how I missed it yesterday). Does this work for non-Docker installs though?

I just tested it and it most certainly does. Hmm, I guess Im a little ambivalent here, I suppose the setting would be about 10% easier? 😆

[barreeeiroo]

Well, the choice is yours haha. Ofc having the feature flag in settings makes it much easier and discoverable. But in case you want to go for the "do it yourself" approach, I would suggest to at least have it documented somewhere in the docs (not just in a discussion). :)

[shamoon]

Still no votes but I think we're ok here

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new discussion for related concerns. See our contributing guidelines for more details.