Update Jenkins core and plugins (2023Q3 edition)

Last time: #18011 (closed)

Preparation

• Go through the changelog, paying attention to changes on how agents connect to controller, config changes that may need update, important changes in plugins, etc.
  • Current version: 2.401.1
  • Available version: 2.414.2

May need attention:

• Remove support for WebSocket agents when running inside Jetty 9. (pull 7101)

Deployment

• Deploy Jenkins upgrade to latest version available
• Use the Groovy and Python3 scripts to generate a list of up-to-date plugins
• Deploy plugin upgrades
• Restart all agents.
• Manually run the Update jobs script (may be needed so XML is valid with current Jenkins):

  sudo -u jenkins /usr/local/sbin/deploy_jenkins_jobs update

• Go through warnings in Jenkins interface.
• Manually uninstall removed plugins via Jenkins interface.
• Manually delete removed plugins from /var/lib/jenkins/plugins.
• Check if we can safely revert to the upstream Jenkins Puppet module

Wrap up

• Restart builds that were interrupted by Jenkins restart.
• Schedule next update.