Server Provisioning and Hardware Lifecycle Questions

Baseline summary (applied immediately after provisioning)

1) SSH hardening

• Enforce Protocol 2, disable root login, use key auth only, limit ciphers/MACs, change default port optionally, enable IdleTimeout.
• Why: prevents password brute-force and limits attack surface.
• Automation: Ansible role to deploy /etc/ssh/sshd_config and restart sshd; cloud-init for first-boot key injection.

2) Account & sudo policies

• Create non-root admin group (wheel/sudo), disable direct root SSH, enforce password complexity & max reuse via pam_pwquality, set sudo to require auth and log via Defaults logfile=/var/log/sudo.log.
• Why: least privilege and accountability.
• Automation: Manage /etc/sudoers.d and PAM via Ansible; use LDAP/AD or SSSD for central auth.

3) Minimal services

• Disable/remove unused daemons (telnet, ftp, rpcbind, avahi). Use systemctl mask/disable.
• Why: fewer services = fewer vulnerabilities.
• Automation: Inventory with Ansible/systemd templates; apply disable list.

4) File permissions

• Enforce /etc/ssh ownership and perms, /etc/shadow 640 root:shadow, home dirs 750.
• Why: prevents info disclosure.
• Automation: Ansible tasks with file module and a remediation playbook.

5) Audit & logging

• Enable auditd rules for execs, auth, sudo, network changes; forward logs to central SIEM (rsyslog/Fluentd).
• Why: detects intrusion and provides forensics.
• Automation: Deploy audit.rules and rsyslog config with Ansible; use ossec/wazuh agents.

6) Kernel tweaks

• Harden network (disable ipv6 if unused, disable ip_forward, enable rp_filter, tighten net.ipv4.conf.* settings), enable ASLR, cgroup hardening.
• Why: reduces kernel-level attack vectors.
• Automation: sysctl.d files via Ansible and sysctl --system; use CIS benchmark scripts.

7) Patch policy

• Apply critical security updates within 24h, regular weekly patch window for other updates; enable unattended-upgrades for security only on non-production.
• Why: timely patching reduces exploit window.
• Automation: configure unattended-upgrades, apt/yum-cron for baseline; orchestration via Ansible/Cron + patch reporting.

Enforcement & Compliance

• Use a configuration management system (Ansible / Salt / Puppet), CIS benchmark scanning (OpenSCAP, Lynis), and remediation playbooks run in CI or on-boot.
• Maintain idempotent playbooks, automated tests (molecule), and centralized reporting for drift detection.

This baseline balances immediate risk reduction with automation to ensure consistent, auditable, and maintainable server hardening.

Approach

• Create a user "deploy" with sudo and an SSH public key.
• Ensure apt lists are updated and required packages installed.
• Start and enable nginx; install a monitoring agent (example: datadog-agent placeholder).
• Keep the snippet minimal and ready to paste as cloud-init user-data.

cloud-init user-data (YAML)

#cloud-config
users:
  - name: deploy
    gecos: Deploy User
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ssh_authorized_keys:
      - "ssh-rsa AAAA...your-public-key... user@example.com"

package_update: true
packages:
  - nginx
  - curl
  - apt-transport-https

runcmd:
  - [ sh, -c, "systemctl enable --now nginx" ]
  - [ sh, -c, "echo 'Installing monitoring agent (example)'; curl -sS https://example.com/install-agent.sh | bash" ]

Notes:

• Replace the SSH key and monitoring-agent install command/URL with your actual public key and vendor instructions (e.g., Datadog, Prometheus node-exporter).
• For production, pin package versions or add apt repositories and GPG keys securely.

Sample Answer (DSC snippet + explanation)

Below is a one-node PowerShell DSC configuration that ensures the IIS role (Web-Server), a specific Windows feature (for example Web-ASP.NET45) is present, deploys a configuration file to C:\inetpub\wwwroot, and starts the W3SVC service.

Configuration IISSiteDeploy {
    Import-DscResource -ModuleName PSDesiredStateConfiguration
    Import-DscResource -ModuleName xWebAdministration -ModuleVersion 2.0.0.0 -ErrorAction SilentlyContinue

    Node "localhost" {
        # Ensure IIS (Web-Server) is installed
        WindowsFeature IIS {
            Name   = "Web-Server"
            Ensure = "Present"
        }

        # Ensure a specific feature (example: ASP.NET 4.5) is installed
        WindowsFeature ASPNET45 {
            Name   = "Web-Asp-Net45"
            Ensure = "Present"
            DependsOn = "[WindowsFeature]IIS"
        }

        # Deploy a simple configuration file to the default site root
        File DeployIndex {
            DestinationPath = "C:\inetpub\wwwroot\index.html"
            Contents        = @"
<!DOCTYPE html>
<html><body><h1>IIS DSC deployed page</h1></body></html>
"@
            Type            = "File"
            Ensure          = "Present"
            DependsOn       = "[WindowsFeature]IIS"
        }

        # Ensure the World Wide Web Publishing Service is running
        Service W3SVCService {
            Name       = "W3SVC"
            StartupType = "Automatic"
            State      = "Running"
            DependsOn  = "[WindowsFeature]IIS"
        }
    }
}

# Compile and apply
IISSiteDeploy -OutputPath .\IISConfig
Start-DscConfiguration -Path .\IISConfig -Wait -Verbose -Force

Why this works

• WindowsFeature ensures roles/features are installed; DependsOn enforces order.
• File resource writes the HTML directly; for larger sites use SourcePath or a DSC pull server.
• Service resource guarantees IIS is started.
• Run the config on the target node (localhost for one-node). For managed environments, you can use Push/Pull modes and add credentials or configuration data as needed.

Edge cases / tips

• Run PowerShell elevated.
• If using WebAdministration resources (xWebAdministration or full DSC WebAdministration), import the module and adjust resources for site/app configuration.
• For production content, prefer SourcePath or package from an artifact repository and verify file permissions with the File resource.

Approach (requirements)
Daily incremental, weekly full, offsite retention 90 days; automate, verify, and test restores.

Tools & configuration

• Backup software: restic (dedup, encrypted snapshots) or BorgBackup.
• Offsite storage: AWS S3 with lifecycle to Glacier/Glacier Deep Archive or S3 Intelligent-Tiering. Use rclone or restic native S3 backend.
• Scheduler: systemd timers or cronjobs invoking restic; use --prune and snapshot tagging.
• IaC for test env: Terraform + Ansible to provision clean test server and restore pipeline.

Policy

• Weekly full snapshot (tag: weekly-full) on Sunday 02:00.
• Daily incrementals other days at 02:00.
• Retention: keep daily for 14 days, weekly for 90 days (restic forget policy: --keep-daily 14 --keep-weekly 13 --prune).

Verification

• Automated verification job after each backup:
  • restic check (repository integrity).
  • sample file checksum compare: upload a known test file, back up, then restic restore to temp and compare sha256sum.
  • application-level smoke test: start container/service against restored data and run basic requests.

Example verification command (restic):

# check repo integrity
restic -r s3:s3.amazonaws.com/mybucket check
# restore sample
restic -r s3:... restore latest --target /tmp/restore && sha256sum /tmp/restore/path/to/testfile

Automate restores in test environment

• CI pipeline (Jenkins/GitLab CI):
  1. Terraform apply ephemeral VM in isolated VPC.
  2. Ansible to install app and stop service.
  3. Pull snapshot (restic restore) into app data dir.
  4. Start service and run smoke tests (curl, DB checks).
  5. Tear down infra and report results.
• Schedule full restore drill monthly; weekly partial restores for critical datasets.

Metrics & monitoring

• Backup success rate (per job) and SLA compliance.
• Job duration and data transferred (bytes/sec).
• Time To Restore (TTR) measured during drills.
• Repository integrity checks frequency and failures.
• Storage growth and cost (S3 usage, Glacier transitions).
• Alerts via Prometheus-exporter or CloudWatch + Alertmanager for failures, long runtimes, missing snapshots.

Notes / trade-offs

• restic+Borg are cost-effective and secure; cloud-native (AWS Backup) simplifies management but may cost more.
• Test restores expensive but essential; automate to reduce manual errors and measure RTO/RPO.

Goal & SLAs

• RTO: 2 hours; RPO: 15 minutes (example — configurable per app)
• Primary failure: complete datacenter loss. Activate secondary region.

Trigger & Decision Points

• Auto-trigger: sustained health-check failures + ops approval
• Manual trigger: on-call confirms outage and declares DR
• Decision: full failover vs. partial (tiered apps) based on criticality

High-level automated steps

1. Failover orchestration (Terraform/CloudFormation + runbook runner)
   • Promote DR IaC stack; create VPC/subnets/security groups
   • Provision compute using pre-baked images/AMIs or VM templates
2. Networking & DNS
   • Update Transit / VPN / Peering; failover BGP if needed
   • Switch Route53/Azure Traffic Manager weights or update DNS TTL
3. Data restore (respect RPO)
   • Attach latest WAL/replication offsets or restore snapshot to nearest point (automated scripts)
   • Validate checksums and schema migrations
4. Configuration & domain rejoin
   • Run cloud-init/Ansible to set hostnames, join Active Directory/LDAP, apply GPOs
   • Re-apply secrets from vault (rotate if needed)
5. Bring up dependencies
   • Start databases, caches, message queues in dependency order with health probes
   • Reconfigure service endpoints to point to new instances
6. Application deployment & configuration
   • Deploy app code/artifacts; run post-deploy migrations and smoke tests
7. Validation & cutover
   • Automated health tests (synthetic transactions, latency/throughput checks)
   • Security checks, log ingestion, monitoring alerts suppression toggles
   • Gradual traffic shift (canary) then full cutover
8. Post-failover tasks
   • Notify stakeholders, run audit, collect recovery metrics, snapshot new state
   • Plan failback when primary is restored

Automation tooling & safeguards

• Use orchestrator (Runbooks + CI/CD pipelines), idempotent Ansible playbooks, IaC, and snapshot-based restores
• Circuit-breakers: stop if critical validation fails; escalate to humans
• Time-boxed steps tied to RTO with parallelizable tasks (network + compute concurrently)

Validation metrics

• RTO timer, RPO verification (last replicated timestamp), synthetic success rate ≥ 99%, performance baselines

This playbook balances automation, checkpoints, and human decisions to meet RTO/RPO while ensuring safe, auditable failover.