Using Apache 2.4 on AWS as a Reverse Proxy: Debugging 502s & Hardening in Practice (with vhost-scoped logs and rotatelogs)

Posted on by Rico

This post summarizes a real troubleshooting session; all company/domain details are anonymized.
Example domains use demodomain.com, e.g., wmsadmin.demodomain.com.

Architecture Overview

  • server-a: Apache 2.4 on AWS (HTTPS ingress, reverse proxy)
  • server-b: The application (WMS / API backend) on the same subnet
  • Goal: Forward external traffic for a specific hostname (e.g., wmsadmin.demodomain.com) from server-a to server-b

Typical vhost (simplified):

<VirtualHost *:443>
    ServerName wmsadmin.demodomain.com

    ProxyPreserveHost On
    ProxyRequests Off

    # Forward to the internal backend
    ProxyPass        / http://10.0.10.20:16890/ connectiontimeout=5 timeout=120 keepalive=On retry=0
    ProxyPassReverse / http://10.0.10.20:16890/

    RequestHeader set X-Forwarded-Proto expr=%{REQUEST_SCHEME}
    RequestHeader set X-Forwarded-SSL   expr=%{HTTPS}
    RequestHeader set X-Forwarded-For   "%{REMOTE_ADDR}s"

    # Some backends dislike chunked uploads; sending Content-Length is safer
    SetEnv proxy-sendcl 1

    # SSL cert directives omitted
</VirtualHost>

Symptom

  • Intermittent 502 Bad Gateway / Proxy Error
  • Root cause: large batch uploads (big files / many orders) made the backend exceed the default proxy timeout
  • Increasing Proxy timeout resolved the issue

Troubleshooting Flow (quick checklist)

  1. Time correlation: Pin the exact second when 502s occurred; cross-check server-a (Apache) and server-b (backend) logs.
  2. Test backend directly from server-a (rule out network / firewall): curl -v --max-time 180 http://10.0.10.20:16890/health
  3. Enable proxy debugging (only while reproducing): LogLevel proxy:debug proxy_http:trace5 Inspect error log for timeout, AH01102 (failed to read status line), Connection reset by peer, etc.
    ➜ After investigation, revert: LogLevel warn
  4. Docker/network (if backend runs in containers): Is the container IP stable? Prefer a user-defined network + service name to avoid IP churn on restarts.

Raise timeouts only on long-running paths

Avoid setting a huge global timeout that can tie up workers. Relax timeouts per URI:

# Regular APIs: keep short
ProxyPass        /api/ http://10.0.10.20:16890/api/ connectiontimeout=5 timeout=60 keepalive=On
ProxyPassReverse /api/ http://10.0.10.20:16890/api/

# Bulk upload / batch processing: longer timeout (e.g., 10 minutes)
ProxyPass        /orders/upload http://10.0.10.20:16890/orders/upload connectiontimeout=5 timeout=600 keepalive=On
ProxyPassReverse /orders/upload http://10.0.10.20:16890/orders/upload

If mod_reqtimeout is enabled, prevent slow uploads from being cut off:

RequestReadTimeout header=60-120,MinRate=0 body=120,MinRate=0
# Some clients stall with Expect: 100-continue; remove if needed:
# RequestHeader unset Expect early

Reading “noisy trace” lines

Lines like:

[proxy_http:trace4] mod_proxy_http.c(...) Access-Control-Allow-Methods: *
[proxy_http:trace3] start body send

appear only because LogLevel is very high. They show backend response headers and the transfer flow.
They’re not errors. To see real issues, revert to LogLevel warn, or find 502 in the access log first and then look at error-level messages around that timestamp.

Per-vhost log files (escape from other_vhosts_access.log)

Specify ErrorLog/CustomLog inside each vhost:

<VirtualHost *:443>
    ServerName wmsadmin.demodomain.com

    ErrorLog  ${APACHE_LOG_DIR}/wmsadmin-ssl-error.log
    CustomLog ${APACHE_LOG_DIR}/wmsadmin-ssl-access.log combined
    ...
</VirtualHost>

If you no longer want the global other_vhosts_access.log on Debian/Ubuntu:

a2disconf other-vhosts-access-log
apache2ctl graceful

(Note: after disabling it, any vhost without its own CustomLog will not be logged.)

No logrotate in the container? Use Apache’s built-in rotatelogs

No cron/logrotate needed—pipe logs to rotatelogs.
Binary paths vary by image:

  • Official httpd:2.4: /usr/local/apache2/bin/rotatelogs
  • Debian/Ubuntu packages: /usr/sbin/rotatelogs (from apache2-utils)

Find it:

command -v rotatelogs || find / -type f -name rotatelogs 2>/dev/null

Example (daily rotation + a “current” symlink):

ErrorLog  "|/usr/local/apache2/bin/rotatelogs -l -f -L /var/log/apache2/wmsadmin-error.log  /var/log/apache2/wmsadmin-error-%Y%m%d.log 86400"
CustomLog "|/usr/local/apache2/bin/rotatelogs -l -f -L /var/log/apache2/wmsadmin-access.log /var/log/apache2/wmsadmin-access-%Y%m%d.log 86400" combined

Prefer rotating by size? Replace 86400 with 10M, etc.

Make the access log more useful: log I/O and latency

Add combined_io to capture request/response bytes and processing time:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O %D" combined_io
CustomLog ${APACHE_LOG_DIR}/wmsadmin-access.log combined_io
  • %I / %O: request / response bytes
  • %D: processing time (microseconds)

Find requests taking >60s (example):

awk '$NF>60000000' /var/log/apache2/wmsadmin-access.log

Common details & best practices

  • ProxyPreserveHost On: many backends route/compose URLs based on Host
  • SetEnv proxy-sendcl 1: convert chunked uploads to explicit Content-Length for better compatibility
  • retry=0: prevent Apache from marking the backend as down for too long
  • keepalive=On: reduce connection setup overhead
  • Capacity: watch MaxRequestWorkers, CPU/RAM, and ulimit -n; if 502s spike only at peaks, concurrency/pools may be exhausted
  • CORS/RateLimit headers: often from the upstream; not directly related to Apache 502s—review for security posture, though

Validation & Monitoring

  • curl to simulate long requests: curl -v --max-time 900 -F "file=@big_orders.csv" https://wmsadmin.demodomain.com/orders/upload
  • mod_status for a quick view of Busy/Idle workers
  • Log-based alerts: trigger on %D (latency) or 5xx ratio
  • Product improvement: make heavy tasks asynchronous (202 Accepted + status polling/SSE/WebSocket) so proxy timeouts can stay reasonable

Final Checklist

  • Increase timeout= only for long-running paths
  • ProxyPreserveHost On, SetEnv proxy-sendcl 1
  • Loosen RequestReadTimeout if enabled
  • Per-vhost ErrorLog/CustomLog, pipe to rotatelogs
  • Use combined_io to capture %I/%O/%D
  • Enable LogLevel proxy:debug proxy_http:trace5 only while reproducing; then revert to warn
  • Test backend from server-a via curl -v
  • Watch workers/resources, backend pools, DB/GC
  • Consider async design for huge uploads/batches

If your image lacks rotatelogs, install apache2-utils (Debian/Ubuntu/Alpine name is the same) or point to the correct path. All references to real domains were anonymized to demodomain.com; you can drop these snippets straight into your environment.