Compliance Logging

Compliance Logging

Regulatory frameworks mandate specific logging requirements -- SOC2, GDPR, HIPAA, and PCI-DSS each define what must be logged, how long logs are retained, and what constitutes auditable evidence, and failing to meet these requirements carries fines, legal liability, and loss of certification

When to Use

• Designing or auditing logging infrastructure for an organization subject to SOC2, GDPR, HIPAA, or PCI-DSS compliance requirements
• Preparing for a compliance audit and verifying that logging meets the framework's controls
• Determining log retention policies across different regulatory jurisdictions
• Deciding what to log and what must be excluded to avoid creating new compliance violations
• Building centralized log aggregation and ensuring tamper evidence for audit trails
• Resolving the tension between comprehensive logging and data minimization requirements

Threat Context

Compliance logging failures create two categories of organizational risk: regulatory penalties and undetectable breaches.

• Equifax (2017): The breach exposed 147 million records. Post-breach investigation revealed that Equifax's logging was insufficient to determine the full scope of the compromise. Inadequate audit trails extended the investigation timeline by months and contributed to a $700 million settlement.
• British Airways (2018): GDPR fine of 20 million GBP. The ICO found that BA's logging and monitoring systems failed to detect the Magecart skimming attack for over two months. Adequate logging of payment page modifications would have triggered alerts within hours.
• Anthem (2015): The largest healthcare breach at the time (78.8 million records). HIPAA audit controls require logging of all access to electronic protected health information (ePHI). Anthem's insufficient access logging made it impossible to determine exactly which records the attacker accessed, complicating notification obligations.
• GDPR data minimization paradox: Article 5(1)(c) requires data minimization -- collect only what is necessary. But Article 30 requires records of processing activities, and Article 33 requires breach notification within 72 hours (which requires sufficient logging to detect breaches). Organizations must log enough to detect breaches and prove compliance while not logging so much that the logs themselves become a liability.

Instructions

1. Map your regulatory obligations to specific logging controls. Each framework has explicit logging requirements:

   • SOC2 (Trust Services Criteria): CC6.1 requires logical access controls with logging of access events. CC7.2 requires monitoring of system components for anomalies. CC7.3 requires evaluation of security events. Logging must demonstrate that controls are operating effectively over the audit period (typically 12 months).
   • GDPR: Article 30 requires records of processing activities including purposes, categories of data subjects, and recipients. Article 5(1)(f) requires integrity and confidentiality of personal data, which demands access logging. Article 33 requires 72-hour breach notification, which is impossible without detection logging.
   • HIPAA (45 CFR 164.312(b)): Audit controls must record and examine activity in information systems that contain or use ePHI. This includes login attempts, data access, data modification, and data export. The Security Rule requires both hardware and software audit mechanisms.
   • PCI-DSS Requirement 10: Track and monitor all access to network resources and cardholder data. Requirement 10.2 specifies exact events: all individual user access to cardholder data, all actions by anyone with root/admin privileges, access to audit trails, invalid access attempts, identification and authentication events, initialization of audit logs, creation/deletion of system-level objects.

2. Implement correct retention periods. Retention is not optional -- both too short and too long create violations:

   • SOC2: Minimum 1 year retention. The audit period is typically 12 months, and auditors require logs covering the entire period.
   • PCI-DSS: Minimum 1 year retention with at least 3 months immediately available for analysis. Older logs may be in archive storage but must be retrievable.
   • HIPAA: 6 years for security documentation and audit logs. This is the longest common retention requirement and often becomes the de facto minimum for healthcare organizations.
   • GDPR: No fixed retention period -- logs must be retained only as long as necessary for the stated purpose. A retention policy must document the purpose and the deletion schedule. Retaining logs indefinitely without justification violates data minimization.

3. Define what to log and what to exclude. The core auditable events are:

   • Always log: Authentication events (login success, login failure, logout, MFA challenge), authorization decisions (access granted, access denied, privilege escalation), data access (who accessed what record, when, from where), data modification (create, update, delete with before/after values where feasible), configuration changes (security settings, user provisioning, role assignments), administrative actions (system startup, shutdown, backup, restore).
   • Never log: Passwords or password hashes, full credit card numbers (log last 4 digits only per PCI-DSS), Social Security numbers, health records content (log access to the record, not the record itself), session tokens or API keys, biometric data.
   • Mask in logs: Email addresses (log j***@example.com), IP addresses in GDPR jurisdictions (IP is personal data -- log truncated or hashed IPs for analytics, full IPs only for security investigation with documented legal basis), names (log user IDs, not full names, unless the audit requires name resolution).

4. Ensure tamper evidence for audit trails. Auditors must trust that logs have not been modified after the fact:

   • Use append-only storage (WORM -- Write Once Read Many) for compliance-critical logs. AWS S3 Object Lock, Azure Immutable Blob Storage, and GCP Bucket Lock provide cloud- native WORM storage.
   • Implement log integrity verification using hash chains or Merkle trees. Each log entry includes a hash of the previous entry, creating a tamper-evident chain.
   • Centralize log collection immediately. Logs that remain on application servers can be modified by an attacker who compromises the server. Ship logs to a centralized system (SIEM, log aggregator) within seconds of generation.
   • Restrict access to the logging infrastructure. The application should be able to write logs but not read, modify, or delete them. Separate the logging system's access controls from the application's access controls.

5. Maintain audit readiness at all times. Compliance is not a point-in-time activity:

   • Run monthly log completeness checks: are all required event types being captured? Are there gaps in the timeline? Are all services and systems sending logs?
   • Conduct quarterly mock audits: pull a sample of logs and verify they contain the required fields, are properly retained, and can be searched and correlated.
   • Package evidence proactively: maintain a running evidence collection that maps each compliance control to the specific log queries, dashboards, or reports that demonstrate the control is operating. When the auditor arrives, the evidence is ready.
   • Document the logging architecture: auditors need to understand how logs flow from source to storage, what transformations occur, what the retention policy is, and who has access.

Details

• SOC2 Type II vs Type I and the logging implications: Type I audits assess control design at a point in time -- the auditor verifies that logging controls exist. Type II audits assess control effectiveness over a period (typically 12 months) -- the auditor verifies that logging controls operated continuously and correctly throughout the period. A single month with missing logs or disabled monitoring can result in a qualified opinion or exception in the SOC2 Type II report. This means logging infrastructure must be treated with the same availability requirements as production systems.

• GDPR's legitimate interest basis for security logging: Under GDPR, processing personal data requires a legal basis. For security logging, the most common basis is Article 6(1)(f) -- legitimate interest. The organization has a legitimate interest in detecting and preventing security incidents. A Legitimate Interest Assessment (LIA) documents that the logging is necessary, proportionate, and that the individual's rights do not override the interest. This assessment should be documented and reviewed annually. Without it, logging personal data (even IP addresses) lacks a legal basis.

• PCI-DSS Requirement 10 in depth: Requirement 10.2 mandates logging specific events. Requirement 10.3 mandates specific fields in each log entry: user identification, type of event, date and time, success or failure, origination of event, identity or name of affected data/system/resource. Requirement 10.4 mandates time synchronization using NTP -- all systems must agree on the time so that log correlation across systems is accurate. Requirement 10.5 mandates securing audit trails so they cannot be altered. Requirement 10.7 mandates retaining audit trail history for at least 12 months.

• Log format standardization across services: Use a consistent structured format (JSON with a defined schema) across all services. Every log entry should include: timestamp (ISO 8601, UTC), event type (from a controlled vocabulary), actor (user ID, service account, or system), action (verb from a controlled vocabulary), resource (what was acted upon), outcome (success/failure), source IP, request ID (for correlation), and session ID. Inconsistent formats across services make audit evidence unreliable because the auditor cannot compare events across systems.

Anti-Patterns

1. Logging PII or PHI without masking. An organization implements comprehensive logging to demonstrate HIPAA compliance but includes patient names and diagnosis codes in access logs. The logs themselves now contain ePHI, requiring the same protections as the clinical database. The logging system has created a second copy of sensitive data, expanding the attack surface and the compliance scope. Log the access event (who, when, which record ID) but not the record content.

2. Retaining logs indefinitely without a documented policy. An organization stores all logs forever because "storage is cheap" and "we might need them." Under GDPR, this violates Article 5(1)(e) storage limitation -- personal data must not be kept longer than necessary. The organization now has years of IP addresses, user activity patterns, and behavioral data with no legal basis for retention. Define retention periods per regulatory requirement, implement automated deletion, and document the policy.

3. No centralized log collection. Logs remain on individual application servers, database servers, and network devices. When an auditor requests evidence, the team must manually collect logs from dozens of systems, hoping nothing was lost to log rotation or disk failures. When an attacker compromises a server, they delete the local logs to cover their tracks. Ship all logs to a centralized, access-restricted system within seconds.

4. Different logging standards across services. One service logs authentication events with user ID and timestamp. Another logs with username and local time. A third does not log authentication events at all. The auditor cannot construct a consistent picture of access patterns across the organization. Define a logging standard with required fields and event types, enforce it through shared libraries or middleware, and validate compliance in CI.

5. Treating compliance logging as a separate system from security logging. Building a dedicated "compliance log" pipeline that captures a subset of events for auditors, while the security team uses a different logging pipeline for detection and response. The compliance logs miss events the security team catches, and the security logs miss events the auditors need. Use a single logging infrastructure that serves both purposes. Apply different retention policies and access controls to the same data rather than maintaining parallel systems.