Replacing AD with Samba: Can OpenLDAP Be the Backend? Practical Feasibility, Options & Migration Guide

Samba AD DC cannot use OpenLDAP as its backend (Samba 4’s AD relies on LDB/sam.ldb).
Do you need a domain? Not always. If you only need file-share ACLs and already manage endpoints via ipguard, you can run without a domain—but RDS/Terminal Services and long-term ACL hygiene will suffer.
The most pragmatic path: deploy a minimal Samba AD (you can skip GPOs) as identity/Kerberos, join your file servers and RDS to the domain; keep OpenLDAP as a sidecar mirror for legacy apps via one-way sync if needed.

Your Context

Goal: file-sharing access control; endpoints are governed by ipguard (no GPO).
You still run Microsoft Terminal Services (RDS).
Question: Can we skip a domain and authenticate with OpenLDAP only?

Not supported: Samba AD DC cannot store its directory in OpenLDAP. AD semantics (Kerberos, replication, ACLs, schema) live in sam.ldb (LDB).
Supported: expose AD’s LDAP for apps, or export attributes from AD into OpenLDAP for legacy consumers (mirror/sidecar, not a backend).

Running without a domain still works for SMB and RDS, but you’ll face:

Fragmented identities: local accounts per server or homegrown sync; weak central password policy.
Brittle ACLs: SMB ACLs bind to local SIDs; migrations/new servers easily break permissions.
RDS limitations: per-user CAL tracking, SSO, RemoteApp, Gateway, brokered sessions are clunky or manual.
No Kerberos SSO: users keep re-typing passwords across services.

Minimal domain (Samba AD) advantages—even with no GPOs:

Single Identity & SSO (Kerberos) across SMB/RDS/internal services.
Stable ACLs with domain groups; migrations/scale-out won’t explode.
RDS behaves properly: licensing tracking, RemoteApp, Gateway, Connection Broker, home folders.
Security & audit: central lockout/expiry; easier future MFA/cloud integration.
Lower long-term TCO: new servers/services reuse the same identity fabric.

Use cases: Windows domain join, Kerberos, SMB ACLs, RDS, (optional) GPOs.
Linux: provision with --use-rfc2307; clients use sssd/winbind.
Caveat: not supported with on-prem Exchange / AAD Connect hybrid; either keep a minimal Windows AD for that, or go cloud-only.

Use case: legacy apps tied to OpenLDAP.
Approach: one-way attribute sync from AD to OpenLDAP (LSC or custom exporter).
Goal: migrate apps to query AD directly later; retire OpenLDAP.

Use case: tiny, static environments.
Limitations: ACL hygiene, RDS licensing/experience, SSO/scale all suffer; long-term cost often higher than expected.

Scenario	Recommendation
1 file server + 1 RDS, almost no growth	You can skip the domain (C), but accept manual identity/ACL sync overhead
≥2 file servers/RDS, frequent changes or growth	Deploy a minimal Samba AD (A); you can still ignore GPOs
Legacy apps need OpenLDAP	Choose A + B: Samba AD primary, one-way sync to OpenLDAP as a bridge

Consistent naming for users/groups; script replication across servers.
Group-based ACLs only; avoid user-direct assignments.
Password sync plan between OpenLDAP and Samba (or strict change workflow).
RDS licensing: prefer per-device CALs in workgroup setups.
Migration/backup SOP: record machine SIDs, account maps, ACLs; rehearse restores.

Spin up two Samba AD DCs (--use-rfc2307 --dns-backend=SAMBA_INTERNAL), add the second DC, set up SYSVOL rsync.
Join one Windows client and one Linux client (sssd) to the domain; validate Kerberos and SMB permissions.
Stage a parallel file server using domain groups for ACLs; compare migration effort vs. your current approach.
If OpenLDAP is required, test one-way sync of 3–5 core attributes to serve a legacy app.

To reduce identity/ACL toil and make RDS/SMB future-proof, adopt a minimal Samba AD (A) today—you don’t have to roll out GPOs.
Keep OpenLDAP only as a temporary mirror (B) where legacy constraints exist.
Consider no domain (C) only for very small, static environments where manual overhead is acceptable.