Imitation Learning for Autonomous Driving in CARLA: Offline Cloning Goes Closed-Loop

A careful result, not a self-driving breakthrough

The current story around Imitation Learning for Autonomous Driving in CARLA is a newly submitted technical report by Jordy Kieto, posted to arXiv on September 15, 2026, under computer science artificial intelligence and robotics [1]. The paper asks a focused question: how much closed-loop driving competence can a compact multimodal policy learn from offline expert demonstrations in the CARLA simulator [1]. That framing matters because behavioral cloning is trained as supervised learning on expert data, while deployment is autoregressive: every predicted action changes the next observation the model receives [2].

The headline finding is encouraging but bounded. The released policy reportedly drives autonomously for hours on training and held-out routes in CARLA, with no collisions in the author’s runs, and it qualitatively transfers to an unseen CARLA town with different road geometry [1]. The report also says the policy recovered from large trajectory deviations, while explicitly declining to claim systematic recovery without controlled evaluation [1]. In other words, the new result is best read as a reproducible simulation experiment showing that a small multimodal behavioral-cloning stack can produce stable closed-loop behavior under simplified conditions, not as proof that offline cloning alone is sufficient for real-world autonomous vehicles.

What the model actually sees and predicts

The system is intentionally compact. The arXiv abstract says the policy uses five-frame histories of RGB images, LiDAR, vehicle telemetry and lane waypoints to predict throttle, brake and steering at 20 Hz [1]. The paper’s PDF specifies a 1.36 million-parameter policy that fuses those modalities through per-modality encoders and a two-layer temporal Transformer [2]. Its input history covers five frames, or about 0.25 seconds at the 20 Hz control rate [2].

The accompanying repository gives a practical view of the architecture. RGB is resized to 96 by 96, LiDAR is represented as a 64 by 64 bird’s-eye-view histogram, telemetry contains five vehicle signals, and waypoints describe the next 25 lane-center points [3]. The model outputs a three-value continuous control vector: throttle, brake and steering [3]. The repository also states that past control commands are not part of the released checkpoint’s input, which keeps the policy’s interface closer to perception, ego-state and route-context fusion than to action-history smoothing [3].

The engineering choice is notable because it avoids a giant end-to-end stack. This is not a foundation model, a vision-language-action driver or a full modular autonomy system. It is closer to a disciplined experiment: given multimodal observations and expert controls, train a supervised policy and then ask whether it remains stable when its own actions close the loop.

The data: small, structured and route-aware

The project’s central contribution may be less the network than the demonstration distribution. The paper reports that demonstrations were collected in three stages, ending with a systematic route-generation procedure that enumerates spawn points, checks feasible maneuvers from the road topology and keeps autopilot rollouts only after verifying the maneuver actually executed [1]. The released policy was trained on 236,882 windows from 448 captures, corresponding to about 3.3 hours of driving [1].

The GitHub README provides the same headline numbers and identifies the training towns as Town01 and Town02 [3]. It also says the dataset is highly imbalanced: 89.5 percent straight, 5.9 percent left and 4.7 percent right [3]. That imbalance is important because many closed-loop driving failures happen at distribution edges: turns, junctions, recovery states, or ambiguous route choices. The report’s route generator tries to address this by deliberately covering turns and straight segments, but the final sampled windows still reflect how much ordinary driving is straight-line behavior [3].

The claim audit adds provenance. It classifies the training-set size, direction mix and approximate 3.3-hour duration as quantitatively measured, recoverable from logs, manifests or checkpoint artifacts [4]. It also records that the released policy is the epoch-10 checkpoint of a training run named “barzksni,” with checkpoint timing tied to logs and metadata [4]. This is unusually helpful because it reduces ambiguity about which trained artifact produced the reported behavior.

Why closed-loop is the right test

The paper repeatedly distinguishes offline prediction metrics from closed-loop driving competence. Offline validation loss measures how well a model imitates expert actions on expert-visited states; closed-loop testing measures what happens after the model’s own errors begin changing the scene it sees [2]. The report frames this as the standard covariate-shift problem in behavioral cloning: a small mistake can push the vehicle into states that were not well represented in the demonstrations [2].

That distinction is especially relevant for autonomous driving. A model can achieve low action error on logged expert trajectories and still drift out of lane when deployed, because the logged data may not contain the recovery states created by its own imperfect steering. The report’s core claim is therefore not simply “the model predicts expert controls,” but “the model remains usable when its predictions are applied back into CARLA at 20 Hz.” The repository’s usage instructions support this closed-loop focus by including a driving script that can run the trained policy in CARLA and toggle it during simulation [3].

The result is promising because the policy appears stable despite being trained offline and despite its modest size. But it is also limited because the strongest closed-loop statements are not presented as a standardized benchmark with formal route-completion, infraction and confidence-interval reporting. The paper itself says it reports offline metrics while distinguishing measured results from qualitative closed-loop observations [1].

Reproducibility is the strongest part of the release

For researchers, the release may be more valuable than the headline performance. The arXiv page says the author released code, a trained checkpoint, an ONNX model, a data sample and an evidence audit [1]. The repository describes a full pipeline: record demonstrations manually or with CARLA autopilot, preprocess recordings into shards and a window index, train the policy, and drive with the trained policy in CARLA [3]. It also includes both Keras and ONNX versions of the trained policy [3].

ONNX export is a practical detail, not just a packaging flourish. The README says the ONNX file is the same network as the Keras checkpoint and reports a maximum output difference of about 3e-7 versus Keras [3]. That kind of parity check helps other researchers reproduce inference without being locked to one framework. The repository also includes a three-second sample of processed data and a windows manifest for quick experimentation without first collecting a full CARLA dataset [3].

The commit history visible at the repository’s current head shows the README was updated to feature the technical report, add paper highlights and include a citation block [5]. That confirms the public code release and paper presentation are aligned in the current state of the project [5].

Limitations the authors do not hide

The most important limitation is that the simulation setting is simplified. The repository says the project is simulation-only, with no other vehicles or pedestrians by default, and that the collection autopilot ignores traffic lights and stop signs [3]. That means the reported closed-loop competence should not be interpreted as urban driving readiness. It is lane following and maneuver execution in a controlled simulator setup, not interaction-rich autonomy.

There are also data and perception caveats. The README says validation windows come from later contiguous parts of the same recordings rather than held-out routes or maps, unless users explicitly hold out tags [3]. It notes that waypoints follow the lane rather than a planned route, so they do not always disambiguate which turn the expert takes at junctions [3]. It also discloses that debug waypoints appear in the camera images during collection and policy mode, which could create a visual cue that is not representative of real-world sensing [3].

The claim audit reinforces this caution by separating measured claims from qualitatively observed claims and unverified statements [4]. That structure is important editorially: the work is strongest where it points to recoverable artifacts, and weaker where it relies on author observation or video evidence rather than controlled evaluation [4].

What it means for real-world autonomous systems

The subject context says the work demonstrates potential for real-world autonomous vehicle systems using imitation learning. That potential is real, but indirect. The project shows that a compact policy can combine camera, LiDAR, telemetry and local route geometry, learn from expert traces, and survive closed-loop deployment in CARLA for extended simulation runs [1]. It also shows that careful demonstration generation may matter as much as model size [2].

However, moving from CARLA to public roads would require a much broader evidence stack: dynamic actors, traffic-rule compliance, weather and lighting variation, sensor calibration, fail-safe behavior, formal route benchmarks, ablations, uncertainty handling and real-vehicle validation. The current release does not claim those results; in fact, its limitation notes make clear that it does not include traffic, pedestrians or traffic-light obedience by default [3].

The most credible takeaway is therefore methodological. Offline behavioral cloning is often dismissed as too fragile for closed-loop control, but this project shows that with multimodal inputs, temporal context and structured data collection, even a compact model can display useful closed-loop behavior in simulation. Its bigger contribution may be the evidence-aware template: release the code, release the checkpoint, release the ONNX export, disclose quirks and grade claims by evidence. That is a healthier way to report autonomous-driving learning experiments than relying on polished demo videos alone.

The bottom line

As of the current 72-hour publication window, Imitation Learning for Autonomous Driving in CARLA is a narrow but valuable result. It does not settle the autonomous-driving problem, and it does not prove real-world deployment readiness. It does show that offline imitation learning, when paired with multimodal sensing and systematic route generation, can produce surprisingly durable closed-loop behavior in CARLA [1]. Just as importantly, it gives the community enough artifacts to inspect, rerun and challenge the claim [3].

Developments

1. Imitation Learning Enhances Autonomous Driving in CARLAArXiv - Artificial Intelligence · Sep 17, 2026, 4:00 AM UTC · 8/10

Sources from the last 72 hours

1. [1][2609.17757] Imitation Learning for Autonomous Driving in CARLASep 15, 2026, 7:08 PM UTC
2. [2]GitHub - JordyKieto/carla-imitation-learningSep 17, 2026, 12:00 AM UTC
3. [3]README: feature the technical report · JordyKieto/carla-imitation-learning@50d28e2 · GitHubSep 17, 2026, 12:00 AM UTC
4. [4]Imitation Learning for Autonomous Driving in CARLASep 15, 2026, 7:08 PM UTC
5. [5]carla-imitation-learning/paper/CLAIMS.md at main · JordyKieto/carla-imitation-learning · GitHubSep 17, 2026, 12:00 AM UTC

AI-generated article based on recent web research, then preserved as a dated editorial snapshot.