Sapan Pandya, Senior Software Engineer and IEEE Senior Member – Interview Series

Sapan Pandya is a Senior Software Engineer and IEEE Senior Member based in New Jersey, with more than ten years of experience in resilient distributed transaction systems. His work spans transaction integrity, integration between software and physical devices, production troubleshooting, and deployment validation in regulated environments. He holds a master’s degree in Computer Science from Stevens Institute of Technology and contributes to scholarly review and technical research.

Pandya’s research explores kiosk cybersecurity, edge AI, and AI-assisted reliability engineering, while his production experience informs his focus on evidence, transaction outcomes, and operational accountability. In this interview, he discusses idempotency, conflicting transaction records, uncertainty in recovery systems, and the controls needed before AI agents can act on mission-critical infrastructure. He distinguishes practical engineering experience from research conducted through literature analysis and controlled or synthetic testing.

Your career has focused on software reliability and distributed transaction systems. How has that experience shaped your understanding of reliability, and what challenges have influenced your perspective on using AI to manage complex systems?

My perspective has shifted from asking whether a component works to asking whether the customer’s transaction is complete and can be accounted for.

Earlier in my career, much of my attention was on application behavior: validating input, handling responses, and making sure the interface followed the intended workflow. As my responsibilities expanded into system integration, production troubleshooting, and deployment validation, I had to look beyond individual components.

A service can finish its work while another service is still waiting for confirmation. Software can record a successful operation while a physical device fails to deliver the expected result. Passing tests for each component does not, by itself, establish that the complete interaction will behave correctly.

That experience has made me particularly interested in the boundaries between systems. What does each component know? What is it assuming? What happens when those assumptions stop being true?

It also shapes how I think about AI. There is real value in helping engineers assemble evidence, recognize patterns, and investigate failures faster. But an explanation of what probably happened is different from a verified account of what actually happened.

I want AI to reduce the effort of investigating a problem without lowering the standard of evidence required to resolve it.

You’ve emphasized that technically correct code doesn’t always guarantee correct transaction outcomes. Can you walk us through a concrete example where a transaction appears to fail because of a timeout but has actually been completed? What can go wrong when the system automatically retries the operation, and how should engineers prevent duplicate transactions?

Consider a hypothetical $50 payment. The application sends a request, the payment service records it successfully, and the response is lost before reaching the application.

The customer sees a timeout even though the payment has already happened. If the application submits a new payment request, it may charge the customer again.

A timeout tells you that the response did not arrive in time. It does not tell you whether the underlying transaction succeeded.

One common safeguard is idempotency: allowing the same logical request to be repeated without creating another financial effect. The application retains an operation identifier across retries, and the receiving service recognizes it as the original request rather than a new purchase.

The important part is not simply attaching an identifier. The service must enforce that identity and preserve enough information to distinguish a completed operation from one still in progress. An identifier cannot protect a downstream action that ignores it.

When the outcome remains uncertain, the recovery path should check the existing transaction and reconcile the relevant records, rather than immediately create a replacement.

I would also put limits on retries and define what happens when automatic recovery cannot establish the outcome. The customer needs a clear pending status and a route to resolution, not repeated charges or an endless processing screen.

In distributed systems, it’s possible for a database, backend service, and physical kiosk to report conflicting information about the same transaction. Have you encountered this type of inconsistency in production, and how was it investigated and resolved? What role, if any, could AI play in reconciling these conflicting records?

Yes, investigating transaction-state inconsistencies has been part of my production work. To explain the reasoning without discussing a workplace incident, I’ll use a hypothetical example.

Consider a generic self-service checkout. Its payment service records a successful purchase, its printer reports accepting the receipt job, and the screen then displays an error. Those observations may describe different stages of the interaction rather than prove that the entire purchase failed.

I would first reconstruct the sequence using the transaction identifier and the relevant service and device records. Then I would establish what each observation proves. A printer accepting a job is not necessarily evidence that paper reached the customer. A request being received is not necessarily evidence that its financial effect was committed.

The resolution depends on which stage failed. Repeating the entire transaction could be inappropriate when only the output stage needs attention. Where the available evidence cannot establish what happened, the affected operation should remain controlled while someone investigates.

The lasting correction also needs to address why the inconsistency was possible, rather than simply repair one record.

AI could help correlate events and identify missing information. I would ask it to present competing explanations and the evidence supporting each. It should help narrow the investigation, not resolve conflicting records by choosing whichever explanation sounds most convincing.

As AI agents evolve from diagnosing production incidents to recommending or executing recovery actions, what safeguards should be in place before an AI system can retry, reverse, or modify a transaction? How can deterministic controls such as idempotency keys, transaction logs, and reconciliation mechanisms prevent AI from introducing additional failures?

I would separate proposing an action from authorizing and executing it.

An AI agent might recommend a retry, but a separate control should determine whether that retry is permitted. It should check the transaction’s current state, the identity of the operation, the agent’s permissions, and the conditions under which that action is safe.

That check needs to apply when the action executes, not just when the model first examined the incident. Another process may have completed the transaction in the meantime.

The tools available to the agent should also be narrow. Permission to retrieve a transaction status is different from permission to reverse a payment or modify a financial record. The agent should not be able to expand its own authority because its preferred solution requires more access.

Idempotency limits duplicate effects. Logs preserve the history needed to investigate. Reconciliation checks agreement between relevant records. None of them, alone, establishes that a proposed action is appropriate.

I would also record the evidence used, the recommendation, the authorization decision, and the actual result. That makes it possible to review the intervention afterward.

These controls should be enforced by testable software outside the model. Instructions telling an agent to be careful are useful guidance, but they are not an authorization mechanism.

Consider a scenario where an AI monitoring system detects an apparent transaction failure and recommends automatically reversing the operation. If the transaction actually succeeded but the confirmation was delayed, that intervention could create a new problem. How should engineering teams design AI-driven recovery systems to recognize uncertainty and avoid taking corrective actions based on incomplete information?

The system needs a genuine “outcome unknown” state. Otherwise, it is likely to turn missing information into an incorrect failure decision.

A delayed confirmation is a reason to investigate, not automatically a reason to reverse a transaction. A reversal is another financial operation with its own consequences. It needs an independently established reason and the appropriate authorization.

I would separate the response into two paths. The investigative path can retrieve status, collect missing events, compare records, and notify the responsible team. The financial-action path remains blocked until its conditions are satisfied.

This prevents an AI confidence score from becoming a substitute for transaction evidence. A model can be highly confident in an explanation that was built from incomplete information.

There also needs to be a defined escalation point. Keeping a transaction pending indefinitely is not a satisfactory outcome for the customer. Someone should own the unresolved case, with enough context to determine the next safe step.

The recovery process should therefore be able to say both “I cannot yet justify changing this transaction” and “Here is what needs to happen to resolve it.”

Recognizing uncertainty is useful only when the system has a practical way to manage it.

One of the biggest questions surrounding autonomous AI is determining when human approval remains essential. Which production recovery actions do you believe AI should be permitted to execute independently, and which should always require human authorization? How should organizations establish these boundaries when financial transactions, regulatory compliance, or customer trust are at stake?

I would draw the boundary around the consequences of being wrong, rather than around whether an action is labeled “AI.”

Read-only investigation is a reasonable starting point, provided the agent can access only authorized information. It can gather diagnostic records, group related alerts, and prepare an incident summary without being allowed to change the underlying transaction.

Some predefined recovery procedures may also be suitable for automation when their conditions are independently verified and their effects are limited.

I would require human authorization when a proposed intervention falls outside an approved procedure, when records conflict and the financial outcome remains unresolved, or when the action would change security controls or the agent’s own permissions. Those are decisions an agent should not authorize for itself.

That does not mean every routine financial operation requires a person to approve it. Established rules-based processing is different from allowing a model to invent a correction during an incident.

Human review must also be meaningful. An approval screen should show the supporting evidence, remaining uncertainty, expected effect, and available alternatives. Asking someone to approve a confident paragraph without that context adds little protection.

These boundaries should be agreed on before deployment. An outage is the wrong moment to decide how much authority an automated system ought to have.

You’ve worked extensively on fault-tolerant backend systems, observability tooling, and integrations between software services and physical hardware. What kinds of production telemetry and evidence would an AI agent need to reliably diagnose an incident? How can engineering teams distinguish between useful AI-generated recommendations and plausible but incorrect conclusions that could worsen an outage?

I would start with evidence that connects the customer’s operation across the systems involved: transaction identifiers, service events, durable state changes, device responses, and communication failures.

Software versions and configuration changes matter too. A failure that begins after a release gives an investigator a useful lead, but timing alone does not prove causation.

The agent also needs to understand the limits of the evidence. Logs can be incomplete, records can arrive late, and clocks may disagree. An absent acknowledgment is not the same as proof that no operation occurred.

A useful diagnosis should separate observations from interpretations. “The client recorded a timeout” is an observation. “The backend was unavailable” is an explanation that requires further support.

I would expect a recommendation to identify the records behind it, explain what remains unknown, and specify what additional check would confirm or disprove its conclusion.

Engineers should then verify consequential recommendations against the original records rather than rely only on the agent’s summary. That is especially important when a summary compresses a long or contradictory event history.

The standard I would use is straightforward: can another engineer follow the evidence and reach a defensible conclusion? If the explanation cannot survive that check, it is not ready to drive a production change.

In your published research on ransomware resilience for lottery and betting kiosks, you evaluated an AI-assisted security framework using a synthetic environment simulating 500 kiosks. What were the most important findings, and what limitations should readers understand when interpreting these results? What additional validation would be necessary before applying the proposed architecture to a live production environment?

I wanted to examine resilience beyond detection: how a terminal environment could limit an attack, recognize it, and recover while preserving evidence of what happened.

The resulting Resilient Security Framework combined preventive controls, AI-assisted anomaly detection, and defined response and recovery mechanisms. The AI component concerned detecting suspicious behavior. This was not an experiment in allowing a language-model agent to independently modify financial transactions.

The evaluation used a synthetic environment representing 500 kiosks over a 72-hour simulation, with behavioral emulations of ransomware attacks rather than a deployment across 500 physical terminals.

Against the study’s antivirus-only baseline, mean detection time fell from 52 to 31 minutes, and mean recovery time fell from 120 to 56 minutes.

Those are results for the combined framework under the experimental conditions. They should not be attributed to AI alone or treated as performance guarantees for a live environment. One useful finding was that earlier detection did not, by itself, deliver the same recovery improvement. Containment and recovery design mattered as well.

The limitations are important. The simulated workloads, network conditions, attack behaviors, and baseline defenses do not represent every operational environment. Different devices and applications would require new validation.

Before production use, I would want representative hardware testing, independent security assessment, longer-running evaluations, and careful measurement of false alarms and processing overhead.

I would also test the recovery process against legitimate transaction activity. Restoring a working software image is not enough if the process loses the information needed to account for completed transactions.

The study provides a basis for further testing. Demonstrating safe behavior in a real operational environment is a separate step.

Your perspective combines years of production engineering with academic research into software reliability and AI-driven security. Can you distinguish between the reliability mechanisms you’ve personally implemented or operated in live production systems and the approaches you’ve explored through research or simulated testing? Which AI-driven recovery capabilities do you believe are already practical today, and which remain largely experimental?

My production work has included software development and integration, transaction validation, troubleshooting, testing, and release validation.

My published research is a separate part of my work, and each study needs to be understood according to its stated methods and limitations.

The ransomware paper evaluates a proposed security framework in a synthetic test environment. My edge-AI paper is a literature-based analysis of on-device intelligence and its possible application to transaction terminals. My IEEE FeedForward article examines an AI-assisted reliability architecture through controlled simulation, including monitoring, escalation, and fallback behavior.

None of those publications should be read as evidence that I deployed an autonomous AI agent with authority over live financial transactions.

For practical use, I would prioritize assistance that engineers can inspect: incident summaries linked to source records, correlations across alerts, and suggested diagnostic checks. Their usefulness still needs to be measured in the environment where they will operate.

The harder problem is open-ended recovery, where a model chooses and executes changes under conditions that were not anticipated. That needs a much higher standard of evidence than a useful diagnostic suggestion.

I see my production experience and research as complementary: one establishes the operational problems I understand, while the other explores approaches that still have to earn their place through validation.

Looking ahead, as AI agents become more capable of detecting anomalies, diagnosing failures, and coordinating recovery across distributed systems, how do you envision the relationship between autonomous systems and human engineers evolving? What technical safeguards, operational evidence, and accountability mechanisms would be necessary before organizations could confidently grant AI greater authority over mission-critical production infrastructure?

I expect AI to take on more of the work involved in assembling an incident picture. Engineers could begin with a useful timeline, affected components, competing explanations, and missing evidence instead of spending so much time collecting that material manually.

That would give people more time for decisions that require understanding the system’s obligations and the consequences of changing it.

I would expand execution authority gradually. An agent could first operate in shadow mode, producing recommendations without acting on them. Teams could compare those recommendations with reviewed outcomes and test how the agent behaves when information is missing, contradictory, or outdated.

The evaluation should include incorrect interventions, missed problems, unnecessary escalation, and recovery from the agent’s own mistakes. A fast recommendation is not valuable if it sends the response team in the wrong direction.

Greater authority should also remain conditional. A model update or major system change may invalidate earlier testing. Teams need a way to reduce or withdraw the agent’s permissions when its behavior no longer meets the required standard.

Human responsibility does not disappear as automation improves. Someone must own the operating rules, the evidence required for a change, and the process for investigating failures.

I see engineers spending less time gathering information and more time defining, testing, and maintaining the conditions under which automated action is justified.

Thank you for the great interview, readers who wish to follow his work on distributed transaction reliability and responsible AI-assisted recovery can connect with Sapan Pandya.