While machine learning applications in IoT devices are getting more widespread, the computational and power limitations of these devices pose a great challenge. To handle this increasing computational burden, edge, and cloud solutions emerge as a means to offload computation to more powerful devices. However, the unstable nature of network connections constantly changes the communication costs, making the offload process (i.e., when and where to transfer data) a dynamic trade-off. In this work, we propose DECOS: a framework to automatically select at run-time the best offloading solution with minimum latency based on the computational capabilities of devices and network status at a given moment. We use heterogeneous devices for edge and Cloud nodes to evaluate the framework's performance using MobileNetV1 CNN and network traffic data from a real-world 4G bandwidth dataset. DECOS effectively selects the best processing node to maintain the minimum possible latency, reducing it up to 29% compared to Cloud-exclusive processing while reducing the energy consumption by 1.9times compared to IoT-exclusive execution. © 2023 IEEE.