Today, enterprises can deploy small cell networks (e.g. Wi-Fi or sophisticated 4G cellular networks based on LTE technology) to go completely wireless, not only for their staff with the latest smartphones and tablets, but also to connect any of their assets, sensors, goods etc. to the Cloud. In addition, enterprises which go after the Internet of Things use IoT gateways as part of a tiered system architecture. Finally, the same enterprises may discover the benefits of edge cloud computing and consider the deployment of a cloudlet. The only issue: all needs to be enterprise-grade. Secure, reliable, with guaranteed quality of service levels. How does this relate to an enterprise micro-datacenter? And what does this small datacenter actually host? Let’s see.

Three distinct trends to watch
We want to point out three trends that are worth to watch: i) small cell network technology and its virtualisation, ii) the evolution of IoT gateways for the Internet of Things (IoT) and iii) the development of Edge (Cloud) Computing technology.
One reason we are interested in these three technology areas is related to economics. 4G cellular small cells, IoT gateways for the industrial Internet of Things and edge computing servers all constitute CAPEX and entail OPEX (e.g. related to management of deployed infrastructure and software). All three technologies can be deployed for a variety of use cases in a number of different ways, from indoor to outdoor, as physical infrastructure boxes or virtualised functions. However, the question is: Are there situations where joint deployment suddenly creates significant cost synergies and extra benefits?
There are some interesting data points that lead us to focus on the enterprise market. Let’s see why.

Virtualisation of small cell networks
This is the first trend we are looking at. With small cell networks we refer here to wireless small cell deployments based e.g. on Wi-Fi and in particular LTE (4G) cellular technology (see [23] for an overview of small cell technologies for enterprises). LTE small cells have already been deployed into homes in form of femto cells. However, the market for slightly bigger, more powerful, and more sophisticated small cells like pico cells for small to medium enterprises and micro cells to cover large office buildings and campuses is growing at increased speed. The Small Cell Forum has forecast that the enterprise market is going to overtake the residential market in size by 2020 [22].

A single small cell system has most often had the appearance of a box (either grey or white) with the box mounted to a ceiling, wall, lamp post, bus station etc. 3GPP has defined how such an access point can be integrated with a cellular network. As it is not very economic and resource efficient to integrate new and numerous access points one by one with the network of a mobile network operator, the concept of an Enterprise Small Cell Concentrator (E-SCC) was born [24]. This is another box that gathers traffic from a number of access points and acts towards the cellular core network as a single ‘super small cell’. In addition, it serves as a mobility anchor, hiding device mobility events that occur within a cluster of access points from the network operators’ core network (as the latter shouldn’t need to bother about them).

So far, traffic from access points (e.g. in an enterprise office) would generally be routed to an operator’s core network components (like Security Gateway, 3GPP Small Cell Gateway and Serving Gateway) possibly miles away. To support local breakout of traffic within the confines of an enterprise, a further function was introduced, called the Enterprise Small Cell Gateway (E-SCG) [24]. The E-SCG enables local breakout of traffic into an Intranet and local integration with PBX systems for unified communication.

Thus, a rather complete small cell system may consist of say 30 access points spread across a building, an E-SCC and an E-SCG. A physical deployment may consist of those 30 physical access points and a single small cell controller that combines the functions of E-SCC and E-SCG [6], [7]. In an enterprise environment, both the small cell access points and the small cell controller would be hosted on-premises with the former spread across a building whilst the latter may reside e.g. in a DMZ within an IT room.

The introduction of small cell controllers enables useful service features like indoor location, geo-fencing, zonal presence and others. For more details see [26].

However, more interesting for us here is to point out a trend towards virtualisation of the small cell controller platforms [21]. Borrowing the concepts from ETSI ISG NFV [4], the different functions one may find on a physical small cell controller like E-SCC or E-SCG can all be considered virtualised network functions (VNFs). Example VNFs include: IPSec gateway, centralised small cell virtualised function (which may include small cell cluster control, an authentication server, firewall, API exposure, etc.), small cell concentrator function (a la E-SCC) and small cell gateway function (a la E-SCG) (for details, see [25, 27, 28].

Thus, to summarise the trend here:
Small cell deployments become increasingly interesting for enterprises. Small cell control infrastructure hosted in enterprise premises gets virtualised, such that it can run on a kind of NFV infrastructure (NFVI) on commodity off-the-shelf hardware (COTS). This reduces cost and increases flexibility. Mechanisms have been developed that allow keeping local enterprise traffic physically local without looping it through a cellular network operator’s core network. This has benefits related to quality of service, privacy and others.

IoT gateways
An IoT gateway is a type of infrastructure that helps to interface IoT end nodes to IoT service platforms and domain-specific applications. The notion of Edge Gateway or IoT Gateway is documented in the reference architecture of the Industrial Internet Consortium IIC [30] and we use the two terms interchangeably here. The architecture of the IIC is split into three tiers:

The Edge Tier consists of things of the Internet of Things (e.g. an industrial robot, a remotely programmable and controllable asthma inhaler) and a proximity network through which end nodes of the IoT (like those robots or new-age asthma inhalers) are connected to an Edge Gateway. The nature of the proximity network will vary considerably depending on the type of environment (e.g. hospital versus manufacturing plant). The network technology used in the proximity network may range from wired serial bus (for connecting machines) to short-range radio (e.g. ZigBee) to LTE small cell networks (say LTE access points and a small cell controller). However, communication with an external service platform (e.g. for analytics, asset management, device management etc.) will pass through the Edge Gateway. The Edge Gateway itself may be connected to the external service platform via a wide area network (WAN, cellular or fixed).

The Platform Tier is made up by the service platforms mentioned above. These functions tend to be independent of the nature of the business, the ‘domain’.

The Enterprise Tier bundles together all the functions that are domain specific and are required for the overall delivery of an IoT application or service. One can imagine different vertical industry sectors constituting different domains with domain-specific business logic (operating a wind farm will be different from running a smart metering system or an automated industrial assembly plant for assembling the next generation of robots).

An Edge Gateway is a bridge between the Edge Tier and the Platform Tier. IoT Gateway functions may cater for a wide range of tasks:

• Directly controlling and managing connected devices and artefacts of the Internet of Things (e.g. command and control for connected sensors and actuators). In several cases, one can imagine that due to latency constraints, this command and control needs to sit in close proximity of sensors and actuators.
• Acting as an agent “in the middle” to efficiently enable remote management of IoT endpoints over the wide area network.
• Hosting a data processing pipeline, where data is gathered from IoT nodes, (temporarily) stored, aggregated, transformed, filtered, subjected to policies etc. before any data might be sent to systems of the Platform Tier and/or Enterprise Tier.
• Acting as a communications hub that interconnects IoT endpoints and also connects them to the wide area network and cloud computing centers. The hub may perform protocol translations (e.g. http vs CoAP). The hub may further aggregate communication to the Cloud (where e.g. functions of the service platform and the domain-specific business logic might reside).
• Implementing security and enforcing policies. At the IoT Gateway, enterprise deep packet inspection can be realised, encryption/decryption can be performed [11], data can be redacted and privacy policies enforced before data is released into the WAN (an example would be denaturing of video streams from security cameras to meet certain privacy policies [15]).
• Hosting operational analytics and real-time analytics in close proximity of devices of the IoT, such that depending on results, immediate corrective action can be taken. This complements big data/batch analytics undertaken in the service platform across the WAN [20].

An interesting example of a distributed IoT computing architecture is General Electric’s Predix, where the Predix Machine software is deployed between devices and the Predix main Cloud [19, 20].
IoT gateways equally benefit from a virtualised infrastructure that is able to run on commodity off-the-shelf hardware (COTS).

To summarise:
IoT gateways can be found as part of deployment architectures of the Internet of Things. They play an important role in connecting IoT nodes amongst themselves and to service platforms and business logic. Thus production data traffic is passing through those IoT gateways and is there often acted upon in a data processing pipeline or in a component of a distributed IoT application.
Like enterprise small cell controllers, IoT gateways are often deployed within the confines of an enterprise or in proximity of the IoT devices. With the size of the Industrial Internet market forecast to outpace the one of Enterprise Cloud Computing and Consumer IoT by 2020 [29], we will likely see more deployments of IoT gateways in the enterprise market.

Edge Computing
As discussed in other places [1, 2, 3], edge computing and edge cloud computing refer to computation (and storage and networking) in proximity of devices in contrast to having e.g. the necessary computation solely deployed in a rather remote datacenter or private/public/hybrid Cloud. If we take the example of mobile edge computing (MEC), then the locations where “edge computing” may take place could e.g. be: at a base station or small cell access point (e.g. LTE macro or micro base station), at a node where baseband processing is implemented (e.g. vBBU), at a traffic aggregation node in the access transmission network (e.g. an aggregation router with a virtualisation environment) or of course in a telco datacenter.

A distributed IoT application can be deployed in multiple ways, e.g. in a three-tiered form: i) The first application part on an IoT node, the second application part on an edge computing platform and the 3rd application (backend) part in a private cloud.

Edge computing is of particular interest when it helps

• to offload heavy compute tasks from constrained devices (e.g. surveillance drones, smart helmets, smart glasses, body-worn cameras, augmented and mixed reality devices) thereby reducing weight, increasing battery lifetime and thus usability.
• to achieve very low latency for the roundtrip from IoT device to application code at the edge and back to the device (e.g. for closed loop control, tactile Internet applications, cognitive assistance services or safety-critical autonomous driving features).
• to avoid exposure of unnecessary data. Computation at the edge makes also sense when it is all about processing mostly local content that doesn’t need to be shared with other functions deeper in the network or in a Cloud, thereby saving cloud computing resources and backhaul bandwidth to the Cloud.
• to extract local context and share that with application code at the edge that can benefit from it (as e.g. done with Mobile Throughout Guidance for video servers).

Edge computing can be enriched by special platform services which are themselves hosted at the edge. Such platform (or middleware) software can expose specialist services to the applications hosted at the edge through APIs. An example is the Radio Network Information Services RNIS as defined in ETSI ISG MEC [5]. RNIS extracts real-time context information from a cellular access network and makes that context available to application software that is hosted at a MEC server via the RNIS API.

Important for our discussion here is that edge computing servers (a la ETSI ISG MEC) are envisaged to run on a virtualised computing platform. A MEC application running on a MEC server can act as an endpoint for a data connection from a device or it can locally (at the edge) act upon the data traffic before passing it through into the operator’s network. Thus, if we consider MEC in the context of a 4G cellular network, then data traffic needs to be ‘intercepted’ and routed to MEC applications that are hosted on a MEC server. 3GPP has already solved this challenge and two mechanisms are generally available for this: LIPA (Local IP Access) and SIPTO (Selective IP Traffic Offload).

Several vendors have implemented edge computing systems using SIPTO as the traffic offload mechanism.
For network operators, go-to-market for edge computing first in the enterprise sector might be a lower-risk strategy compared to attempting nation-wide coverage with fine-granular edge computing capabilities from day one.

To summarise:
Applications running on edge computing servers (including MEC servers) may constitute communication end points for data connections with devices or they may act upon data packets that pass through edge servers. Many proposals suggest that the edge computing systems be realised using some virtualisation infrastructure. They can be deployed in both fixed and wireless environments and the exact deployment location (i.e. the ‘edge location’) will vary depending on the specific use cases. Deployment for selected enterprise use cases may be easier in the short-term than regional/nation-wide deployment of edge computing capabilities.

Where things come together
Having briefly discussed above three items (small cell networks, IoT gateways and edge computing servers), we now hypothesise that the three (so far rather independent technology areas) can possibly benefit from each other and help to lower go-to-market hurdles. Here several observations which apply to the deployment architectures of some or all of the three components: small cell networks, IoT gateways of distributed IoT systems and edge computing nodes.

1. All three have a role to play in the enterprise market (in addition of course to possibly other markets): e.g. LTE small cell deployments with numerous access points and small cell controllers, IoT gateways in particular for the industrial Internet of Things and Edge Computing servers where they shall support bandwidth-hungry applications (avoiding bandwidth bottlenecks on the path to cloud computing centers), low-latency applications or privacy-sensitive applications, possibly enriched with local radio access network context.

2. All three technologies adopt the concept of virtualisation for the sake of benefiting from lower cost of COTS hardware and much higher flexibility and agility regarding deployment of software on top of such hardware. Virtualisation typically comes along with the use of hypervisors, use of virtual machines or some other containerization technology (like Linux containers, Docker [14] or Unikernels [12, 13]).

3. Key functions can be located at the same ‘edge location’, exactly on the same spot. In other words, the hosting location of a virtualised small cell controller, mobile edge computing platform and virtualised IoT gateway can actually coincide. Consider the example of a manufacturing plant: Across the plant, LTE (or in the future 5G) small cell access points are deployed. All access points are connected to a small cell controller function that is hosted on a virtualised infrastructure and COTS hardware in an IT room at the plant. In addition, the plant has also deployed IoT gateway software onto the same COTS hardware (e.g. a data processing pipeline and an operational analytics engine). And, to support high-reliability IoT cloud applications that have been designed in a three-tiered way and that shall benefit from an ultra-low latency data loop between production line and application, an edge computing platform that exposes via an API contextual radio cell information in real-time to the IoT application has been deployed to the same virtualised infrastructure in the same IT room. This common infrastructure is essentially a near-edge micro-datacenter which is managed through some single virtual infrastructure management software for the sake of cost efficiency and simplicity.

4. Where on-premises systems are architecturally knitted together with the 3GPP-type core network infrastructure of say a cellular network operator, local traffic offload is achieved through same or similar principles. Here this would be the case for the LTE small cell deployment and the mobile edge computing platform that either terminates LTE traffic or acts upon it in pass-through mode. Recall: What traffic might want to be better kept local within the enterprise instead of sending it through the network operator’s network: local communication between LTE end user cellular devices and the enterprise PBX system (achieved through the E-SCG function), traffic between LTE end user devices and the local Intranet (again, via the E-SCG), data traffic between local IoT nodes (as passing through the IoT gateway), data traffic between end nodes and the edge computing platform.
Today, both enterprise small cell gateways (E-SCG) and MEC platforms make use of traffic offload mechanisms as defined by 3GPP (e.g. SIPTO).

5. In an IoT enterprise environment, all three technology areas (small cell networks, IoT gateways and edge computing servers) have to meet high expectations regarding Quality of Service, security, reliability, availability and manageability. In this respect, they all might benefit from common solutions and common shared mechanisms. Examples of shared mechanisms are:

• Software probes and platform analytics to assess and predict the health and load status of all platform services. Monitoring of metrics and KPIs in order to achieve service level agreements [17].
• Event logging infrastructure [18] to support cloud-native application design and microservices.
• Traffic classifiers, traffic routing, virtual switches, DNS, DHCP servers.
• Caching.
• Service function chaining mechanisms at the edge.
• API gateways for client access to many microservices [16].
• Support for different and preferred ways of containerization (virtual machines, Docker containers, Unikernels [13], and all related tools).
• Integration of edge (cloud) infrastructure with main cloud (private datacentre, private cloud, public cloud etc.) e.g. for IoT gateway and edge computing platform (e.g. use of a common PaaS layer) [3].

Such shared implementation could help driving down the overall cost and improve the business case for adoption.

Thus, if there are enterprise scenarios, where the use of small cell networks makes sense together with the use of IoT gateways and edge computing platforms (a la ETSI MEC), the costs related to hardware infrastructure and features that render the systems ready for mission-critical use could possibly be shared as could the costs related to management software of the underlying infrastructure (compute, storage, networking). In addition, some system functions might only need to be implemented once (as a platform service) instead of multiple times (e.g. a traffic offload function). A major prerequisite for such a scenario is of course the decoupling of hardware and software and this trend is firmly under way in all three areas under the notion of virtualisation. Another prerequisite is an open system architecture where system or platform services expose their services via APIs, which can be dynamically discovered by newly deployed applications which are hosted on this edge infrastructure. A topic for further study is management and orchestration of all virtualised functions and the (possibly shared) roles of the enterprise and the network operator in this regard.

Summary
We said that three different technology areas are gaining importance in particular for Internet of Things use cases in the enterprise market: Small Cell Networks, IoT Gateways and Edge Computing platforms. All three areas are adopting the concept of virtualisation. As a result this leads to a separation of the virtualisation infrastructure (compute, storage, networking) and the system applications which are hosted on top of this infrastructure (small cell controller software, IoT gateway functions, edge computing platform services).

As a result, all three technology areas can benefit from a common, shared virtualisation infrastructure and its management and control.

We also said that because small cells (LTE, 5G), IoT gateways and edge computing platforms are of special interest to the enterprise market (e.g. Industrial IoT), high standards with respect to quality of service, real-time capability, security, reliability and availability will become important and in various cases business critical. As long as the core functions of above three systems can all be flexibly deployed on the same virtualisation infrastructure in a multi-vendor fashion and as long as those core functions (like small cell concentrator or operational analytics engine of an IoT gateway function) can tap into common platform services via open interfaces (e.g. for event logging), the costs to render such systems enterprise-grade can be shared across the three (today standalone) system areas.

As a result, one would expect that overall system deployment costs with enterprises can be reduced compared to independent, standalone deployments. At least, any incremental increase in cost driven by enterprise requirements for security, reliability and QoS should be moderated due to the sharing of common platform functions.

This could furthermore be important also in the context of 5G, where the 5G system architecture (including radio technologies) shall support the requirements of different vertical industries, from eHealth to Smart Factory [9, 10, 11]. In some of these vertical industries, cellular technology will be in competition with other wireless and ‘wireline’ technology and proprietary solutions that over time got fine-tuned to achieve desired reliability and industrial-use quality [8]. Thus, if extra system features in 5G must to be afforded to meet those stringent enterprise requirements, it will help if those features can be designed such that they can be used by multiple (enterprise IoT) applications and virtual network system functions (like small cell controller functions, edge compute platforms etc.).

Above discussion gives the notion of lite datacenter or micro-datacenter an interesting new dimension, in the context of enterprise IoT applications.

Disclaimer: Views stated, opinions expressed or analyses provided are solely those of the individual authors as private persons and shall not be considered or interpreted as views, opinions, or analyses of any of the companies they may be liaised with, employed by or do work for.

This article can be downloaded in pdf here.

References
[1] Klas, Guenter. “About Mobile Edge Cloud Computing – An Introduction”. 8 Nov 2015.
https://yucianga.info/wp-content/uploads/2015/11/15-11-08-About-Mobile-Edge-Cloud-Computing-%E2%80%93-An-Introduction-v1.pdf

[2] Klas, Guenter. “Fog Computing and Mobile Edge Cloud Gain Momentum – Open Fog Consortium, ETSI MEC and Cloudlets”, 22 Nov 2015. https://yucianga.info/wp-content/uploads/2015/11/15_11_22-_Fog_computing_and_mobile_edge_cloud_gain_momentum_Open_Fog_Consortium-ETSI_MEC-Cloudlets_v1_1.pdf

[3] Klas, Guenter. “Edge cloud to cloud integration”. 4 Feb 2016.
https://yucianga.info/wp-content/uploads/2016/02/16_02_04_Edge_cloud_to_cloud_integration_for_IoT_v1.pdf

[4] ETSI ISG NVF. http://www.etsi.org/technologies-clusters/technologies/nfv

[5] ETSI ISG MEC. http://www.etsi.org/technologies-clusters/technologies/mobile-edge-computing

[6] Huawei, “Five trends to Small Cell 2020. MWC 2016”, Feb 2016
http://www.huawei.com/en/news/2016/2/Small-Cell-White-Paper

[7] Nokia, “Nokia Small Cells – innovative ways to expand coverage and capacity for the future”. 2015
https://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwjfiKfz7tvMAhWrK8AKHVZ0CtYQFggcMAA&url=http%3A%2F%2Fnetworks.nokia.com%2Fsites%2Fdefault%2Ffiles%2Fdocument%2Fnokia_small_cells_brochure.pdf&usg=AFQjCNFLLRVCi66PopmtxlghVi7cRqRhYg&sig2=qa-bMZZEIdFsR-kKv6-8Wg&bvm=bv.122129774,d.ZGg

[8] Texas Instruments, “Understanding wireless connectivity in the Industrial IoT“ https://www.ti.com/seclit/wp/swry016/swry016.pdf

[9] 5GPPP, “5G empowering vertical industries”
https://5g-ppp.eu/wp-content/uploads/2016/02/BROCHURE_5PPP_BAT2_PL.pdf

[10] 5GPPP. “5G and e-Health”, Sept 2015
https://5g-ppp.eu/wp-content/uploads/2016/02/5G-PPP-White-Paper-on-eHealth-Vertical-Sector.pdf

[11] 5GPPP, “5G and the Factories of the Future”, White Paper, 2015
https://5g-ppp.eu/wp-content/uploads/2014/02/5G-PPP-White-Paper-on-Factories-of-the-Future-Vertical-Sector.pdf

[12] Haddad W., Mahkonen H., Manghirmalani R. “Unikernels meet NFVS“, Ericsson, Oct 2011,

[13] 2016 Unikernels and More: Cloud Innovators Forum Session Information, Xen project.
http://wiki.xenproject.org/wiki/2016_Unikernels_and_More:_Cloud_Innovators_Forum_Session_Information#Unikernels:_The_Past.2C_the_Present.2C_the_Future

[14] Juniper, “Docker and microservices bring new agility and economy to the data center.” Apr 2016. http://www.juniper.net/uk/en/insights/containerized-applications/

[15] Davies N., Taft N., Satyanarayanan M., Clinch S., Amos B., “Privacy Mediators: Helping IoT Cross the Chasm”, HotMobile ’16 February 26-27, 2016, St. Augustine, FL, USA http://eprints.lancs.ac.uk/78255/

[16] Stine, M, “Migrating to Cloud-Native Application Architectures”, O’Reilly 2015 https://download3.vmware.com/vmworld/2015/downloads/oreilly-cloud-native-archx.pdf

[17] Kobielus J, IBM, “Deploying analytics microservices in the cloud”, Feb 2016 http://www.ibmbigdatahub.com/blog/deploying-analytics-microservices-cloud

[18] Kobielus J, IBM, “Fogs, logs and cogs: The newer, bigger shape of big data in the Internet of Things”, Feb 2015 http://www.ibmbigdatahub.com/blog/fogs-logs-and-cogs-newer-bigger-shape-big-data-internet-things

[19] General Eletric. “Predix – Overview” https://www.predix.com/overview

[20] General Electric. “Predix: The Industrial Internet Platform”, platform brief. Feb 2016 https://www.predix.com/sites/default/files/predixplatformbrief-021716.pdf

[21] Small Cell Forum Champions, Rome, Sept 2015,
http://www.smallcellforum.org/site/wp-content/uploads/2015/11/Rome-Champions-Day-report.pdf

[22] Small Cell Forum, “Crossing the Chasm, Small Cells Industry November 2015“. http://www.scwsasia.com/files/crossing_the_chasm_small_cells_industry_2015.pdf

[23] Small Cell Forum, “Enterprise overview”, Document 102.06.01. Jan 2016
http://scf.io/en/documents/102_-_Release_Six_-_Enterprise_Overview.php

[24] Small Cell Forum, “E-SCN network architectures”, Document 067.06.02. Jan 2016
http://scf.io/en/documents/067_-_Enterprise_small_cell_network_architectures.php

[25] Small Cell Forum, “Virtualization for small cells: Overview”, Document 106.06.01. June 2015
http://scf.io/en/documents/106__Virtualization_for_small_cells_Overview.php

[26] Small Cell Forum, “Small Cell Services API”, Document 152.06.01, Jan 2016
http://scf.io/en/documents/152_-_Small_cell_services_API.php

[27] Small Cell Forum, “Small cell virtualization: Functional splits and use cases”, Document 159.06.02, Jan 0216
http://scf.io/en/documents/159_-_Small_cell_virtualization_functional_splits_and_use_cases.php

[28] Small Cell Forum, “Network aspects of virtualized small cells”, Document 161.06.01, June 2015
http://scf.io/en/documents/161__Network_aspects_of_virtualized_small_cells.php

[29] Financial Times, 24 Apr 2016, p15

[30] Industrial Internet Consortium, “Industrial Internet Reference Architecture”, 2015
http://www.iiconsortium.org/IIRA-1-7-ajs.pdf