Not Invented Here Syndrome

Not Invented Here Syndrome

Gartner Professional Advice - Machine Learning Life cycle

End to End Machine Learning and analytics Architecture

Page 29

History of APIs

   High Resolution Image

High Resolution Image

High Resolution Image

High Resolution Image

Collection of Forgotten Interfaces

The Fun-damental Laws of Enterprise Architecture

Compiled by: Shak Kathirvel  Linus' Law - Given enough eyeballs, all bugs are shallow. Linus' Law - Simple programs never work the first time. Complex programs never work. Ryan Singer - So much complexity in software comes from trying to make one thing do two things. Sturgeon's Law - Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. Gall's Law - A complex system that works is invariably found to have evolved from a simple system that worked. Conway's Law - Any piece of software reflects the organizational structure that produced it. Brooks' Law - Adding manpower to a late software project makes it later. AND The bearing of a child takes nine months, no matter how many women are assigned. Wirth's Law - Software gets slower faster than hardware gets faster. Gate's Law - The probability of a bug manifesting itself in software quadruples when said software is being demonstrated. Amara's Law - We tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run. Kranzberg's First Law of Technology - Technology is neither good nor bad; nor is it neutral. Classen's Law - In order to achieve a linear improvement in usefulness over time, it's necessary to have an exponential increase in technology over time. Gustafson's Law - In computer engineering, any sufficiently large problem can be efficiently parallelized. Koomey's Law - The energy of computation is halved every year and a half. Moore's Law - An empirical observation stating that the complexity of integrated circuits doubles every 24 months. Malik's Laws of Service Oriented Architecture - Click here for full list. Hofstadter's Law - Estimates are called estimates for the same reason that fishing isn't called catching. Hofstadter's Law - Double your estimate and replace with next unit of time. For example: original estimate: 6 weeks. Double: 12 weeks. Next unit of time: 12 months. Niven's Law - Any sufficiently and rigorously defined magic is indistinguishable from technology. Niven's Law - Ethics change with technology. Sayre's Law - In any dispute, the intensity of feeling is inversely proportional to the value of the stakes at issue. Douglas Adams - A common mistake people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools. Bill Gates - If the car industry behaved like the computer industry over the last 30 years, a Rolls-Royce would cost $5 and get 300 miles per gallon. General Laws Campbell's Law - The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor. Sowa's Law of Standards - Whenever a major organization develops a new system as an official standard for X, the primary result is the widespread adoption of some simpler system as a de facto standard for X. (e.g. Gold!!) _________________________________________________

Data is Hyper liquid.

Data, information, and knowledge – The Trinity of enterprise value

Based on the fundamental premise that data, information, and knowledge, are three intricate, indispensable and interdependent elements, all modern HyperScale systems will be built to unearth its untapped business values and business models with unprecedented processing power, speed and accuracy. We view data as simple discrete facts that become information when they are combined into meaningful structures, which subsequently becomes domain knowledge when put together in context. For strategic and growth needs, invariably all enterprises explore new avenues to harness the value buried deep inside this domain knowledge base by generating metrics, categorizing results, assessing values, making forecasting and predictions and finally making informed decisions.

Data has been there for a long time. What is different now? The 3Vs. The vast amount of dataset (Volume) that are generated now a days are distributed, diverse, disparate and heterogeneous in nature. They are structured, unstructured and semi- structured (Variety). Besides, these data set are growing at exponential pace (Velocity). For example, a single blade in a GE gas or wind turbine, generates 500 gigabytes of data each day and that’s just one blade. So that means in 30 days you're generating as much content as the print collection of the Library of Congress. There are about 4,500 GE gas turbines around the world, each with dozens of blades. There are around 22,000 GE wind turbines all outfitted with multiple sensors.

We recognize and convinced that the 3Vs of the modern data characteristics - Volume, Velocity and Variety - generates enormous challenges in areas of Storage, Retrieval, Security, Sharing, Analysis and Reporting. On the flip side they also open up massive business opportunities throughout different phases of data life cycle. Besides the 3Vs nature of the data we also recognize the density of information embedded in them. That is where true enterprise value lies. Without it any enterprise will be flying blind.

Data is Liquid and it is everywhere

Nature of today’s HyperScale data has striking similarities with water – very precious resource but abundant on earth. Needless to say it is one of the essential element for survival of all living forms.  Remarkably, HyperScale data shares the basic characteristics of water – precious, abundant, storable, transportable, transformable, vulnerable and susceptible to contamination and process able for consumption. 

Much like the way massive resources and infrastructures – dams/reservoirs, electric power stations, security and backup systems, purification and filtration systems, distribution-channels, maintenance, monitoring systems, billing/accounting/monetization systems – are built to harness the power of water, we believe that HyperScale data needs massive storage structures, security systems, high speed networks, HyperScale processing machines, monitoring and regulatory structure to extract and harness the vast power embedded in them.

Data Security – Multilayered approach

With big data comes big responsibility. Its security. A multilayered data security strategy – Prevention, Detection, and Policy/Administration - could be implemented as it is proven efficient to achieve all facets of data security. It maximizes the security controls at each layer as its thwarts the intruder’s malicious intensions and efforts at multiple levels.

Preventive controls stop intruders from gaining unauthorized access to our data.  Detective controls centralize auditing and reporting across the organization so that either security breach or compromised system can be swiftly detected and necessary actions be taken. Using policies / administrative controls, unlimited and ad-hoc access to application data can be prevented but the same time allow legitimate administrative activity.

HyperScale Data Security

HyperScale data security layer has been designed to include a comprehensive data security management and governance model. It is implemented as 3 pronged approach.

·       2 Level Identification,

·       Data Encryption and

·       Data de-identification.

Two Level authentication: It provides secure access to our data by authenticating more than just the user’s password. The 2nd level is something in addition to the password. It delivers strong authentication with a range of easy 2nd level verification options—phone call, text message, or mobile app notification—allowing users to choose the method they prefer.

Data Encryption: We ensure that the data we handle is encrypted when it is at rest and as well in transit. We recognize that Data level encryption is more efficient than application level encryption since the same data is shared by wide array of applications. Rather than coding encryption and decryption algorithm into each application, we chose to handle encryption and decryption steps at the data level. When each application needs to process the data, it is decrypted at the data level and securely transmitted using our SSL channels. 

Data de-identification. Besides encryption, we secure our data by de- identification which redacts sensitive data out of the application layer. Users looking at an application may see asterisks instead of actual information. For example, social security numbers might reveal only the last four digits for reference purposes. The data in the database is encrypted, but redacted when viewed.

Shak Kathirvel

Pragmatism and Enterprise Architecture

Two sides of same coin.

The role of architecture, albeit neither overtly tangible nor receives attention grabbing headlines, plays a critical role in the predictable daily operations and deliverance of committed services to customers. Those two attributes are more strategic than tactical. Predictability is a proxy for risk management, the ability to respond swiftly at a time of misfortune. Deliverance is a measure of dependability, trust, and branding that translates into leverage in negotiations, which in turn becomes a decisive factor when choosing a partner among equals. Good IT system architecture makes the critical difference between:

a) an enterprise's strategic advantage and an Achilles' heel

b) Citadel and McMansion

c) near-zero technical debt and a do-over

In the field of architecture, there are 2 complementary and contradicting concerns: Purity and Pragmatism. As always, there are proponents and detractors on both side of the aisle. Purists weigh interoperability of disparate building blocks, mutual co-existence, development and deployment simplicity, and low/optimal operational costs as uncompromising principles when developing architectural blue prints. Pragmatists weigh creating competitive moat through technical superiority, protecting intellectual properties with patents, and finessing the licensing model to maximize profitability when developing arch models. Understanding what the right mix of the two concerns helps make short and long term decisions that are both strategic and tactical.

How would it look without them?

An infrastructure engineer can easily read the specs, order, and assemble a bunch of pizza box-like server units from a myriad of brand name local vendors or white label vendors in Asia. They then download a personally delectable flavor of OS from an open source mirror or a professional services vendor and install them on top of the pizza boxes. Finally, they choose a run-of-the-mill web and app server combo from a traditional/reputed shop or chose a combo from state of the art neo noir style frameworks from an intrepid, iconoclast developer/group and configure them to their heart's content and put them into production—without architects.

A self-proclaimed network engineer orders run-of-the-mill network switches, routers, HBA, and cables from the Sears-style catalogue (even I've ordered from a 600+ page catalogue), creates a DMZ layer with iron-clad firewalls to thwart hackers, and configures authentication and authorization layers so that only the clan can get in - without architects.

In the same way, a database engineer follows instructions in the installation wiki and FAQ's of a traditional SQL or NoSQL Server product from a reputed vendor or picks one of many open source flavors, installs, and configures into the boxes the infrastructure engineer has built for them and turn them on in production—without architects. 

A group of nocturnal application developers can write a bunch of cool applications and put them into production—without architects.

Voila, your entire IT system is now up and running. Currently, thousands of production servers run in enterprise data centers—without architects. It is not atypical that a bunch self-styled, innovative, young, and ambitious professionals make critical decisions on how to build servers, create databases, write applications, and operate the entire system.

Where is the Breaking point?

The ad-hoc IT infrastructure buildout, somewhat mercurial application development, and deployment and maintenance process will of course work, but it will only take you so far. Without incorporating pragmatic architectural considerations in the buildout, development, and operating model we know things break down at some point.

If history is any guide, the breaking point is always scale, performance, and the 99.999999% availability of the IT system for a 24/7 business model. Scale and performance are indicators of sustained business growth, and the 99.999999% availability is an indicator of brand, trust, and service commitment to customers.

If you need an example, look no further than the initial web application developed by Pierre Morad Omidyar, a software developer by trade and founder of eBay. Omidyar began to write the original computer code for an online venue to enable the listing of direct person-to-person auctions for collectable items. He created a simple prototype on his personal web page and on Labor Day in 1995 he launched an online service called Auction Web, which would eventually become the auction site eBay. The service was hosted on a website Omidyar had originally created for information on the Ebola virus.

Another example is the initial web application developed by Mark Zuckerberg, founder and software developer at Facebook, in his dorm room. There are several examples like this if you scour the web.

In both cases, the initial development and operating model - without incorporating established architectural considerations – of the IT system did not last long due to scale and performance shortcomings when their web based business applications experienced sudden and exponential growth.

Where the rubber meets the road

The complexity of technological choices and integration challenges that come along with them demands skill and experience in how to stitch together and tune different building blocks into the right set of useful/consumable products and services. Physical Infrastructure (H/W), data (DB's), application development platforms (middleware, frameworks), applications (CRM, HR, ERP, custom apps, etc.) and an array of deployment and operational tool sets are 4 critical components of any IT company to keep the revenue generating engine humming and running. Integration of infrastructure, application development, and database management tools require the skills and leadership beyond those of engineers and developers.

Architects:

• Give physical shapes to concepts that have business value
• Provide the blueprint
• Compile the tool list and building blocks so that IT professionals and/or system engineers can build the infrastructure, and database folks can start constructing the data models, their ownership, and the intricate relationship between them
• Specify details to the developers or software engineers so that they can focus on coding the  applications
• Guide the development of operating manual/standard operating procedures so that operational folks can start their long journey of learning how to "drive" the system on a day to day basis

Thanks,

Shak Kathirvel, Application Architect