The job description of an engineer in the oil and gas industry has been rewritten in recent years, and much of this shift is due to technology. The industry, and society in general, is becoming more and more focused on short-term results. The big data revolution, with huge data sets being transmitted in real-time, has called for rapid processing and decision-making. On top of this, engineers are under more pressure than ever to deliver immediate results in the form of safe, reliable and profitable operations has intensified. Change always carries threats and opportunities with it. On one hand, this presents a challenge to long-held, conservative ways of working, and will require major adjustment to attitudes in the industry. However, the market is calling out for ambitious, tech-minded graduates willing and able to combine emergent, game-changing technologies with traditional engineering skills. Any job market depends heavily on supply and demand. The energy and chemical industry is no … [Read more...] about Big data and cloud skills – the key to a prosperous career in the energy industry?
There’s data, and then there’s big data. So, what’s the difference?Big data in general refers to sets of data that are so large in volume and so complex that traditional data processing software products are not capable of capturing, managing, and processing the data within a reasonable amount of time.These big data sets can include structured, unstructured, and semistructured data, each of which can be mined for insights.How much data actually constitutes “big” is open to debate, but it can typically be in multiples of petabytes—and for the largest projects in the exabytes range.Often, big data is characterized by the three Vs:an extreme volume of dataa broad variety of types of datathe velocity at which the data needs to be processed and analyzedThe data that constitutes big data stores can come from sources that include web sites, social media, desktop and mobile apps, scientific experiments, and—increasingly—sensors and other devices … [Read more...] about What is big data? Everything you need to know
The way we manage and monitor networks is morphing. Passive, reactive tools are being replaced by more proactive network analytics systems that give the entire network team a single source of truth about network behavior and a much deeper understanding of where infrastructure issues are hiding and what to do about them.Before IT was forever changed by the arrival of mobile devices, virtualization and cloud apps, fixing network problems was relatively simple because users plugged into the network from one location to access local applications and resources.But with the proliferation of diverse wireless clients – a range of hardware using different versions of different operating systems (the permutations can quickly scale into the thousands) – and the use of applications and services that are often not under IT’s control, getting to the heart of individual user and systemic client network problems has become the new nightmare.This is particularly acute in the … [Read more...] about Network analytics will change everything
Data is clearly not what it used to be! Organizations of all types are finding new uses for data as part of their digital transformations. Examples abound in every industry, from jet engines to grocery stores, for data becoming key to competitive advantage. I call this new data because it is very different from the financial and ERP data that we are most familiar with. That old data was mostly transactional, and privately captured from internal sources, which drove the client/server revolution. New data is both transactional and unstructured, publicly available and privately collected, and its value is derived from the ability to aggregate and analyze it. Loosely speaking we can divide this new data into two categories: big data – large aggregated data sets used for batch analytics – and fast data – data collected from many sources that is used to drive immediate decision making. The big data–fast data paradigm is driving a completely new architecture … [Read more...] about 5 top data challenges that are changing the face of data centers
The Economist has a series of articles right now on the impact of data on business, and pointing out how new data driven companies are rapidly not simply transforming sectors but eliminating competitors. Their argument is that these internet giants are going to require regulators to look at new ways to avoid monopolies and anti-competitive practices. It describes data as a more valuable resource than oil in the modern world.When we look at firms like Amazon, Lyft, Google and what they have done to retail, taxi firms and advertising it is hard to understate just what a dramatic impact these firms have had. Yet fifteen years ago the idea that Amazon would be one of the world’s largest data center service providers would have seemed ridiculous, Amazon are a retailer, what has that to do with data centers? Google are a search engine with some adverts, what have they to do with mobile phones? Apple make some shiny laptops and computers, but mobile phones and speech recognition? Not … [Read more...] about Are you choosing to lose in a data-driven world?
Splunk takes machine data in the form of log files, or other computer-generated text files, and turns it into operational intelligence. Machine data (log files), as data goes, is not particularly exciting as compared to say, human genomic information, but Splunk turns our "digital exhaust" into electronic gold for those who are willing to spend a little time to examine it. In fact, machine data is the fastest growing area of big data in enterprises; it includes every user transaction, every system message, every suspicious activity, and every machine-to-machine interaction. And believe it or not, it's exciting to see what's happening on your network. Splunk's powerful Search Processing Language (SPL), originally based on the Unix pipeline and SQL, makes data analysis much more fun and much less complex than you'd think. SEE: Research: How big data is driving business insights in 2017 (Tech Pro Research) The big data part of Splunk comes from the log files, events, and other … [Read more...] about How to tap into Splunk’s big data analytics engine
The majority of employed data scientists gained their skills through self-learning or a Massive Open Online Course (MOOC) rather than a traditional computer science degree, according to a survey from data scientist community Kaggle, which was acquired by Google Cloud earlier this year.Some 32% of full-time data scientists started learning machine learning or data science through a MOOC, while 27% said that they began picking up the needed skills on their own, the 2017 State of Data Science & Machine Learning Survey report found. Some 30% got their start in data science at a university, according to the survey of more than 16,000 people in the field. More than half of currently employed data scientists still use MOOCs for ongoing education and skillbuilding, the report found, demonstrating the potential of these courses for helping people gain real world skills. SEE: Job description: Data scientist (Tech Pro Research)Data scientist took the no. 1 spot in Glassdoor's Best Jobs in … [Read more...] about Report: 59% of employed data scientists learned skills on their own or via a MOOC
Big data has many applications, but none perhaps as ubiquitous and overlooked as geolocation. All of the mapping and navigation systems we use daily depend upon geo data—data that is constantly being refreshed. The range of things we map is mind boggling, from submarine cables to Napoleon's retreat from Moscow, and you can find many of them at ESRI, a private company founded in 1969 that works in the Global Information Systems (GIS) space. ESRI has a popular solution called ArcGIS for managing big data that enables companies to visualize and analyze information at terabyte scale—revealing patterns, trends, and relationships in a way that reports don't, or can't. Since big data is stored in many different places, a common challenge in creating GIS applications is getting fast access to disparate data. Developers have to constantly change the application to accommodate different data stores. I recently spoke with Mansour Raad, senior software architect at ESRI and a regular … [Read more...] about Here’s one key to managing geolocation data at scale
Amazon had a need for a non-relational, highly scalable approach to data, and every enterprise is now swimming in NoSQL as a result. As Amazon CTO Werner Vogels outlined, Amazon's "straining database infrastructure on Oracle led us to evaluate if we could develop a purpose-built database that would support our business needs for the long term." Dynamo was born, initially as a published paper that launched a series of promising NoSQL databases like Apache Cassandra. In so doing, Amazon arguably began to lay the foundation for a clear and present danger to Oracle's database reign. Dynamo wasn't the first NoSQL database, nor is the subsequent DynamoDB the biggest challenger to Oracle. But by scratching an internal itch for greater scale, Amazon set up the start of a database war that threatens Oracle's long-term dominance. Amazon didn't set out to topple Oracle. Far from it. As Vogels described, Amazon was simply trying to take care of its increasingly demanding database needs, as … [Read more...] about How Amazon helped bring NoSQL to the enterprise mainstream
There can be many steps to data preparation when it comes to big data, but some of the principal processes include collecting the data, cleaning it for abnormalities, filtering it so you can discard data that you know you won't need, and normalizing the data so it is congruent and can integrate with the rest of the big data you have in your data repositories. According to a poll by data mining and crowdsourcing firm CrowdFlower, data scientists and analysts can spend as much as 80% of their time cleaning and preparing data, which is often a painstakingly grueling and laborious process. For this reason alone, the idea of outsourcing the data cleaning and preparation process is beginning to gain traction within organizations. SEE: IT leader's guide to big data security (Tech Pro Research) "There are mountains of data that come into organizations from diverse sources, but there is also data locked up in internal company systems themselves," said Rob Consoli, chief revenue officer of … [Read more...] about Why your company might want to consider outsourcing big data preparation